April 23, 2018 · 2 min read

Cs231n — assignment 1 — softmax

This problem set really bothered me.

The original problem: linear classifier with a softmax loss function

Softmax is simple, it is 
    softmax = e^x / Sum([e^k for k in item])

Then you take the cross-entropy loss: 
    loss = -np.log(softmax)

This is nice because the best score possible gives us a softmax equal to 1, and -np.log(1) gives us our loss which ends up being 0.

That's easy to say, but you need to remember you're dealing with vectors and matricies, which make everything ugly stupid.

So your forward pass looks like this: 
    loss = 0.0
    scores = X.dot(W)
    for i in range(scores):
      correct_class = scores[i][y[i]]
      top = np.exp(correct_class)

      exponentized_bottom = np.exp(scores[i])
      bottom = sum(exponentized_bottom)

      loss += -np.log(top/bottom)

easy as shit. The trick there is that X and W are matricies, and then you have to go row by row.

So, conceptually, each row in the scores has an effect on the loss

But what is X? What is W?

X is easy. It's N x M. And that means W is M x C. And that means the resultant Scores matrix is N x C.
These values have meaning: N is the number of data points. One additional N is one additional data point.
M is the number of features. That means W needs to be unique for different datapoint types, the types that have different features need bigger M's.
And C is just the number of classes. 
         X                         W                            scores

      [[a, b],                [[g, h, i],              [[ag+bj, ah+bk, ai+bl],
 N     [c, d],  (dot)    M     [j, k, l]]    =   N      [cg+dj, ch+dk, ci+dl],
       [e, f]]                                          [eg+fj, eh+fk, ei+fl]]

       M                           C                              C
And let's expand the calculation of loss across all three rows, and because rows correspond to datum we can say look at the loss as calculated per datum and across all three data: 
    score = e^(ah+bk) / (e^(ag+bj) + e^(ah+bk) + e^(ai+bl))
    loss_from_first_datum = -log(score) 

    score_2 = e^(cg+dj) / (e^(cg+dj) + e^(ch+dk) + e^(ci+dl))
    loss_from_second_datum = -log(score_2)

    score_3 = e^(ei+fl) / (e^(eg+fj) + e^(eh+fk) + e^(ei+fl))
    loss_from_first_datum = -log(score_3) 

    overall_loss = (loss_from_first_datum + loss_from_second_datum + loss_from_third_datum) / 3
But what does it all mean?

Let's look at dL with respect to different parts of dW. For instance, g. g shows up in all three rows, so we know that all three data are affected by g. But only score_2 has g in both the numerator and the denominator.

So we're flirting with the hard part of this problem. The part where we figure out gradient descent, or a dW matrix we can update W by to incrementally decrease our lives loss.

It may not be obvious, but trying to find out how to put together a dW matrix is not straightforward. You can do it via chain rule, I guess. First step is to undo that awful divided by three thing: 
    chain rule: df(g(x)) = f'(g(x)) * g'(x)
    therefore: doverall_loss =

← All writing