A bigram model predicts the next character from the current character alone. It is the first model in Andrej Karpathy’s makemore, and it is small enough to show every number at once: 27 inputs, 27 outputs, and a single 27 × 27 weight matrix W.
The vocabulary is a–z plus ., which marks the start and end of a word. The word emma gives five training examples: .→e, e→m, m→m, m→a and a→.. Pick one, read the forward pass left to right, then switch to the backward pass and read it right to left.
Training examples from emma · . marks a word boundary
Pass
Gradient descent · learning rate 2
Click a letter on the left to change the input, or any row on the right to change the target. On small screens, scroll the diagram sideways.
Forward: a row lookup, then a softmax
The input is one-hot: 1 for the current character, 0 everywhere else. Multiplying a one-hot vector by W just selects one row, so this “linear layer” is really a lookup table. With e as input, the 27 logits z are row e of W, one score per possible next character. That’s why only the edges leaving e light up.
Softmax turns scores into probabilities. Exponentiating makes every score positive; makemore calls these counts because they play the part of a bigram count table. Dividing by their sum (the ÷Σ node) makes them add up to 1.
The loss is the negative log-probability of the character that actually came next. y is one-hot on that character, so −y·log p is zero in every row but one. With the starting weights, e→m gets p = 0.03 and a loss of 3.54. For scale, guessing uniformly scores −log(1/27) ≈ 3.30: the random weights do slightly worse than knowing nothing.
Backward: everything reduces to p − y
Switch to Backward. Each cell now shows ∂L/∂(that value): how quickly the loss changes if that value goes up. Read right to left, starting from ∂L/∂L = 1.
The loss is a plain sum, so every −y·log p term gets a gradient of 1. Multiplying by y then zeroes every row except the target. y is data, not a parameter: it decides where gradient can flow.
At the probabilities, the target’s gradient is −1/p, which is −34.39 when p = 0.03: an unlikely answer gets a huge push. Softmax tames it. By the time it reaches the logits, the gradient is simply p − y: −0.97 for m, and +p for every other character. Each wrong character is pushed down in proportion to the probability it took.
Because the input is one-hot, the gradient on W is zero everywhere except row e, and that row is p − y. The coloured edges leaving e show it.
Taking a step
Train one step applies W[e] ← W[e] − 2·(p − y). Press it a few times: p(m) climbs and the loss falls. Only row e changes, so examples that start with another character are untouched. Examples that share the row are not: after training e→m, click the a row on the right and the loss for e→a has gone up.
Now try m→m and m→a. Both read row m, so the best the model can do is split the probability between them, which leaves each with a loss of log 2 ≈ 0.69. That floor is the cost of seeing only one character of context, and the reason the next models in makemore look further back.