One tiny network drawn as neurons and weights, then as the matmuls that compute it.
Neural networks are drawn as neurons joined by weights, but computed as matrix multiplications. Here is one tiny network drawn both ways, so you can watch one become the other.
It reads three characters and scores the next one, like the model in Andrej Karpathy’s makemore, only smaller: 6 characters (. a e l n o), 2 numbers per character, 4 hidden neurons and a batch of 10 examples from ella and noel. Softmax and the loss work exactly as in Training a bigram model, so they are left out here.
Training examples from ella, noel · . marks a word boundary
