03 of 12
Neural networks
A neural network is a stack of very simple pattern-finders. Each layer notices something a bit more abstract than the last: an edge, then a shape, then something that looks like a face. Nobody draws those features by hand. They come out of the examples during learning.
Stacks like this existed in the 1980s and stalled, because the miss faded as it went backward. The 2012 image-contest winner was a deeper stack that could actually be trained on ImageNet, with graphics chips. In 2015 extra wires that let the miss jump over layers made fifty- and hundred-layer stacks trainable. In 2017, attention let every word look at every other word in one step, which is what made language models affordable to grow.
Why not one giant layer?
The same small detector for an eye can run everywhere in a photo. That reuse is why AlexNet even fit on the graphics chips of 2012. A giant layer on raw pixels was the older idea, and it ran out of memory. In 2015, extra connections between layers made it possible to train stacks that were fifty layers deep.
What does a layer actually do?
Each unit adds up weighted inputs, then bends the total so it is not just a straight line. That bend is what lets later layers build a face from edges. Because the stack is smooth, a miss at the end can still change the early layers.
What did attention change?
The older language stacks read a sentence left to right and forgot the beginning. In 2017, Ashish Vaswani and colleagues at Google published 'Attention Is All You Need': every word looks at every other word in one step. That design is the transformer, and it is why you could grow a language model on a scrape of the web.