04 of 12
Language models
A language model is a machine that has seen so much text that it got good at guessing the next word. That habit packs a lot of the world's writing into one place. If you can guess what usually comes next, you can draft, summarize, translate, and answer questions.
Around 2018 to 2020, attention, word pieces, and a scrape of the public web produced systems that could finish a paragraph, translate, and emit code-shaped text from that one habit. The chat products that landed in 2022 used the same engine after a narrower practice pass: follow instructions, decline some requests, prefer answers people ranked as better.
Why does guessing the next word write an email?
The public web is full of email, memos, and Stack Overflow. OpenAI's 2020 model could already finish a paragraph in those shapes. ChatGPT, in 2022, was the same engine after a pass of follow this request. You type, and it keeps guessing pieces until a reply has been assembled.
Why does it invent facts?
In 2023 a New York lawyer, Steven Schwartz, filed a brief with fake case citations ChatGPT had invented. The model was guessing a fluent next sentence.
How much can it keep in mind at once?
Labs now advertise windows of hundreds of thousands of words. Cursor still asks for a notes file in the repo, because stuffing four hundred files into every question is slow, expensive, and still drops the constraint from last quarter. A decision from last month is not in there unless someone wrote it down.