01 of 12
Data
You don't teach a kid what a dog is by reading them a definition. You point at one, then another, then a cat, and after enough pointing they just know. A machine needs the same thing, only more of it. That pile of examples is data.
In 2009 a Stanford group led by Fei-Fei Li released ImageNet. People looked at pictures and typed what they saw. In 2012 a system trained on ImageNet won the year's photo-recognition contest by a wide margin. Speech products moved the same way, on thousands of hours of recorded talk with transcripts, instead of someone writing sound rules by hand.
Why not just write the rules?
Rules were hard to scale because researchers needed to write a rule for every situation that counted as that object. Fei-Fei Li's bet was to label millions of photos instead.
What if the training data is wrong?
The Times sued OpenAI because ChatGPT would reproduce Times articles almost verbatim. GitHub Copilot would complete licensed functions from public repos a person was not free to copy. Getty went after the image models for scraping its archive. If a general model hasn't been trained on your data, it guesses, and those guesses are hallucinations.
Who decides what goes into the training data?
A lot of the text under ChatGPT comes from Common Crawl, a public copy of the web, including pages nobody offered as training data. Time reported in 2023 that contractors in Kenya, working through Sama, an SF firm, labeled sexual abuse, hate, and violence so OpenAI could teach the model what not to say. They were paid about two dollars an hour.