10 of 12
Senses
You can read that the soup is ready, and you can also see the steam and hear it simmer. A machine that only reads is like a friend who only takes phone calls. It misses what is happening in the room.
For a long time, seeing and hearing were their own industries. In the early 2020s that wall came down. Teams trained the same family of next-piece habits on captions next to pictures, and on thousands of hours of recorded talk. You could paste a photo into a chat and ask a question. A screenshot is just a picture of software, which is how an agent starts to work on a desktop.
Why is a screenshot enough to use software?
In late 2024 Anthropic showed Claude looking at a desktop and clicking. OpenAI's Codex, by 2026, does the same in the background while you keep working. A screenshot is a picture of software, and clicking only works if the look is good enough to find the button.
Why does it invent text that isn't in the photo?
Pictures that came with captions taught the system to describe a scene. Invoice tools still invent line items that are not on the page. The model will also invent a button label on a grayed-out Save, because it is describing a UI it has seen in training, not the pixels in front of it.
Why is a kitchen still harder than a studio?
OpenAI's Whisper was trained on internet video audio. A Zoom call with two people talking over each other is still where meeting notes invent commitments nobody made.