The Washington Post just published a really cool breakdown of where chatbot training data really comes from. It’s just one data set being studied here — Google’s C4, which is used by Meta’s LLaMa and some other models — but it reveals a lot about how LLMs learn from the web, and what they’re really learning.
The least surprising bit? There’s a whole lot of Wikipedia in there.









