LECTURE // LARGE LANGUAGE MODELS
Deep Dive into LLMs like ChatGPT — Andrej Karpathy. Part 1 of 13, covering 0:00–19:00: the pretraining data pipeline and tokenization.
What this lecture argues
ChatGPT is not magic — it is the output of a concrete, learnable pipeline. Internet text is filtered, chopped into tokens, and compressed into the parameters of a neural network; that network is then reshaped into an assistant through imitation and, later, practice. Understanding each stage is what lets you tell when to trust the tool and when to check it.
SETUP
What is actually happening when you type into that text box and hit enter?
This is meant as a comprehensive but general-audience introduction to large language models like ChatGPT. The goal is to build mental models for what this tool actually is: it is obviously magical and amazing in some respects, really good at some things, not very good at others, and it has a lot of sharp edges to be aware of. What should you be putting into that text box, and what exactly are these words that get generated back — what are you actually talking to? The plan is to go through the entire pipeline of how this stuff is built, kept accessible to a general audience, and along the way to talk about some of the cognitive and psychological implications of these tools.
SETUP
What does "training data" actually look like before it becomes a language model?
Building something like ChatGPT happens in multiple stages, arranged sequentially. The first is the pretraining stage, and its first step is to download and process the internet. To get a sense of what this looks like in practice, Hugging Face published a data set called FineWeb, documented in detail in a blog post — every major LLM provider (OpenAI, Anthropic, Google, and so on) has some internal equivalent of it.
The goal is a huge quantity of very high-quality documents with very large diversity, because diversity and quality are what give the model its knowledge. Achieving that is complicated and takes multiple stages to do well. One thing worth noting up front: the finished FineWeb data set is only about 44 terabytes of disk space — small enough to nearly fit on a single modern hard drive. Even though the internet is enormous, once you're working with just text and filtering it aggressively, the result isn't actually that large.
Visual not reproducible here
~1:30 — the lecture shows the FineWeb blog post's own diagram of its multi-stage filtering pipeline (URL filtering → text extraction → language filtering → deduplication → PII removal). Worth opening the FineWeb blog post directly to see it.
DEFINITION
What has to be stripped out of a web page before it can teach a model anything?
The starting point for most of this data is Common Crawl, an organization that has been scouring the internet since 2007. As of 2024 it had indexed 2.7 billion web pages: crawlers start from a set of seed pages, follow every link they find, and keep indexing outward. This raw crawl data is then filtered in several stages:
URL filtering removes pages from blocklisted domains — malware sites, spam, marketing, adult content, and so on. Text extraction strips the raw HTML markup, navigation, and CSS down to just the page's actual text; the crawlers save the raw HTML, which is mostly computer code, and only the underlying content is useful. Language filtering uses a classifier to guess each page's language and keeps only pages above a chosen threshold — FineWeb keeps pages that are more than 65% English, which is itself a design decision: filter out most Spanish, and the resulting model will simply be worse at Spanish, because it never saw much of that language. Different companies make different calls here about multilingual coverage. After that comes deduplication and, finally, PII removal — detecting and filtering out things like addresses and Social Security numbers.
Visual not reproducible here
~4:00 — a browser inspector view of a page's raw HTML, to show how much markup surrounds the actual text content.
What survives this pipeline is ordinary-looking web text: an article about tornadoes in 2012, an odd medical trivia page about adrenal glands, and so on — just web pages, filtered down to their text.
Visual not reproducible here
~5:30 — browsing actual FineWeb example documents on the Hugging Face dataset page.
EXAMPLE
Once you have terabytes of clean text, what do you actually do with it?
Concatenating even just the first 200 web pages from this data set already produces a massive wall of raw internet text — and there are millions more pages behind it. This whole mass is what you want a neural network to internalize: to model how this text flows, to learn its patterns. But before any of it can be fed into a neural network, it has to be turned into a specific representation, because the technology underlying these networks expects a one-dimensional sequence of symbols drawn from a finite set. Text, read left to right and top to bottom, is already one-dimensional in that sense — the question is what the symbols should be.
Visual not reproducible here
~7:30 — a zoomed-out view of the concatenated 200-page text sample, showing it as a continuous "tapestry" before zooming back in.
DEFINITION
Why not just feed the model raw 1s and 0s?
Underneath any text file is a UTF-8 encoding: a raw sequence of bits, each one either 0 or 1. That's already a one-dimensional sequence over a finite (in fact minimal) set of symbols, so in principle it could be fed straight into a network. But sequence length is a precious, finite resource for these networks, and a raw bitstream makes for an extremely long sequence over only two symbols. What you actually want is a trade-off: a larger vocabulary of symbols in exchange for a shorter sequence.
Visual not reproducible here
~9:30 — a visualization of the raw bitstring underlying a chunk of text, then the same data re-grouped into byte values, then re-drawn as a strip of unique emoji standing in for each byte value.
A first, naive compression step: group every 8 consecutive bits into a single byte. Since 8 bits have exactly 256 possible on/off combinations, this turns the sequence into one over 256 possible symbols, each 8 times shorter than the bit sequence. It helps to stop thinking of these byte values as numbers and instead as arbitrary unique IDs — you could replace each one with a unique emoji and nothing about the underlying idea would change; there would just be a sequence of 256 possible "emoji."
Production language models push this further still, because a shorter sequence remains valuable even at the cost of a larger vocabulary. The technique used is byte pair encoding (BPE): repeatedly find the pair of consecutive symbols that occurs most often in the data, mint a brand-new symbol for that pair, and rewrite every occurrence of the pair with the new symbol. Iterate as many times as you like, and each iteration shrinks the sequence length while growing the vocabulary size. In practice, a vocabulary of around 100,000 symbols turns out to be a good setting — GPT-4, specifically, uses 100,277 symbols. This whole process, converting raw text into these symbols, is called tokenization, and the symbols themselves are tokens.
EXAMPLE
What does the model actually see when you type a sentence?
A website called Tiktokenizer makes GPT-4's tokenization concrete: type in text (using the "cl100k_base" setting, GPT-4's base tokenizer) and it shows exactly how that text gets chopped into tokens. "Hello world" turns out to be exactly two tokens: hello (token ID 15339) and world (token ID 1917, including the leading space). Small changes shift the tokenization in ways that aren't obvious from the text alone: joining the words as "helloworld" produces different tokens (an "H" token followed by an "L world" token, without the leading "he"); adding a second space between "hello" and "world" inserts a distinct token for that extra space; capitalizing to "Hello World" changes the token count to three instead of two. Tokenization is also case-sensitive throughout.
Visual not reproducible here
~14:30 — the Tiktokenizer interface itself, with each token in the input text highlighted in a different color and its numeric ID shown below. One example line from FineWeb is shown tokenizing into a sequence of length 62.
CLAIM
What is the actual size of what a language model has to learn from?
Run this tokenizer over the entire FineWeb data set, and the 44 terabytes of text becomes a sequence of about 15 trillion tokens. The crucial mental shift is to stop treating any of these token IDs as meaningful numbers — they are just unique IDs, atoms of text with no arithmetic relationship to each other. What's been built so far is nothing more than a very long, one-dimensional sequence of these atoms. The next step is to start training a neural network to model how they follow one another.