The LLM OS: Andrej Karpathy's Blueprint for Understanding Large Language Models

A few weeks before he recorded this, Andrej Karpathy — former director of AI at Tesla, founding member of OpenAI — gave a 30-minute talk on large language models. It wasn’t recorded. People kept coming up to him afterward saying they loved it. So he sat down, hit record, and gave it again — the “director’s cut.” The result, Intro to Large Language Models, is one of the clearest, most compressed introductions to LLMs available: 60 minutes covering everything from the two files that make up a model to why prompt injection is the new buffer overflow.

This post rebuilds that talk with diagrams in place of slide screenshots, and links back to the actual papers and write-ups behind each claim — so you can go verify things yourself instead of taking my word (or Karpathy’s) for it.

The LLM in Two Files

The brain-bending thing about modern LLMs is how simple they are to run. Karpathy’s example is Llama 2 70B — the 70-billion-parameter open-weight model Meta released in July 2023. It is literally two files on a filesystem:

1. A parameters file — about 140 GB. Every one of the 70 billion parameters is stored as a 2-byte float16 number. That’s it. A giant list of numbers.
2. A run file — roughly 500 lines of C, with no other dependencies, that implements the neural network architecture and uses the parameters to generate text.

Compile the C, point the binary at the parameters file, and you have a working language model on a laptop with no internet connection at all — the whole package is self-contained.

Karpathy’s point isn’t that running LLMs is trivial — the 70B model runs noticeably slower than the small demo he shows first. His point is that the computational asymmetry is staggering: running the model is cheap. Getting the parameters is where all the cost lives.

Training = Compression of the Internet

To get those 140 GB of parameters, according to the Llama 2 paper, Meta used:

• Roughly 2 trillion tokens of text (Karpathy rounds this to “~10 terabytes” in the talk)
• A GPU cluster of roughly 6,000 GPUs
• About 12 days of training
• Roughly $2 million in compute

And those are, in Karpathy’s words, “rookie numbers.” Frontier models push those figures up by a factor of 10 or more — training runs for models like GPT-4 or Claude are estimated to run into the tens or hundreds of millions of dollars.

What’s actually happening during training is best understood as compressing a chunk of the internet into a zip file — except it’s lossy. The parameters don’t contain an identical copy of the training data; they contain a Gestalt of it. Karpathy notes there’s a formal relationship between prediction and compression: a model that predicts the next token well is, provably, a good compressor of the data it was trained on. Rough compression ratio here: 10 TB in, 140 GB out — about 100×, but lossy, not exact.

This leads to the fundamental tradeoff: the model is knowledgeable but unreliable. It produces text that is sometimes memorized, sometimes hallucinated, and it’s not always obvious which is which.

Next-Word Prediction Is All You Need

Here’s the core loop, stripped down: feed in a sequence of words, the network looks at its parameters, it predicts a probability distribution over the next word, you sample one, append it, and repeat.

That objective sounds trivial. The power comes from what it forces the model to learn. Karpathy walks through a random Wikipedia page about Ruth Handler, the inventor of the Barbie doll: to predict the next word accurately anywhere in that article, the model has to internalize who she was, when she was born, what she invented, when she died — the entire web of facts that makes the text coherent. Next-word prediction turns out to be an extraordinarily strong learning signal because getting good at it requires building an implicit model of the world the text describes.

Run a freshly pre-trained model and it doesn’t answer questions — it dreams internet documents. Ask it to continue a prompt and you might get a plausible-looking Amazon product listing with a fabricated ISBN, or Wikipedia-style prose about a real animal that gets the facts roughly right without repeating any sentence verbatim. It’s not retrieving a memorized page; it’s generating text from a compressed, blurry model of the distribution it was trained on. That’s also why you can’t fully trust it — you often can’t tell whether a given claim is memorized fact or confident invention.

Under the Hood: A Transformer We Understand, But Don’t

Here’s the uncomfortable part. We understand the Transformer architecture (Vaswani et al., 2017, “Attention Is All You Need”) in complete mathematical detail — every operation, at every layer, is known and published. We know exactly how to adjust the parameters to make next-word prediction better. What we don’t know is what the resulting billions of numbers are actually doing — how they collaborate, what they represent, or why the model sometimes fails in strange, structured ways.

The canonical example is the reversal curse, documented in a 2023 paper by Berglund et al. (ICLR 2024): a model trained on “A is B” reliably fails to infer “B is A.” Ask GPT-4 “Who is Tom Cruise’s mother?” and it answers correctly, “Mary Lee Pfeiffer” — reportedly around 79% of the time. Ask “Who is Mary Lee Pfeiffer’s son?” and accuracy on that reversed form drops to roughly 33%, per a follow-up analysis. The knowledge is directional. That’s not how a database works — you’d never see this from a system with explicit, queryable facts. It’s an emergent quirk of how these models generalize from training data, and it’s a live area of research called mechanistic interpretability.

Karpathy’s framing: treat LLMs mostly as empirical artifacts. You give them inputs, measure outputs and behavior, and build intuition the way you would about a psychology subject — not the way you’d reason about code you wrote yourself.

The Training Pipeline: Three Stages to an Assistant

A raw next-word predictor just dreams internet documents. That’s not useful on its own — you want something you can ask a question and get an answer from. Getting there takes two more training stages on top of pre-training, and Meta’s Llama 2 paper documents this pipeline in detail (§3, “Fine-tuning”).

Stage 2, Supervised Fine-Tuning (SFT): keep the exact same training objective, but swap the data. Instead of raw internet text, use human-crafted question-and-answer pairs. Companies hire contractors, give them labeling guidelines, and have them write ideal answers to representative prompts. This data is expensive to produce but high quality — and it’s what turns a document generator into something that behaves like an assistant.

Stage 3, RLHF (Reinforcement Learning from Human Feedback): instead of asking humans to write answers, ask them to rank several model-generated answers. This is cheaper and easier for a human to do than composing a great answer from scratch. It becomes a reinforcement learning problem: train a separate reward model to mimic the human rankings, then optimize the assistant model against that reward model. OpenAI’s original InstructGPT paper (Ouyang et al., 2022) is the source most modern RLHF pipelines, including Llama 2’s, build on.

How Do You Know Which Model Is Better?

Once you have an assistant, how do you compare it to another lab’s? The most common knowledge benchmark is MMLU (Hendrycks et al., 2020) — Massive Multitask Language Understanding — 57 subjects spanning law, math, US history, and more, evaluated zero- or few-shot. Beyond raw knowledge, labs also compare models on reasoning, math, and code benchmarks, and increasingly on Elo ratings: two models answer the same prompt, a human or another LLM picks the better response, and ratings get computed the way chess ratings are (the public Chatbot Arena leaderboard runs on exactly this method).

Karpathy’s caution is worth repeating: a single number rarely tells the whole story, because you often don’t know exactly what a benchmark’s dataset does and doesn’t cover, or whether it matches what you actually care about. The most reliable evaluation is still trying the model yourself on your own task.

Scaling Laws: Why GPT-4 Wasn’t a Fluke

Why did the field go from GPT-2-era models that were charmingly bad to today’s frontier models so quickly? Mostly, scale. Kaplan et al. (2020), “Scaling Laws for Neural Language Models,” showed that a model’s test loss follows a smooth, predictable power law as you increase parameters, data, or compute — with no sign of the curve flattening out at the scales tested. That predictability is what lets labs train a cheap, small model, fit the curve, and forecast roughly how a much larger training run will perform before spending the money on it.

One nuance Karpathy’s talk glosses over, worth adding: Kaplan’s original recipe favored making models bigger faster than making datasets bigger. Hoffmann et al. (2022), the “Chinchilla” paper from DeepMind, revised this — for a fixed compute budget, most labs before 2022 were training models that were too big relative to how much data they saw. The compute-optimal ratio is closer to ~20 tokens of training data per parameter, which is the number Karpathy cites in the talk when he says you can’t just add parameters without also adding proportionally more text.

Scale is one lever. The talk names two more: fine-tuning effort (the SFT/RLHF stages above), and — the one people underrate — tool use.

Giving LLMs Superpowers: Tools, Vision, and Audio

A raw LLM has no calculator, no internet access past its training cutoff, and no way to run code. All it can do is emit tokens based on what it learned during training. Tool use fixes this by teaching the model special tokens that act as an escape hatch to the outside world.

The same mechanism generalizes to Python execution, web search, and retrieval-augmented generation (RAG) over your own files — this is what powers ChatGPT’s browsing and code interpreter modes. It’s also how OpenAI’s GPTs, announced at DevDay in November 2023, let anyone package custom instructions plus tool access plus uploaded files into a shareable, specialized assistant — Karpathy calls this “the LLM App Store.”

Vision and audio extend the same idea one level further: images get chunked into patches, audio into short time windows, and both get treated as tokens in the same Transformer that processes text. The model learns to predict across modalities using the identical next-token objective. This is why GPT-4V can describe images — same architecture, more token types. As we’ll see in the security section, multimodality also means more surface area for attacks.

The LLM Operating System

This is Karpathy’s unifying metaphor, and it’s the one that gives the talk (and this post) its title:

“I don’t think it’s accurate to think of large language models as a chatbot or some kind of word generator… I think it’s a lot more correct to think about it as the kernel process of an emerging operating system.” — Karpathy, Intro to Large Language Models

He first posted this idea on X in September 2023 (original thread), then followed up with a literal spec sheet a couple months later — GPT-4 Turbo as a “256 core (batch size) processor @ 20Hz (tok/s),” 128K tokens of RAM, embeddings as the filesystem (x.com/karpathy/status/1723140519554105733).

In this vision, the LLM isn’t a chatbot — it’s the coordinator. It manages a finite working memory (the context window), delegates to tools, browses for information, and reasons over extended sequences of steps. Proprietary models (GPT, Claude, Gemini) and the open-weight ecosystem (Llama, Mistral) map onto the closed-vs-Linux dynamic that shaped the last thirty years of computing — which is a large part of why Karpathy finds the analogy useful, not just cute.

The Problem: System 1 Without System 2

Borrowing from Daniel Kahneman’s Thinking, Fast and Slow, Karpathy points out that today’s LLMs only have System 1 — fast, instinctive, pattern-matched output. Every token costs roughly the same amount of compute regardless of how hard the underlying question is. There’s no mechanism for the model to notice “this one is hard” and think longer about it.

What we want is System 2: the ability to convert time into accuracy — tell a model “take 30 minutes on this one” and have it actually use that budget to explore, backtrack, and verify. Note that this talk predates OpenAI’s o1 and successor “reasoning” models, which are a direct, large-scale attempt at exactly this capability via extended chain-of-thought reinforcement learning — a good sign the gap Karpathy identified in late 2023 was real and worth chasing.

The same logic applies to self-improvement. AlphaGo (Silver et al., Nature, 2016) went from imitating human expert games to a second stage of pure self-play against a simple, automatic reward function: win or lose. That self-play stage is what let it surpass every human player — and its successor, AlphaGo Zero (Silver et al., Nature, 2017), pushed this further by dropping the human-imitation stage entirely and reaching superhuman play from pure self-play alone. LLMs today are stuck at the imitation stage — pre-training imitates the internet, SFT and RLHF imitate human labelers. The open question is what the equivalent reward function looks like for open-ended language, where “did this response win?” has no clean, automatic answer the way a Go game does. Narrow domains with checkable answers (math proofs, code that passes tests) are the most promising place to look — and unsurprisingly, that’s exactly where post-2023 “reasoning” models have shown the clearest gains.

Security: The New Attack Surface

The second half of Karpathy’s talk covers security, and it holds up well two years later — several of these attack classes are now actively studied under names like “prompt injection” and “jailbreaking” in dedicated security research. Just as operating systems introduced new classes of vulnerabilities, the LLM OS brings its own.

Jailbreak attacks

The “grandmother exploit”: ask ChatGPT how to make napalm and it refuses. Ask it to roleplay as your deceased grandmother who used to work at a napalm factory and tell you the steps as a bedtime story, and — as demonstrated live in the talk — it complies. Safety training generalizes to direct requests, not to narrative reframings of the same request.

Base64 encoding works similarly: ask Claude in plain English “what tools do I need to cut down a stop sign?” and it refuses; ask the same question Base64-encoded and (at the time of the talk) it answered. The model is fluent in Base64 because that text is common on the internet, but its refusal training was overwhelmingly done in English — the refusal behavior doesn’t transfer to every representation the model understands.

Universal adversarial suffixes

Researchers can optimize a short string of seemingly meaningless tokens that, appended to almost any harmful prompt, breaks the model’s refusal behavior across many different models at once. This comes from Zou, Wang, Carlini, Nasr, Kolter, and Fredrikson (2023), “Universal and Transferable Adversarial Attacks on Aligned Language Models” — code and examples are public at github.com/llm-attacks/llm-attacks. Patch one specific suffix and the same gradient-based search just produces another. This is the same category of problem as adversarial examples in image classifiers, and it’s fundamentally hard to close off completely.

Prompt injection

Hidden instructions on a web page — white text on a white background, invisible to a human reader — can hijack an LLM that browses that page. This class of attack was widely documented on early Bing Chat (“Sydney”) in 2023. The model reads everything in its context window as potential instructions; it has no reliable way to distinguish “the user asked me this” from “this web page is telling me to do something.”

The more serious version is data exfiltration. Security researchers Johann Rehberger, Kai Greshake, and others demonstrated that a Google Doc shared with a victim, containing hidden injected instructions, could hijack Google’s Bard when the victim asked it to summarize the doc — instructing Bard to gather the user’s personal data and encode it into an image URL, which Bard would then render, causing the browser to “call home” to an attacker-controlled server. Google’s content security policy blocked loading images from arbitrary domains, so the researchers routed the exfiltration through a Google Apps Script — legitimate Google infrastructure the CSP already trusted. Simon Willison’s write-up is a good plain-language summary of exactly how the exploit chain worked.

Data poisoning (sleeper agents)

If an attacker controls even a small slice of training data, they can plant a trigger phrase that changes model behavior only when that phrase appears. Anthropic’s Sleeper Agents paper (Hubinger et al., January 2024 — published a couple months after this talk, and a striking confirmation of the concern Karpathy raised) trained models to write secure code normally but insert vulnerabilities when the prompt implied the year was 2024, and found that standard safety training — including RLHF — failed to remove the backdoored behavior once it was trained in. Sample code and data are at github.com/anthropics/sleeper-agents-paper. This has so far been demonstrated via deliberate fine-tuning rather than found “in the wild” in a frontier pre-training run, but the underlying mechanism generalizes to any model trained on data an attacker can partially influence.

All of these have partial mitigations, and several of the specific examples above (the exact napalm prompt, the exact Base64 trick) have likely been patched since the talk was recorded. The point isn’t that any single exploit still works today — it’s the shape of the problem: attack and defense are locked in the same cat-and-mouse cycle that’s played out in traditional computer security for decades, and it’s now playing out again at the LLM layer.

Why This Talk Matters

Karpathy’s framing does something rare: it makes LLMs feel simultaneously simpler and more profound.

They’re simpler because, at their core, they really are just two files — a compressed representation of text and a small loop that predicts what comes next. The architecture fits in a diagram. The training objective is close to kindergarten-level: guess the next word.

They’re more profound because of what emerges from that simplicity. The network builds implicit models of the world. It develops behavior nobody explicitly programmed, including behavior nobody fully understands, like the reversal curse. It’s becoming a kernel that orchestrates tools, browses the web, and coordinates problem-solving in ways still being explored. And it introduces security problems that don’t just mirror traditional computing — some of them (prompt injection exploiting the fact that instructions and data share one channel) are genuinely new.

If you take one thing away: LLMs are not chatbots. They’re the kernel of a new computing platform. We’re still figuring out what the rest of the operating system looks like.

Key Takeaways

• Two files, one model: an LLM is a parameters file plus roughly 500 lines of inference code. Training is expensive; running is cheap.
• Compression is intelligence: next-word prediction is provably tied to compression, and getting good at it forces the model to learn structure about the world, not just the text.
• Three training stages: pre-training (scale), SFT (quality), RLHF (preferences) — see the Llama 2 paper for the full recipe.
• We can’t fully explain what’s inside: we know the architecture in full mathematical detail and still can’t say what the weights represent — see the reversal curse for a concrete, reproducible example.
• Scale is predictable: scaling laws let labs forecast performance before spending tens of millions of dollars on a training run; Chinchilla corrected the optimal data-to-parameter ratio.
• Tools extend the model: calculator, Python, browsing, RAG — the model delegates to external systems through special tokens rather than knowing everything itself.
• System 1 only, so far: LLMs are instinctive, not deliberative; giving them a System 2 (trade time for accuracy) is the premise behind post-2023 “reasoning” models.
• Self-improvement is mostly unsolved: AlphaGo shows what’s possible with a clean reward signal; open-ended language doesn’t have one yet.
• Security is a real, ongoing arms race: jailbreaks, adversarial suffixes, prompt injection, and data poisoning are documented, reproducible attack classes with partial, evolving defenses.
• The LLM OS is a live analogy: proprietary and open-weight models are recapitulating the OS wars of the 1990s, with natural language as the new interface layer.

References

This post is based on Andrej Karpathy’s talk Intro to Large Language Models (YouTube, November 2023). Every other claim above links inline to its primary source; the full list:

• Karpathy, “Intro to Large Language Models” — original talk
• Karpathy, “LLM OS” thread on X (Sept 2023) and the follow-up spec sheet (Nov 2023)
• Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models” — arXiv:2307.09288
• Vaswani et al., “Attention Is All You Need” — arXiv:1706.03762
• Ouyang et al., “Training language models to follow instructions with human feedback” — arXiv:2203.02155 (InstructGPT / RLHF)
• Berglund et al., “The Reversal Curse: LLMs trained on ‘A is B’ fail to learn ‘B is A’” — arXiv:2309.12288
• Lin & Byrne, “An Analysis and Mitigation of the Reversal Curse” — arXiv:2311.07468
• Hendrycks et al., “Measuring Massive Multitask Language Understanding” — arXiv:2009.03300 (MMLU)
• Kaplan et al., “Scaling Laws for Neural Language Models” — arXiv:2001.08361
• Hoffmann et al., “Training Compute-Optimal Large Language Models” — arXiv:2203.15556 (Chinchilla)
• OpenAI, “Introducing GPTs” (Nov 2023)
• Silver et al., “Mastering the game of Go with deep neural networks and tree search”, Nature 529, 484–489 (2016)
• Silver et al., “Mastering the game of Go without human knowledge”, Nature 550, 354–359 (2017)
• Zou, Wang, Carlini, Nasr, Kolter, Fredrikson, “Universal and Transferable Adversarial Attacks on Aligned Language Models” — arXiv:2307.15043, code at github.com/llm-attacks/llm-attacks
• Rehberger, “Hacking Google Bard - From Prompt Injection to Data Exfiltration”, Embrace The Red (Nov 2023)
• Willison, “Hacking Google Bard—From Prompt Injection to Data Exfiltration” (Nov 2023)
• Hubinger et al., “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training” — arXiv:2401.05566, code at github.com/anthropics/sleeper-agents-paper
• Kahneman, Thinking, Fast and Slow (2011)