← back to sections

Context: what an LLM actually sees

agents-101 · 2026-05-25 · the messy version

Last post we ended with a clean line: LLMs live on context. System prompt, your messages, the assistant's messages, all stitched together.

That definition is correct. It is also a bit too clean. It hides almost everything that starts to matter the moment you build with these things instead of just chatting with them.

So this one is the messy version.

We will look at what context literally looks like under the hood, what it is made of (this unit is called "token"), why every model has a hard limit on it, and the most non-obvious thing about LLM chat: every time you press enter, you are also re-paying for the entire conversation so far.

What context actually looks like

Context is not a vibe or a feeling the LLM has about you. It is a literal list of messages.

If you peek behind the curtain of ChatGPT or Claude.ai and look at what actually gets shipped off to the model, it looks roughly like this: 

[
  { "role": "system",    "content": "You are a helpful assistant." },
  { "role": "user",      "content": "hi" },
  { "role": "assistant", "content": "Hello! how can I help?" },
  { "role": "user",      "content": "what is the capital of France?" }
]

That is it. A JSON-shaped list. The system message sits at the top, then the back and forth.

Every time you press enter, this list is rebuilt with your new message tacked on the end, and the whole thing is sent off to the model. The model reads the list like a script and writes the next line which is the next assistant message. That reply gets appended to the list. Repeat.

The messages array. System prompt on top, then user-assistant back and forth stacking up over the session.
Context is just a stack of messages, growing with each turn. The whole stack goes to the model on every call.

Once you see context as a list, a lot of things click. The LLM is not "in a conversation with you" in any felt sense. It is reading a transcript. Every. Single. Time.

Below messages: tokens

There is a layer below messages. The model does not see your text as words. It sees it as tokens.

A token is a chunk of text. Sometimes a whole word, sometimes a part of a word, sometimes a single character or a punctuation mark. Rough rule of thumb for English: 1 token is about ¾ of a word.

You can try this yourself on OpenAI's tokenizer page, it is a bit addictive to play with.

Why care about tokens? Because tokens are the unit of everything that follows:

For now, the only thing to remember is, when I say "context is too long" or "context size," I really mean it in tokens. Not words, not messages, tokens.

The context window

Every model has a context window. That is the maximum number of tokens it can hold in one shot.

Picture a whiteboard with a fixed size. You can keep writing on it, but once it is full, something has to be wiped off to make room for the new stuff.

A whiteboard slowly filling up with messages. Once full, older messages need to be wiped to make room.
The context window is a whiteboard with a fixed size. Everything has to fit, or something has to be wiped to make room. Easiest is to remove the older messages like a sliding window. Memory compaction will be discussed soon!

Some rough numbers as of 2026 (these change pretty often tbh):

These sound huge. A full novel is roughly 80K-100K tokens. So Claude can fit an entire novel into one go.

But the catch is that the window has to hold everything at once. Your system prompt, every user message you have sent in this session, every reply the model has given you, AND the response it is about to generate. All of it counted against the same window.

For long chats with a lot of back and forth, you fill it up faster than you would guess.

The hidden cost? You re-send everything, every time

There exists KV Cache's etc. which providers like Claude, OpenAI and Gemini implement to cache the attention vectors, which is what gets processed to form the output from this input, but the cost is reduced by around 50%. Good, but the cost graph still rises. The fact that we re-send the entire convo everytime still remains

Remember when I said LLMs are stateless? That every new request is a fresh stranger?

Well, here is the bill for that statelessness.

Because the model has no memory of your conversation, the only way to keep the convo going is to send the whole transcript back, every single turn.

By the time you are 50 turns deep in a chat, every new message you send is shipping fifty turns of history over the wire, getting re-processed by the model, and getting billed in tokens.

The cost growth is not linear. It is roughly quadratic in the length of the conversation. A single 100-turn chat is way more expensive than two separate 50-turn chats that cover the same stuff.

ChatGPT and Claude.ai hide this from you. They handle the truncation, they decide when to summarise old turns, they eat the bill. The moment you start building directly on top of the API, that bill is on you.

More context is not always better

When you first see "200K tokens," the instinct is to stuff it. Whole codebases. Full documentation sites. Every chat the user has ever had with you. Don't do that.

As context grows, a few things happen. The cost goes up (obvious from above). The model gets slower, because each token in the input adds work on every step of generation. And, the interesting one, quality drops in weird places.

There is a thing called "lost in the middle" that has been studied a fair bit. When you cram a lot into the prompt, the model pays disproportionate attention to the start and the end, and disproportionately little to whatever is buried in the middle. So you can shove a 50-page PDF into the context, ask a question whose answer lives on page 24, and the model will fumble it. Not because it cannot read page 24. Because page 24 got drowned out.

A rough curve showing model attention high at the start and end of a long context, dipping in the middle.
Models pay more attention to the start and end of context, less to the middle. Stuff buried in the middle gets fumbled.

I won't pretend to fully understand the math of why this happens (ML is still not my forte), but the empirical pattern is real and you should treat it as a given when you build.

The mental model I use: context is more like RAM than a hard drive. It is finite, it is fast, and what is in it actively shapes what the model can think about right now. You do not want it cluttered with junk.

So how do you manage context?

Once you accept that context is a finite resource you have to be deliberate about, a few options open up:

  1. Trim. Just drop old messages once they stop being relevant.
  2. Summarise. Squash the first 30 turns into a short summary and keep that in place of the raw turns.
  3. Retrieve. Don't put the whole knowledge base in the prompt. Pull only the chunks that are relevant to the current question. This is what people mean when they say "RAG."
  4. Externalise memory. Save important facts about the user somewhere outside the context (a database, a file, whatever) and pull them back in only when needed. This is the "memory" feature ChatGPT and Claude.ai offer.

We will cover each of these in its own post later. For now the point is just managing context is its own skill. Some people are calling it "context engineering" these days. Whatever you call it, you cannot ignore it once you go past toy examples.

How to perceive context


Next post, finally, we open the OpenAI SDK and chat with an LLM from Python. Once you have seen context as a literal messages array, the SDK is almost boring. You build the list, you send it, you get a reply back, you append it to the list, you send it again. That is the loop. After that, we make the model actually do things, and that is where "AI Agents" starts to mean something.