The hard limit behind every AI conversation. Why it is a constraint rather than a bug, and why a bigger window is not uniformly better.
Every conversation with an AI has a hard limit, a point where it can no longer see what came before. Most people have already hit it without noticing. The repetition, the contradictions, the forgotten instructions are not bugs. They are a fundamental constraint of how these models work.
A context window is the maximum amount of text, input and output combined, that a model can process in a single interaction. Everything outside it is invisible to the model. Everything inside is what it actually works with.
The desk analogy makes it concrete. You have a fixed surface area. You can only spread out so many documents. When the desk is full and you add one more, something falls off the other end. The model cannot look at what is on the floor.
Context is measured in tokens, roughly three or four characters each. A business email is about 200 tokens. A full book runs into the hundreds of thousands. Sizes differ sharply by model: Claude sits at 200,000 tokens across most models with experimental features at a million, GPT-4 at 128,000, and some smaller models at only 4,000 to 8,000.
Everything competes for that same fixed resource: the system prompt, the conversation history, documents retrieved through RAG, and your current message. As you approach the limit, models behave differently. Some truncate the oldest content, some summarise, some just stop.
The assumption worth correcting: a bigger window does not mean equal attention across all of it. Research shows models struggle with information buried in the middle of a long context. Beginning and end get the most attention. This is the documented lost in the middle problem. A large window is useful, but not uniformly reliable, and you pay for every token you push in.
| Model or concept | Size or meaning | What it means for you |
|---|---|---|
| Token | Roughly three to four characters | Not a word, not a letter. The unit everything is measured in. |
| Business email | About 200 tokens | A useful reference point for estimating. |
| Claude | 200,000 tokens in most models, experimental up to 1 million | Handles long documents comfortably. |
| GPT-4 | 128,000 tokens | Still large, noticeably smaller than Claude. |
| Smaller models | 4,000 to 8,000 tokens | A 100-page legal document will not fit. You have to chunk it. |
| What fills it | System prompt, history, RAG documents, your message | All of it competes for the same fixed space. |
| Lost in the middle | Attention drops for content buried mid-context | Put the important parts at the start or the end. |
A context window is the maximum amount of text, input and output combined, that an AI model can process in a single interaction. Everything outside the window is invisible to the model. Everything inside is what it actually works with.
Because it left the window. The conversation filled up the available space and the oldest parts fell off the edge. That is not a bug, it is a fundamental constraint of how these models work.
A token is roughly three or four characters, not exactly a word and not exactly a letter. Context windows are measured in tokens. A business email is around 200 tokens, a full book runs into the hundreds of thousands.
No. Research shows models often struggle with information buried in the middle of a very long context, while the beginning and end get the most attention. This is the documented lost in the middle problem. You also pay for every token you push in.
No. Context windows do not persist across sessions. Every new conversation starts fresh and the window resets. Long-term memory is a separate architecture problem that the context window does not handle.
The system prompt, the conversation history, any documents retrieved through RAG, and your current message. All of it competes for the same fixed resource, which is why long conversations with large retrieved documents hit the limit fastest.
No manual subtitle track is available for this video. This page therefore uses a checked chapter summary instead of publishing auto-generated captions as a transcript.
Every conversation with an AI has a hard limit, a point where it can no longer see what came before. Most people have already hit it without knowing. This is part six of the series.
Why AI loses track in long conversations, repeats itself, contradicts itself and forgets instructions. Not a bug, a fundamental constraint of how these models work, and understanding it changes how you build with them.
The definition. A context window is the maximum amount of text, input and output combined, that a model can process in a single interaction. Outside the window is invisible, inside is what it works with.
The desk analogy. A fixed surface area, only so many documents at once. When the desk is full and you add another, something falls off the other end. The model cannot look at what is on the floor.
Tokens as the unit. Roughly three or four characters each, not exactly a word and not exactly a letter. A business email is about 200 tokens, a full book hundreds of thousands.
Sizes across models. Claude at 200,000 tokens in most models with experimental features reaching a million, GPT-4 at 128,000, some smaller models at only 4,000 to 8,000. For a 100-page legal document that difference decides whether you can process it at all or have to chunk it.
What competes for the space: system prompt, conversation history, documents retrieved through RAG and your current message. As the limit approaches, models behave differently, some truncating the oldest content, some summarising, some simply stopping.
The correction that matters. A bigger window does not mean equal attention throughout. Research shows models struggle with information buried in the middle, while the beginning and end get the most attention. This is the documented lost in the middle problem.
Two further limits. A large window does not guarantee correct use of everything in it, and you pay for every token you push in. Context windows also do not persist across sessions, every conversation starts fresh. Long-term memory is a separate architecture problem.
The summary: the AI forgot because it left the window, the desk filled up and the conversation fell off the edge. Now you can design around the constraint. Next up, what happens when one agent is not enough.
I work 1:1 with freelancers, consultants, coaches and small teams on practical AI workflows and automation. First call is free, 15 minutes, no pitch.
Book a free intro call