Context Window and Why AI "Forgets"
Context window as a fixed token budget per call, why AI appears to 'forget' in long chats, and 3 strategies to preserve critical context.
Context Window: The Bounded View Behind AI's "Forgetting"
In Lesson 5, we learned everything a model processes is measured in tokens. The Context Window is the fixed token budget allocated per model call - explaining why AI appears to "forget" earlier context in long sessions.
1. What a Context Window Is: A Fixed Token Budget Per Call
The Context Window represents the total maximum tokens a model can process in a single request - including: System Prompt + Chat History + New Message + Output to be generated.
Core Rule: This limit is fixed per model (e.g., 128k, 200k tokens) and does not expand dynamically, regardless of conversation length.
The window slides forward - it never grows
As new messages come in, the oldest ones fall out of view. The model only ever sees what's still inside the frame.
System prompt
Always sent - never slides out
Context window (last 5 messages)
Simplified to message slots for clarity - the real budget is measured in tokens, not message count.
2. Why AI "Forgets": Application Truncation, Not Model Amnesia
When conversations exceed the token budget, the application (not the model) must truncate history - retaining recent turns while discarding or summarizing earlier messages:
- The model doesn't experience human memory decay - it simply is no longer provided the truncated tokens in the current request.
- Analogy: The Context Window operates like a moving train window: you only observe what fits in the frame right now; past scenery vanishes completely from view despite existing behind you.
3. Distinguishing Context Windows from Long-Term "Memory"
In a single chat session, tokens pushed beyond the Context Window are permanently lost - unless the product incorporates dedicated persistence layers (RAG or Persistent Memory).
Without explicit engineering, truncated history cannot be retrieved by the model.
4. Application: Why HR Chatbots Self-Contradict & How to Prevent It
Scenario: An HR specialist chats 60+ messages with an AI assistant to build an onboarding flow. Message #3 specified "we only offer permanent and probationary contracts, no seasonal terms". At message #58, HR asks to "draft a 3-month seasonal checklist", and the AI complies without protest because message #3 slid out of the window!
Message #3 falls out of the window by message #58
The same sliding-window mechanic from above, applied to the HR chatbot conversation.
“Our company only has 2 contract types: permanent and probationary - no seasonal contracts.”
out of window by #58“Draft an onboarding checklist for a new hire on a 3-month seasonal contract.”
in viewResult: The AI drafts the checklist as asked - contradicting message #3, not out of carelessness, but because it simply can't see it anymore.