The Context Window Paradox: To Get More From Your AI, Give It Less
When I started working with LLMs that have massive, million-token context windows, I fell into a trap. I figured that more context meant less work for me. I could just dump a project's entire history into the chat and trust the model to keep everything straight. It seemed logical. But as I found out while co-authoring a technical presentation with an AI, this assumption is not just wrong; it’s backward. My attempts to treat the AI like a partner with a perfect memory led to subtle but persistent quality issues. It was only when I started being deliberately restrictive with the context that the results genuinely improved. This led me to a counterintuitive conclusion: to get the most out of a large context window, you have to actively manage and constrain it.
My Setup: Crafting a Presentation with an LLM
For this experiment, I set out to create a new slide deck, "Introduction to Agents," using Gemini 2.5 Pro as my authoring partner. My methodology is rooted in agile principles—starting with a high-level outline and iteratively building out the content. The goal is to offload the heavy lifting of drafting, allowing me to focus on the core narrative and technical accuracy.
I began with a highly structured initial prompt. This approach, refined from my previous work on pair-authoring with AI, was designed to establish clear rules of engagement for our collaboration from the start.
Hello,
I need your help preparing a presentation. Here is the context and the way I would like to work with you.
1. The Goal:
My objective is to create a [e.g., 15-minute] presentation for [e.g., the AI For Software Delivery festival]. The final output should be a complete script, ready to be put onto slides.
2. Source Materials:
Here are the links and documents you should use...
3. Our Workflow (How We Will Work):
We will follow a structured, step-by-step process. I want to approve each step before we move to the next.
Step 1: High-Level Plan...
Step 2: Section Planning...
Step 3: Slide-by-Slide Drafting...
Step 4: Finalizing Sections...
4. Content Structure (What I Want for Each Slide):
A) On-Slide Text: ... minimalist and concise.
B) Accompanying Speech: ... comprehensive and explain the concepts...
5. Tone and Style:
... Professional and Neutral ... Not Promotional...
Your First Task:
Please review the source materials I provided and propose a high-level outline for the presentation.
This prompt was my attempt to front-load the process with as much structure as possible.
The Subtle Failure of a Well-Laid Plan
Even with this detailed plan, I began noticing a subtle degradation in quality as our conversation grew longer. The failure wasn't catastrophic, but it manifested as a slow accumulation of small inconsistencies and extra work.
My chat history filled up with micro-corrections. At one point, the AI generated speech that claimed I worked at Thoughtworks on a project I had explicitly described from a previous job at Mimecast. The fix was a simple prompt:
fix for B) Accompanying Speech (Revised):
###
For example, I previously worked on a phishing detection team here at Thoughtworks.
@@@
I worked in Mimecast, not in Thoughtworks.
###
Other times, I had to refine the AI's understanding of the scope: We analyzed an email, not We'd analyze a URL. While individually minor, the frequency of these small edits increased as the context grew. The AI's attention seemed to drift, anchored more by recent turns in the conversation than by the foundational context we had established. The presentation began to feel less like a single, coherent story and more like a collection of loosely related talking points that required my constant, vigilant correction.
Principles That Actually Work
This experience led me to refine my approach, focusing on two principles that proved to be more effective.
1. Enforce a Disciplined Pace (Decompose and Verify)
My initial prompt already specified a slide-by-slide workflow. However, I discovered that I had to actively enforce this pace. At times, the AI would try to rush ahead, offering to generate multiple slides or an entire section at once. I learned to be firm, using prompts like always wait my approve and before we will move to section 2, combine all approved slides and speeches for section 1.
The key insight here was that it wasn't enough to state the plan; I had to actively rein in the AI's eagerness to generate. This forced discipline kept each unit of work small and verifiable. By confirming the correctness of each individual slide before moving on, I ensured the foundation for the next one was solid.
2. Proactively Distill the Context
This was the most impactful change. Acknowledging the limitations of the AI's attention, I began a practice of "proactive context distillation." After completing and approving a section, I would instruct the AI to summarize what we had just created.
"Excellent. Please summarize the content of the two slides we just created for Section 1 into a concise overview."
This technique serves two purposes. First, it acts as a "context refresher" for the AI, collapsing the detailed turn-by-turn history into a dense, high-signal summary. As explained by IBM, a context window isn't a perfect memory; a clean summary places the most important information front and center. Second, it keeps the most relevant information active, preventing the model's attention from being diluted by the noise of a long conversation.
An Interesting Realization: It’s For Me, Not Just the AI
Initially, I viewed context distillation as a trick to manage the AI. But a valuable realization was how much it improved my own thinking. The act of requesting and reviewing these summaries forced me to constantly re-evaluate the presentation's narrative arc. Was Section 2 a logical continuation of Section 1? Did the key messages connect?
This process mirrors what we know about human cognition. Our own working memory is limited. By creating these summaries, I wasn't just helping the AI; I was building a better mental model for myself. The distillation process became a forcing function for clarity, ensuring that I, the human author, remained in firm control of the narrative.
Closing the Loop: Refining the Process with the AI
After the presentation script was complete, I tried one final experiment. I asked the AI to become my process consultant.
"Now, analyze our chat for understanding our pattern of communication and work under the presentation. After that, check my initial prompt... Do we need to improve the original prompt or does it look good enough?"
The AI analyzed our entire interaction and proposed an improved V2 of my starting prompt. The new version added subtle but important clarifications, such as describing the process as iterative and flexible. This idea of having the tool refine its own operating instructions isn't new to my workflow; I'd explored a similar pattern when getting Copilot to improve its own rules. Applying it to a conversational authoring process, however, felt like a significant step forward. It’s a powerful demonstration of using the tool not just to execute a task, but to reflect on and improve the very process of collaboration itself.
The Central Paradox of AI Collaboration
This leads to the central, counterintuitive truth I discovered: To effectively leverage an AI's massive context window for a complex project, you must actively manage and constrain the context you provide it.
This paradox exists because a large context window is not a perfect memory. It is a probabilistic field of attention. Without deliberate guidance, the AI can get lost in the details, overweighting recent conversation turns and losing the foundational plot.
Effective use, therefore, requires more than just good prompting. It requires:
Active Context Management: You must act as the session manager.
Enforced Discipline: A structured, step-by-step pace that you actively maintain.
Strategic Summarization: Periodically compressing the state of the project to maintain focus.
Conclusion: From Prompter to Director
Treating a generative AI as a simple instruction-follower for complex work is a path to mediocre results. The initial promise of "thinking less" is a mirage. Instead, these tools invite us to think differently.
The real leverage comes from shifting our role from a mere prompter to that of a director. Our job is not just to provide instructions but to manage the state, curate the context, and guide the narrative. By enforcing a disciplined pace, proactively distilling the context, and even using the AI to refine our methods, we don't just mitigate the tool's weaknesses; we sharpen our own thinking. The AI provides the immense power of generation, but human oversight provides the coherence and intent that turns raw output into a valuable, finished product. This collaborative dance is the future of knowledge work.