HOW THE NUMBERS GOT THIS BIG
The original GPT-3 context limit of 2,048 tokens wasn't an architectural law — it was a training and inference cost decision. Attention is quadratic: doubling the context length quadruples the compute required to process it. The jump from 4K to 8K to 32K to 128K happened as hardware improved, sparse attention mechanisms matured, and companies decided that the capability gains justified the infrastructure cost. The jump to 1M required more fundamental changes.
Anthropic's path to Claude's 1M token window relied on a combination of techniques: efficient attention implementations, context caching that avoids re-processing stable prefixes on every request, and careful management of KV cache memory. The practical result is that a 1M token request doesn't cost 500x more than a 2K request — though it still costs substantially more, which is why most production deployments don't use the full window on every call.
Gemini 1.5 Pro's 1M window (extended to 2M in later versions) used a different architecture — a Mixture of Experts model where routing decisions allow the network to process long sequences without proportionally scaling compute at every layer. Both approaches represent genuine engineering solutions to what looked for several years like a hard physical limit. The fact that two different architectures independently solved the long-context problem within months of each other suggests the problem was always more tractable than it appeared.
WHAT FITS INSIDE A MILLION TOKENS
A million tokens is roughly 750,000 words of English text — about ten full-length novels, or the complete works of Shakespeare three times over. In code terms, it accommodates a medium-sized monorepo: the entire React source code fits comfortably, with room for its test suite and documentation. In document terms, it holds a multi-year legal case file, a corporate knowledge base, or a year's worth of meeting transcripts.
For AI application design, these numbers translate to use cases that were previously impossible or impractical. Analysing a codebase for security vulnerabilities across all files simultaneously, rather than file by file. Summarising an entire legal discovery document set for a specific question. Reviewing a year of customer support tickets to identify patterns without chunking and losing cross-ticket context. Translating a large documentation corpus while maintaining stylistic consistency across the entire document set.
The qualitative shift is elimination of the chunking problem. RAG systems that break documents into chunks introduce a retrieval step that can miss context when the relevant information spans chunk boundaries, or when the question requires synthesising information from multiple locations in the corpus. With enough context to fit the entire corpus directly, the retrieval step becomes optional — and retrieval errors become irrelevant. That's a meaningful architectural simplification for a broad class of applications.
THE NEEDLE-IN-A-HAYSTACK PROBLEM
Benchmarks revealed a counterintuitive problem early in the large-context era: models didn't actually use all the context they could accept. The "needle in a haystack" tests — where a specific piece of information is planted at different positions in a long document and the model is asked to retrieve it — showed that models performed significantly worse when relevant information was placed in the middle of very long contexts compared to near the beginning or end.
This "lost in the middle" phenomenon has partially improved with newer model generations. Claude and Gemini show substantially better retrieval performance across positions than their predecessors, and the training improvements that delivered this are reflected in production performance. But "better" isn't "perfect." For tasks requiring precise recall of specific details from a very long context, models can still miss or conflate information buried deep in a dense document, especially when similar-looking text appears at multiple points.
The practical implication: long context is more reliable for reasoning and synthesis tasks than for precise factual retrieval. Asking a model to read a 200-page report and identify the main strategic risks it describes is a task where large context excels. Asking it to find the exact figure cited in Table 7, paragraph 3 is a task where the chunked retrieval approach with an exact-match fallback may still outperform raw stuffing. Knowing which kind of task you're building for determines the right architecture.
RETRIEVAL VERSUS STUFFING: THE ARCHITECTURE QUESTION
The emergence of reliable large context windows didn't kill RAG — it forced a more honest accounting of what RAG is actually good at. The argument for RAG was always partly that context windows were too small. Now that they're not, the remaining arguments for RAG are clearer: cost control, dynamic knowledge (documents added after context is loaded), precise retrieval of specific spans, and the ability to cite exactly which document a piece of information came from.
The argument for large context stuffing is equally clear: simpler architecture, no retrieval errors, better cross-document reasoning, and no chunking artefacts. For datasets that are stable (updated infrequently), bounded (fit within the window), and used for synthesis rather than lookup, stuffing the full corpus is often the right call. The infrastructure cost of context caching makes repeated large-context requests substantially cheaper than they would otherwise be, which shifts the economics toward stuffing for stable corpora.
The emerging consensus among teams that have run both approaches in production: hybrid architectures often win. Use semantic retrieval to identify the relevant sections of a large corpus, then stuff those sections — not the full corpus — into context. This gives you the precision of retrieval and the cross-passage reasoning of direct context without paying for unused tokens. The right boundary between retrieval and stuffing is a product decision, not a technical one, and it should be tuned based on your actual use case rather than adopted as a default.
WHERE THE CEILING ACTUALLY IS
The context window number is a ceiling, not a quality guarantee. A 1M token window means the model can technically accept 1M tokens — not that it performs equally well at 10K and 900K tokens, not that inference at full context is fast, and not that it's cost-effective for every use case. Latency increases with context length: a full 1M token request on Claude takes meaningfully longer to process than a 50K request, and the time-to-first-token on very long contexts can be uncomfortable for interactive applications.
Cost is the less-discussed ceiling. At current pricing, filling a 1M token context on Claude Opus costs approximately $5 in input tokens alone — before any output is generated. For use cases requiring many requests against a large corpus, this adds up. Context caching reduces the cost for repeated stable prefixes substantially (cached tokens are an order of magnitude cheaper than non-cached), but the base cost of very large contexts remains a meaningful constraint on where large-context approaches are economically viable versus where a traditional database or search approach is preferable.
The honest picture for 2026: large context windows have genuinely expanded what AI applications can do, eliminated entire categories of architectural complexity, and made tasks that required careful document management trivially simple. They haven't replaced the need to think carefully about what information a model actually needs for a given task. The teams building the best AI applications aren't the ones stuffing the most tokens — they're the ones who understand the task well enough to give the model exactly what it needs, whether that's 5,000 tokens or 500,000.