RAG Is Dead: The Rise of Context Engineering

26 June 2026 · Bedrock AI

Retrieval-Augmented Generation was the defining AI architecture pattern of 2023 and 2024. By 2026, most practitioners who built those systems have moved on. Not because retrieval is useless, but because it was only ever one piece of a larger problem. That larger problem has a name: context engineering. This article explains the shift, what context engineering actually involves, and what AI architects now spend their time building.

What Is Context Engineering?

Context engineering is the discipline of designing systems that give AI models the right information at the right time. It encompasses everything that enters an LLM’s context window: retrieved documents, memory, conversation history, user identity, tool definitions, schemas, policies, and instructions. RAG is one mechanism within that system. Context engineering is the whole discipline.

Andrej Karpathy gave the concept its clearest public articulation in mid-2025, describing it as “the delicate art and science of filling the context window with just the right information.” The term has since been adopted across the practitioner community as the most accurate description of what building production AI actually involves.

Why Did RAG Fall Short?

RAG solved a real problem: large language models hallucinate when they lack grounding data, so embedding documents in a vector store and retrieving them at inference time gives the model something accurate to work with. In 2023, that was a genuine step forward.

The limitations became visible as deployments scaled.

Retrieval is not understanding. A vector similarity search returns chunks most semantically similar to the query. It does not understand what the model needs to reason well. Relevant context is not always retrievable by similarity; sometimes what matters is a schema, a constraint, a user role, or a prior decision made three conversation turns ago.

Context failures are the leading cause of AI breakdowns in production. Research into enterprise AI deployments has consistently identified missing, stale, or conflicting context as one of the primary failure modes, not model capability. Teams that treated RAG as a complete solution found their systems hallucinating anyway, just on different things.

Agentic systems need more than document retrieval. When an AI system can take actions across multiple tools, it needs not just retrieved documents but tool state, task history, inter-agent outputs, and carefully managed working memory. A vanilla RAG pipeline is architecturally insufficient for these use cases.

What Does Context Engineering Replace?

This framing is slightly misleading. Context engineering does not replace RAG; it subsumes it. Retrieval from a vector store remains a valid and commonly used mechanism. What context engineering replaces is the assumption that retrieval is the whole problem.

The shift is from a single pattern (find relevant chunks, inject them) to a managed discipline: deliberately designing every slot in the context window, choosing what goes in, what stays out, and what form it takes. The AI architect moves from configuring a pipeline to designing an information architecture.

What Does a Context Engineering System Actually Contain?

The most useful framework is to think of context as having four layers, each the responsibility of the AI architect:

A RAG system typically addresses only the first row. A context engineering system addresses all four, deliberately, with explicit architectural decisions made for each layer.

What Do AI Architects Actually Build Now?

Diagnosing the business first means understanding which of these layers is causing the failure. Many organisations discover their AI systems underperform not because the model is weak and not because retrieval is poor, but because the system lacks user context (so every user gets the same generalised response) or lacks environmental context (so the model offers to take actions it cannot actually complete).

An AI architect designing a context engineering system makes decisions across five areas:

1. Context retrieval strategy. Which sources of knowledge does the system need access to? Vector stores for semantic search, relational databases for structured lookup, APIs for live data, and knowledge graphs for entity relationships are all valid retrieval primitives. The choice depends on the data structure, the query type, and acceptable latency.

2. Memory architecture. What does the system need to remember across turns, sessions, and users? Short-term memory (the current conversation window) and long-term memory (retrievable from a persistent store) serve different purposes and carry different costs. An architect decides how much history to retain in-window, what to compress and store, and what to discard.

3. Context compression. Large context windows are available, but they are not free. Token cost scales with context length, and irrelevant content degrades model performance. Compression strategies include summarisation, entity extraction, and structured formatting that lets the model parse relevant sections efficiently.

4. Context routing in multi-agent systems. Each agent needs only the context relevant to its task. Routing is the architectural decision about what each agent sees: sharing too much creates noise; sharing too little creates errors. Poor routing produces some of the hardest-to-debug failures in production agentic systems.

5. Context freshness. Retrieved documents go stale. Policies change. Prices update. The architect must design for staleness: cache invalidation policies, retrieval recency filters, and in some cases real-time retrieval rather than indexed lookups.

These are architectural decisions, not configuration choices. They require someone who understands data structures, inference costs, failure modes, and the business process the system is meant to serve. This is precisely why the AI architect role has become central to any serious AI deployment.

Is RAG Still Useful?

Yes, conditionally. Vector similarity retrieval remains the right tool for unstructured document search where queries are open-ended and the corpus is large. It is not the right tool when the knowledge structure is better represented as a graph, when the answer requires joining structured data, or when the query type is deterministic enough to be served by a SQL lookup.

GraphRAG (graph-based retrieval augmented with entity relationships) has emerged as a stronger pattern for enterprise knowledge bases where relationships between entities matter as much as the content of individual documents. Hybrid retrieval, combining dense vector search with sparse keyword search and a re-ranking step, is now the production standard for high-stakes systems where precision matters.

The architect’s job is to choose the right retrieval primitive for the data type and query pattern, then situate it correctly within the wider context engineering system. That is a more demanding role than configuring a single RAG pipeline, and it is why organisations that hire AI architects rather than simply buying AI tools get better outcomes.

How Do You Know When to Invest in Context Engineering?

The diagnostic signal is usually one of three symptoms:

• The model gives technically correct answers that are contextually wrong (missing user or environmental context)
• The system performs well in demos but degrades with real users over longer sessions (missing conversational context management)
• The system retrieves documents accurately but still produces hallucinated or irrelevant responses (context is retrieved but not structured well for model reasoning)

Each symptom points to a specific layer of the context engineering system that needs attention. This is why AI architecture starts with diagnosis: the fix for a user-context failure and the fix for a retrieval-quality failure are completely different architectural interventions, and conflating them wastes months of engineering effort.

FAQ

Is context engineering just prompt engineering with extra steps?

No. Prompt engineering optimises the instructions you give the model. Context engineering designs the systems that supply the model with information: retrieval pipelines, memory architectures, tool registries, and compression strategies. The scope is entirely different, and the skills required to do it well are closer to systems architecture than to prompt crafting.

Do I still need a vector database if I move to context engineering?

Probably yes, for semantic retrieval from unstructured documents. The vector store becomes one retrieval primitive within the context engineering system rather than the architecture itself. It sits alongside relational databases, graph stores, and API connections depending on the data types in play.

What tools do AI architects use for context engineering?

Common tools include LlamaIndex and LangChain for pipeline orchestration, Qdrant, Pinecone, and pgvector for vector storage, Neo4j for graph-based retrieval, Redis for short-term memory and caching, and MCP (Model Context Protocol) for standardised tool-context delivery in agentic systems.

How long does it take to migrate from a RAG pipeline to a proper context engineering architecture?

For a small system with one retrieval source and a single agent, a few weeks of architectural work. For an enterprise system with multiple agents, data sources, and user profiles, it is a design and build project measured in months. The first step is always a diagnostic: understand which context layers are missing before deciding what to build.

Does context engineering require a full-time AI architect?

Not always. Many SMEs benefit from a fractional AI architect who designs the context architecture, establishes the patterns, and upskills the internal team. The architecture can then be maintained by engineers. The design decisions, however, require architectural judgement that most development teams do not yet have and cannot quickly acquire.

Bedrock AI maps your systems, team and workflows to show where AI actually pays, before you spend a pound building. Book a strategy call.