Victor Johnson
A large language model (LLM) can be capable of solving a complex problem and still produce an incorrect answer because it received the wrong information. It may miss an important document, rely on outdated instructions, forget a previous decision or become distracted by irrelevant content. This is where context engineering becomes important.
Context engineering is the practice of designing, selecting, organising and maintaining the information an AI model receives when performing a task. It goes beyond writing a good prompt. It involves deciding which instructions, documents, memories, tool results and conversation details should be available to the model at a particular moment.
For simple chatbot applications, a system prompt and recent conversation history may be sufficient. For AI agents that retrieve documents, call APIs, use tools and perform multi-step tasks, managing context becomes a core part of the system architecture. This guide explains what context engineering is, how it differs from prompt engineering, why it matters for AI agents, and how to implement it in production systems.
What Is Context Engineering?
Context engineering is the systematic management of information supplied to an AI model so it can perform a task accurately, efficiently and within the intended constraints. The context is everything available to the model during a particular inference request. Depending on the application, it may include:
- System instructions defining the model’s role and behaviour.
- The user’s current request.
- Relevant conversation history.
- Documents retrieved from a knowledge base.
- Agent memory and previously recorded decisions.
- Tool descriptions, parameters and results.
- Structured application state and task progress.
- Security, permission and business-rule information.
The challenge is not simply to provide more information. It is to provide the right information, in the right form, at the right time. Imagine an AI support agent answering a customer’s question about a delayed order. It might need the customer’s current order status, the relevant delivery policy and the latest carrier update. Providing thousands of unrelated support conversations would not improve the answer. It could increase processing costs, introduce conflicting information and make the model less reliable. Good context engineering ensures the agent receives the information that matters.
Context Engineering vs Prompt Engineering
Prompt engineering and context engineering are related, but they solve different problems. Prompt engineering focuses on how instructions are written. It involves choosing wording, examples, constraints and output formats that help a model respond correctly. Context engineering, on the other hand, focuses on the complete information environment surrounding those instructions. It determines what information is included, where it comes from, how it is organised and when it should be refreshed.
| Aspect | Prompt Engineering | Context Engineering |
|---|---|---|
| Primary focus | Instruction design | Information selection and management |
| Typical inputs | Instructions and examples | Instructions, memory, documents, tool outputs and state |
| Main question | How should the model be instructed? | What should the model know right now? |
| Scope | Usually an individual prompt or template | The wider application and agent workflow |
| Common techniques | Few-shot examples, structured prompts | RAG, memory management, filtering, summarisation and context routing |
| Common failure | Ambiguous instructions | Missing, stale, conflicting or excessive information |
A well-written prompt cannot reliably compensate for missing evidence. Likewise, relevant documents may not help if the model receives unclear instructions about how to use them. Reliable AI systems need both.
Why Context Engineering Matters for AI Agents
An ordinary LLM request may consist of one question and one response. An AI agent can operate over many steps, interact with external systems and accumulate information throughout a task. Each step changes what the agent needs to know.
For example, an agent researching software vulnerabilities might first search public advisories, then inspect package versions, compare affected configurations and prepare a report. The information needed during the initial search is different from the information required when writing the final assessment. Without context management, the agent may carry every search result and intermediate message into subsequent requests. This creates several problems.
Context Windows Are Limited
LLMs process information within a finite context window, measured in tokens. Even models supporting large context windows cannot accept unlimited information. Long-running agents can accumulate substantial histories containing tool responses, retrieved documents and intermediate decisions.
When the available context becomes too large, the application may need to truncate, summarise or selectively retain information. Poor truncation can remove critical requirements, while poor summarisation can distort facts or omit exceptions. Context engineering provides a deliberate approach to deciding what stays and what can be discarded.
More Context Does Not Always Mean Better Answers
Adding more documents does not automatically improve accuracy. Irrelevant material can distract a model from the task, while repeated or contradictory instructions may make it harder to determine which information should take priority.
A model analysing an application’s authentication implementation may benefit from the relevant source files, configuration and framework documentation. Adding the entire repository, unrelated issue discussions and historical deployment logs could make the analysis slower and less focused. The goal is context relevance rather than context volume.
AI Agents Need Information Across Multiple Steps
Agents frequently need to remember decisions and results from earlier stages. A travel-planning agent may need to retain the destination, travel dates, budget and selected hotel while discarding temporary search results that are no longer useful.
Similarly, a coding agent may need to remember the selected implementation approach and files already modified without repeatedly including every terminal output. Context engineering helps maintain task continuity without allowing the working context to grow indefinitely.
Context Affects Security and Privacy
AI agents may process API responses, customer records, internal documents and other sensitive information. If the application retrieves or stores information without respecting access boundaries, the model may receive data that the current user is not authorised to access.
Retrieved documents and tool responses may also contain malicious instructions intended to influence agent behaviour. Context engineering must therefore consider data access, source trust, information sensitivity and instruction boundaries, not only response quality. The OWASP AI Exchange provides guidance on AI security threats, including prompt injection, sensitive data exposure and agent-related risks.
The Main Components of an AI Agent’s Context
A practical way to design context engineering is to divide the information available to an agent into distinct components. These components work together to provide the model with the information needed for each task, while avoiding unnecessary or outdated material.
1. Instructions and Policies
Instructions define what the agent is expected to do, how it should behave and which constraints apply. These may include the agent’s role, output requirements, task objectives and application policies. Stable instructions should be managed separately from retrieved documents or user-supplied content. This helps prevent lower-trust information from being treated as authoritative instructions.
2. Conversation and Task History
Conversation history provides continuity between user messages. However, not every previous message remains relevant. Applications can retain recent exchanges while summarising older discussions or extracting important decisions into structured state. For example, a customer support agent may need the current issue, previous troubleshooting steps and agreed next action rather than the complete conversation transcript.
3. Retrieved Knowledge
Retrieval-Augmented Generation (RAG) allows applications to fetch relevant information from documents, databases or search systems and include it in the model’s context. Retrieval is particularly useful when the model needs current information, organisation-specific knowledge or evidence that was not part of its training data. Useful retrieval context should include enough source information to distinguish verified evidence from unsupported statements.
4. Agent Memory
Memory stores information that may remain useful beyond the immediate request. Working memory can hold the current task state, while episodic memory can record previous interactions or outcomes. Semantic memory can retain useful facts, and procedural memory can represent reusable workflows or instructions. Not all memory should be inserted into every request. The application should retrieve relevant memories according to the current task, user and permissions.
5. Tool Results and Environment State
AI agents often interact with tools such as web search, databases, code execution environments and external APIs. Tool outputs may contain important facts, but they can also be large, outdated or untrusted. An agent checking a deployment does not necessarily need the entire output of every command. It may need the deployment identifier, current status, relevant error messages and the next required action. Structured tool responses can make this information easier to validate and use.
How Context Engineering Works in Practice
Context engineering is best understood as a process that runs throughout an agent’s execution rather than a one-time operation before the first model call. A typical workflow contains six stages, beginning with identifying the task and ending with evaluating whether the model received the right information.
Step 1: Understand the Current Task
Identify the user’s objective and determine which information is necessary. For a question about a product return, the relevant context may include the order details, purchase date and applicable return policy. For a software security review, the relevant context may include dependency versions, affected components and vulnerability advisories. The information requirements should follow the task rather than a fixed template containing everything available.
Step 2: Retrieve Relevant Information
Collect information from appropriate sources, such as application databases, document repositories, agent memory or external tools. Retrieval may combine keyword search, semantic search and metadata filtering. The system should apply access controls before exposing retrieved information to the model.
Step 3: Filter and Rank the Results
Retrieved information should be evaluated for relevance, freshness, reliability and duplication. For example, when reviewing a vulnerability, an official vendor advisory may deserve priority over an older article describing a preliminary finding. The application can rank documents, remove duplicates and exclude content that does not support the task.
Step 4: Assemble the Context
Organise the selected information into a clear structure. Instructions, user requests, retrieved evidence and tool outputs should remain distinguishable. For complex tasks, structured formats such as JSON may be useful for application state, while labelled document excerpts may be more appropriate for evidence.
Step 5: Update Context During Execution
As the agent works, it may discover new information or complete intermediate steps. The application should update the context accordingly. Completed steps can be summarised, irrelevant tool output can be removed, and important decisions can be retained.
Step 6: Evaluate the Result
After execution, evaluate whether the agent received sufficient and appropriate context. A poor response may be caused by a retrieval failure, missing memory, stale information or conflicting instructions rather than by the underlying model. Identifying the cause helps improve the context pipeline.
Practical Examples of Context Engineering
Context engineering becomes easier to understand when applied to real agent workflows. The following examples show how selecting and maintaining the right information can improve task execution without unnecessarily increasing the amount of context sent to the model.
Example 1: Customer Support Agent
A customer asks, “My order has not arrived. Can I request a refund?” A poorly designed agent might receive the complete support knowledge base and a long history of unrelated interactions. A context-engineered agent would instead retrieve the information directly related to the customer’s request:
- The customer’s authorised order record.
- The latest delivery status.
- The refund policy applicable to that purchase.
- Relevant previous support actions.
- The current request and any necessary response instructions.
If the carrier later updates the delivery status, the agent should refresh that information rather than continue relying on an earlier result. This approach reduces unnecessary context and makes the response more relevant.
Example 2: AI Coding Agent
Consider an AI agent asked to fix a failing authentication test. Sending the entire codebase to the model may be inefficient. A better approach is to identify the failing test, retrieve the related source files, inspect the relevant configuration and include only the necessary dependency information.
As the agent investigates, it can maintain a compact working state:
task: Fix failing authentication test
status: investigating
relevant_files:
- tests/test_auth.py
- app/auth.py
observations:
- Test fails during token validation
- Failure began after dependency update
next_step: Check token validation compatibility
This state can be updated after each significant step. The agent does not need to repeatedly process every earlier command output to understand its progress.
Example 3: Security Research Agent
An AI agent investigating a newly disclosed vulnerability may collect vendor advisories, CVE records, affected version ranges and remediation information. The context pipeline should prioritise authoritative sources and retain publication dates.
If two sources disagree, the agent should preserve that disagreement rather than silently merge the information into one unsupported conclusion. For example, the agent might retain:
{
"vulnerability": "Example vulnerability",
"affected_product": "Example package",
"vendor_advisory_checked": true,
"version_status": "Requires verification",
"conflicting_sources": true,
"next_action": "Confirm affected versions"
}
This prevents uncertain information from being presented as verified.
Example 4: Multi-Agent Research Workflow
A research system may use one agent to collect sources, another to analyse evidence and a third to prepare a report. Passing the entire conversation history between all agents is not always necessary.
Instead, the research agent can provide a structured handoff containing source references, verified findings, unresolved questions and relevant limitations. The analysis agent can then work with a focused evidence package. This also makes it easier to review which information influenced the final output.
Context Engineering Techniques for Production AI Systems
Several techniques can be combined to build an effective context-management pipeline. The right combination depends on the application’s tasks, available information sources and requirements for accuracy, security and performance.
Retrieval-Augmented Generation
RAG supplies relevant external information when needed. A typical implementation retrieves candidate documents, ranks them and inserts selected excerpts into the model’s context. For better results, retrieval should consider document structure, metadata, access permissions and freshness.
The Microsoft Azure AI Search documentation provides an introduction to RAG architectures and retrieval design.
Context Summarisation and Compaction
Long conversations and tool histories can be compressed into shorter representations. A useful summary should preserve the current objective, important facts, decisions, constraints, unresolved questions and references needed for subsequent work. Summarisation should not silently transform uncertainty into certainty or remove information that affects security-sensitive decisions.
Selective Memory Retrieval
Instead of loading all stored memories, retrieve only those relevant to the current request. Memory retrieval should account for user identity, task context, information age and permission boundaries. Stored memories should not automatically be treated as current instructions or authorisation.
Structured Context and State
Structured data helps separate important application state from unstructured conversation history. For example, an agent performing a multi-step workflow can maintain fields for task status, completed actions, pending approvals and unresolved issues. This allows the application to update specific information without rewriting the entire conversation.
Context Routing
Different tasks may require different information sources. A coding task may need repository files and test results, while a financial analysis task may need authorised transaction records and reporting rules. Context routing selects the appropriate retrieval tools, memory sources and instructions for the current task. This avoids sending every available knowledge source to every model request.
Context Caching
Applications may cache frequently used information to reduce repeated retrieval and processing. Caching can improve efficiency when the same reference documents or stable instructions are required across multiple requests. However, cached information must be refreshed when its source changes. Sensitive cached content must also remain isolated according to user and tenant permissions.
Best Practices for Context Engineering
Building an effective context pipeline requires more than selecting relevant documents. Production AI systems must also manage information quality, source trust, security and performance. The following practices help maintain reliable context as applications become more complex.
Keep Context Relevant and Task-Specific
Start with the information required to complete the current task. Avoid including entire documents or conversation histories when a smaller selection contains the necessary facts. Use retrieval filters, document ranking and structured state to keep context focused.
Separate Instructions From Untrusted Content
System instructions, user requests, retrieved documents and tool outputs should remain distinguishable. An instruction found inside a web page or retrieved document should not automatically receive the same authority as an application-level instruction. This separation is important for reducing prompt-injection risks.
The OWASP LLM Prompt Injection Prevention Cheat Sheet explains relevant defensive techniques.
Preserve Source Provenance
When context comes from external information sources, retain useful metadata such as the document title, source, retrieval time and relevant version. This helps an agent distinguish current evidence from historical material. It also supports debugging when an answer is based on an incorrect or outdated document.
Manage Context Freshness
Information that changes frequently should not remain in context indefinitely. Order statuses, account permissions, vulnerability details and system configurations may need to be retrieved again before important decisions. Applications should define freshness requirements according to the type of information and the consequences of using outdated data.
Avoid Blind Summarisation
Summarisation can reduce token usage, but it can also remove important details. For high-impact tasks, preserve original references and critical constraints alongside the summary. Where necessary, allow the agent to retrieve the original evidence again.
Apply Access Controls Before Retrieval
A model should not receive information simply because it exists in a connected database or knowledge repository. The retrieval layer should enforce the current user’s permissions before returning documents or records. Context engineering should complement application security, not replace it.
Protect Sensitive Information
Avoid inserting passwords, API keys, authentication tokens or unnecessary personal information into prompts and tool outputs. Also consider whether sensitive information could be captured by logging, tracing or observability services. Apply data minimisation, appropriate retention controls and access restrictions throughout the context pipeline.
Measure Context Quality
Context engineering should be evaluated using more than token counts. Useful measurements include:
- Retrieval relevance and coverage.
- Accuracy of answers grounded in retrieved evidence.
- Frequency of missing or outdated context.
- Token consumption and latency.
- Success rates for multi-step agent tasks.
- Permission and information-boundary violations.
- Accuracy of summarised task state.
A context pipeline that uses fewer tokens but regularly omits essential information is not necessarily better.
Common Context Engineering Mistakes
Even well-designed AI applications can experience context-management failures. Some of these problems affect response accuracy, while others introduce unnecessary costs or security risks.
Including too much information: Large amounts of irrelevant material can reduce focus and increase cost.
Using stale retrieval results: Previously retrieved information may no longer reflect the current system state.
Treating memory as verified truth: Stored memories can be outdated, incomplete or incorrect.
Mixing instructions with retrieved content: Untrusted documents may contain text designed to redirect agent behaviour.
Removing important details during summarisation: Critical constraints, unresolved questions and source references can disappear.
Ignoring tool-output size: Large API responses and command outputs can consume substantial context without adding useful information.
Failing to evaluate retrieval quality: An apparently incorrect model response may originate from poor document selection rather than model reasoning.
Addressing these problems requires ongoing measurement and refinement rather than a single prompt change.
How to Start Implementing Context Engineering
For teams building an AI agent, a practical starting point is to create a simple context pipeline before introducing complex memory systems or multiple agents. First, define the information required for each major task. Identify which sources provide that information and which permissions apply.
Next, build retrieval and filtering mechanisms that select relevant records rather than returning everything available. Keep the agent’s current task state in a structured form, including important decisions, completed steps and pending actions. Introduce summarisation when conversation and tool histories become too large, but preserve original evidence when accuracy matters.
Finally, evaluate the pipeline using realistic tasks. Check whether the agent receives the right information, whether it respects access boundaries and whether its answers remain accurate as the task progresses.
Frameworks such as LangGraph can help manage stateful agent workflows, while LlamaIndex provides tools for building retrieval and knowledge-based applications.
For further reading, Anthropic’s Effective Context Engineering for AI Agents discusses practical context-management techniques, including context selection, memory and long-running agent workflows.
The central design principle is straightforward: an AI agent should receive the information it needs to perform its current task, without unnecessary data, outdated context or misplaced authority.
AI Agent Memory: types, design and implementation
