Context Engineering Part 1: Why AI Agents Forget (And How to Fix It)

Understanding the four ways AI memory fails and the first two solutions

---

This article was originally published in LambdaTest’s blog here

The AI Memory Problem: An Introduction

Picture this: you're at a restaurant, and the waiter has a wonderful memory. They recall:

• The 50-page menu has every dish on it

• All of the conversations with customers from the past week

• Every request that was made that was special

• The ingredients in each dish

• Every order and table number

• All ways to pay are available

Sounds wonderful, doesn’t it? The problem is that when you ask, "What's good for lunch?" the waiter stops. They have so much information that they can't pay attention to your simple question.

If you don't do proper context engineering, this is what happens to AI systems.

AI agents are getting smarter and better at doing more complicated tasks, but they have a big problem: their "working memory" is limited. AI models can only hold so much information in their active context at any one time, just like the waiter who is too busy to help you. If you load them too much, they slow down, make mistakes, or even stop working altogether.

Context engineering is the field that deals with controlling what AI systems remember, forget, and find at each step of their work.

---

What is Context Engineering?

Leading AI researchers define context engineering as,

"The delicate art and science of filling the context window with just the right information for the next step."

Let's look at what "context" means in AI to get a better idea of this:

Understanding the Context Window

A context window is the amount of text that modern large language models (LLMs) can "pay attention to" when they make a response.

It's like your computer's RAM:

• The CPU is what gives the LLM its reasoning power.

• The RAM is the memory that is used to work (the context window).

• The hard drive is a place to store things for a long time (like documents and external databases).

As Andrej Karpathy explained, "LLMs are like a new kind of operating system. The LLM is like the CPU and its context window is like the RAM."

What Goes Into Context?

AI context usually has three parts:

1. Instructions

   • System prompts

   • Few-shot examples

   • Tool descriptions

   • Response format requirements

   • Behavioral guidelines

2. Knowledge

   • Domain facts

   • Company policies

   • Historical data

   • User preferences

   • Retrieved documents

3. Tools

   • Available function descriptions

   • API specifications

   • Previous tool call results

   • Error messages and feedback

---

Why Context Engineering Matters: The Four Failure Modes

When context isn't handled correctly, AI systems break down in certain, predictable ways. Drew Breunig identified four critical failure modes that plague AI agents:

1. Context Poisoning

What happens is that the AI remembers a hallucination or mistake and keeps it in its memory between interactions.

The DeepMind team documented this problem in their Gemini 2.5 technical report while the AI was playing Pokémon:

"An especially egregious form of this issue can take place with 'context poisoning' – where many parts of the context (goals, summary) are 'poisoned' with misinformation about the game state, which can often take a very long time to undo."

Real-world scenario:

Turn 1:
User: "My account number is 12345"
AI: "Got it, account 54321" (misheard)
AI saves to memory: account_number = 54321

Turn 2 (next day):
User: "Check my account"
AI: "Looking up account 54321..." (wrong every time)

Impact: The AI will always work with wrong information until someone fixes its memory. If the poisoned context changes the AI's goals or strategy, it can "become fixated on achieving impossible or irrelevant goals."

2. Context Distraction

What happens is that the AI gets too much information and starts to focus on things that aren't important instead of the task at hand.

As Drew Breunig explains, "Context Distraction is when the context overwhelms the training." The Gemini 2.5 team observed this while playing Pokémon:

"As the context grew significantly beyond 100k tokens, the agent showed a tendency toward favoring repeating actions from its vast history rather than synthesizing novel plans."

Instead of using what it learnt to come up with new plans, the agent became obsessed with doing things it had done before in its long history.

Real-world scenario:

Context includes:
- 500 lines of user's purchase history
- 200 product descriptions
- 50 previous conversations

User asks: "What's my current cart total?"

AI responds: "You previously bought a blue sweater on March 3rd for $29.99,
and before that you looked at red shoes on February 28th..."
(Never actually answers the cart total!)

Impact: The AI gets distracted by accumulated context and either repeats past actions or focuses on irrelevant details instead of addressing the actual query.

3. Context Confusion

What happens: Superfluous context influences responses in unexpected ways.

Drew Breunig defines this as "when superfluous context influences the response". The problem: "If you put something in the context, the model has to pay attention to it."

Even more dramatic: when researchers gave a Llama 3.1 8B model a query with all 46 tools in the GeoEngine benchmark, it failed. However, when the team reduced the selection to just 19 relevant tools, the project succeeded – even though both options were well within the 16k context window.

Real-world scenario:

Context includes: "User is from Texas"

User asks: "What's a good barbecue sauce recipe?"
AI assumes: Texas-style BBQ
AI responds: "Here's a great Texas BBQ sauce recipe with..."

But user wanted: Korean BBQ sauce (never specified because seemed obvious to them)

Impact: The AI gets sidetracked by all the context it has and either does the same thing again or focuses on things that aren't relevant instead of answering the question.

4. Context Clash

What happens is that too much context changes how people respond in ways that are hard to predict.

Drew Breunig describes this as "when parts of the context disagree."

A Microsoft and Salesforce research team documented this brilliantly. They took benchmark prompts and "sharded" their information across multiple chat turns, simulating how agents gather data incrementally. The results were dramatic: an average 39% drop in performance.

Why? Because "when LLMs take a wrong turn in a conversation, they get lost and do not recover." The assembled context contains the AI's early (incorrect) attempts at answering before it had all the information. These wrong answers remain in the context and poison the final response.

Real-world scenario:

Context piece 1: "Free shipping on orders over $50" (general policy)
Context piece 2: "No free shipping on furniture items" (category exception)
Context piece 3: "Holiday sale: Free shipping on everything!" (temporary promo)

User: "Do I get free shipping on this $75 chair?"
AI: "Uh... yes? No? Maybe? It depends on... wait..." (confused)

Impact: AI makes wrong guesses based on information that isn't directly related. The model has to pay attention to even information that isn't relevant in the context window, which hurts performance.

Anthropic's research emphasises that "context is a critical but finite resource" that must be managed carefully to avoid these failures.

---

The Four Pillars of Context Engineering

Top AI researchers and companies have agreed upon four basic ways to manage context well. This post will talk about the first two pillars. The last two will be covered in Part 2.

---

Pillar 1: WRITE (Structured Context Storage)

Core Principle: Don't make AI remember everything that's in active memory. Store data in formats that are easy to find and organised outside the context window.

Technique 1.1: Scratchpads

A scratchpad is a place where the AI can write down temporary notes while it works on a task.

How it works:

Think of it like sticky notes on a desk:

• 🟨 Yellow sticky: "Don't forget: Delta is the user's favourite airline."

• 🟦 Blue sticky: "Important: The budget is $500 at most."

• 🟩 Green sticky: "To Do: Look up the prices of flights for October"

The AI can:

• When they learn something new, they should write it down on a sticky note.

• When you need to remember something, read old sticky notes.

• To understand what the sticky notes say, read them all.

  The AI doesn't just remember everything; it writes things down and looks them up when it needs to.

Example Usage:

AI Task: "Analyze 5 research papers and synthesize findings"

Turn 1: Read Paper 1
  Scratchpad: "paper_1_key_finding: Machine learning improves accuracy by 15%"

Turn 2: Read Paper 2
  Scratchpad: "paper_2_key_finding: Neural networks require large datasets"

Turn 3: Read Paper 3
  Scratchpad: "paper_3_key_finding: Ensemble methods reduce overfitting"

Turn 4: Synthesize
  AI reads all scratchpad notes
  Context: Only the key findings (not the full papers!)
  Generates: Comprehensive synthesis

Anthropic's multi-agent research system uses this approach extensively: "The Lead Researcher begins by evaluating the method and saving the plan to Memory to maintain context, as exceeding 200,000 tokens in the context window will result in truncation, making it crucial to preserve the plan."

Technique 1.2: Structured Memory Systems

AI systems need more than just temporary scratchpads; they need long-term memory that lasts between sessions.

Three Types of Memory:

a) Semantic Memory (Facts & Knowledge)

Like knowing things about someone:

• "Sarah likes her coffee with no sugar and no cream."

• "Sarah can't eat peanuts because she's allergic to them."

• "Blue is Sarah's favourite colour."

The AI keeps track of things about you and what you like. It doesn't remember when it learnt these things; it only knows that they are true.

b) Episodic Memory (Past Events)

Like remembering certain times:

• "We fixed the login problem last Tuesday by changing the password."

• "We set up the new database two weeks ago."

• "We tried approach A last month, but it didn't work."

The AI can remember certain events and what happened. Like your photo album, each memory has a date and a story.

c) Procedural Memory (How-To Knowledge)

Like knowing how to make a cake:

• Step 1: Combine the sugar and flour

• Step 2: Put in the milk and eggs

• Step 3: Put in the oven at 350°F for 30 minutes.

The AI knows how to do things. You follow the steps in a recipe book to get something done.

Key Design Principles:

1. Schema-Driven: Use structured formats (JSON, databases), not free text

2. Time-Stamped: Track when information was added

3. Tagged: Enable filtering by category, importance, freshness.

4. Versioned: Track changes to memories over time

5. Queryable: Support efficient lookup and retrieval

---

Pillar 2: SELECT (Intelligent Context Retrieval)

The main idea is not to load everything. Get only what you need for the task at hand.

Technique 2.1: Retrieval-Augmented Generation (RAG)

RAG is probably the most important method in modern context engineering. It lets AI work with knowledge bases that are much bigger than their context windows.

How RAG Works:

Think about being in a big library with 10,000 books. "What's the recipe for chocolate chip cookies?" someone asks.

Without RAG (The Dumb Way):

• Take all 10,000 books to your desk.

• Try to read them all at once.

• Feel overwhelmed and lost

• Take a long time to find the answer

With RAG (The Smart Way):

Step 1 - Organise the Library (Done Once):

• Put a label on each book that says "Cooking", "History", "Science", etc.

• Make a card catalogue that works like a magic search index.

• Every card points to books that are related.

Step 2 - Smart Search (Every Question):

• Question: "Chocolate chip cookie recipe?"

• The magic catalogue says, "Check the 'Baking Cookbook' on shelf 5!"

• Grab ONLY that one book (not all 10,000!)

Step 3 - Quick Answer:

• Read only the cookie section (not the whole book)

• Find the recipe

• Give the answer

You only had to work with one book instead of ten thousand! That's RAG: finding the needle without having to carry the whole haystack.

The RAG Pipeline in More Detail:

┌─────────────────────────────────────────────────┐
│ INDEXING PHASE (Done Once)                      │
│                                                  │
│  1. Split documents into chunks                 │
│     "Large Document" → [Chunk1, Chunk2, ...]   │
│                                                  │
│  2. Create embeddings for each chunk            │
│     Chunk1 → [0.23, 0.45, ...] (vector)        │
│                                                  │
│  3. Store in vector database                    │
│     VectorDB.insert(chunk_id, embedding, text) │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│ RETRIEVAL PHASE (Every Query)                   │
│                                                  │
│  1. User asks question                          │
│     "How do I reset my password?"              │
│                                                  │
│  2. Create question embedding                   │
│     Question → [0.21, 0.47, ...] (vector)      │
│                                                  │
│  3. Find similar chunks                         │
│     VectorDB.similarity_search(question_vec)   │
│                                                  │
│  4. Retrieve top K most similar chunks          │
│     Returns: [Chunk45, Chunk12, Chunk89]       │
│                                                  │
│  5. Load ONLY those chunks into context         │
│                                                  │
│  6. Generate response                           │
│     AI uses just 3 chunks (not all 1000!)      │
└─────────────────────────────────────────────────┘

Advanced RAG Techniques:

a) Hybrid Search (Keyword + Semantic)

Think of two friends helping you look for a toy you lost:

• Friend 1 (Semantic Search): "Did you lose something round and red? I It might be in the toy box with other balls!"

  • Understands meaning and finds similar things

• Friend 2 (Keyword Search): "You said 'red ball'? I'll look for anything labelled 'red' or 'ball'!"

  • Looks for exact words and labels

• Together: They put their results together and show you the best ones!

  • Friend 1 found three types of balls: a red ball, a red bouncy ball, and a rubber ball.

  • Friend 2 found: a red ball, a ball pit, and a red balloon.

  • Combined: red ball (both found it!), bouncy ball, rubber ball ← Best results!

Using both methods together finds better answers than using just one!

b) Contextual Chunk Retrieval

Imagine reading a storybook, and you find the perfect sentence on page 42:

Without Context:

• You only read that one sentence

• "...and then she opened the door."

• Wait... who is "she"? What door? Why?

With Context (The Smart Way):

• Read the page before (page 41): "Sarah walked nervously toward the old house..."

• Read the target page (page 42): "...and then she opened the door."

• Read the page after (page 43): "Inside, she found the mysterious box..."

Now you understand! Sarah is the character, it's an old house door, and she's searching for something.

The lesson is to not just take the exact piece; take a little bit before and after to get the whole picture!

c) Reranking for Precision

I think about what the best pizza toppings are:

First Search: Someone quickly grabs 10 random toppings from the pantry:

• Pepperoni ✓ (good!)

• Chocolate chips ✗ (weird on pizza...)

• Mushrooms ✓ (good!)

• Gummy bears ✗ (definitely not!)

• Cheese ✓ (perfect!)

Reranking (The Smart Judge): Now a pizza expert looks at these 10 items and rates them:

• Pepperoni: 9/10 ⭐⭐⭐⭐⭐

• Cheese: 10/10 ⭐⭐⭐⭐⭐

• Mushrooms: 8/10 ⭐⭐⭐⭐

• Chocolate chips: 1/10 ⭐

• Gummy bears: 0/10

Final Selection: Pick the top 5 rated items!

The reranker is like a specialist who double-checks the first search and puts the best items at the top!

RAG Impact:

If done right, RAG can make a big difference in performance. Research indicates that employing RAG for tool descriptions can enhance tool selection accuracy by threefold in comparison to loading all tools into context. The Berkeley Function-Calling Leaderboard demonstrates that every model does worse with more tools, even when they are within their context window limits.

Technique 2.2: Dynamic Context Loading

Imagine you're packing for a day trip, and your backpack can hold 10 pounds:

Priority 1 - Critical (MUST pack):

• Water bottle (2 pounds) → Always pack this!

• Your phone (1 pound) → Must have!

• Current weight: 3 pounds

Priority 2 - Important (Pack if room):

• Lunch box (2 pounds) → Got room? Yes! Pack it.

• Sunscreen (0.5 pounds) → Got room? Yes! Pack it.

• Current weight: 5.5 pounds

Priority 3 - Nice-to-Have (Fill remaining space):

• Comic book (1 pound) → Still have room? Yes!

• Frisbee (0.5 pounds) → Still have room? Yes!

• Skateboard (4 pounds) → Will this fit? No! (would make it 11 pounds)

• Current weight: 7 pounds

Final backpack: Water, phone, lunch, sunscreen, comic book, frisbee = 7 pounds (under 10 limit!)

The AI does the same thing! It packs the most important information first, then adds more until the backpack (context window) is nearly full, but never overflowing.

Technique 2.3: Metadata Filtering

The Smart Pre-Filter Strategy:

Imagine you're looking for your red LEGO car in a huge toy room:

Dumb Way:

• Search through ALL toys (dolls, blocks, cars, puzzles, stuffed animals...)

• Takes forever!

• Find 100 toys, most not even cars

Smart Way (Metadata Filtering):

• Filter 1: "Only show me cars" (not dolls, not blocks)

  • Now looking at 20 cars instead of 1000 toys

• Filter 2: "Only red ones" (not blue, not yellow)

  • Now looking at 5 red cars

• Filter 3: "Only LEGOs" (not Hot Wheels, not wooden cars)

  • Found it! 1 red LEGO car

Why this is brilliant:

• Faster: Searched 5 items instead of 1000

• More accurate: Every result is relevant

• Respects rules: Won't show you toys that belong to your sibling

• Fresh: Can filter to "only toys bought this month"

The AI does this with information – it filters first, then searches!

---

Real-World Example: Enterprise Documentation Assistant

Let's see how WRITE and SELECT work together in practice:

Challenge: The company has 10,000 pages of documentation. The AI assistant should answer employee questions.

Without Context Engineering:

• Try to load all 10,000 pages → Context overflow

• Load random pages → Wrong answers

• Load recent pages → Misses critical info

With Context Engineering (WRITE + SELECT):

WRITE Phase:

Step 1: Organize all documentation
  - Create labels and categories
  - Add metadata (department, date, topic)
  - Build search index

Step 2: Store in structured format
  - HR policies → HR database
  - IT procedures → IT database
  - Product docs → Product database

SELECT Phase:

Employee asks: "What's our PTO policy?"

Step 1: Identify intent → HR question
Step 2: Filter metadata → HR database only
Step 3: RAG search → "PTO policy" sections
Step 4: Retrieve → 3 relevant pages (not 10,000!)
Step 5: Load into context → Just those 3 pages
Step 6: Generate answer → Fast and accurate!

Result: Fast, accurate answers without context overload!

---

How We're Applying WRITE and SELECT at LambdaTest

At LambdaTest, we're deeply invested in building intelligent AI-powered systems. Here's how we're applying these first two pillars (without revealing proprietary details):

Our WRITE Strategy

Structured Information Architecture

We organise information hierarchically, following best practices from industry research:

Level 1: System Policies (rarely change)

• Core functionality rules

• Platform capabilities

• Security requirements

Level 2: Feature Documentation (monthly updates)

• Feature specifications

• Integration guidelines

• API references

Level 3: User Context (session-based)

• Current workflow state

• User preferences

• Active task details

Level 4: Interaction History (turn-based)

• Recent actions

• Generated outputs

• Feedback received

Why this works: AI quickly knows where to look. Need a core rule? Check Level 1. Need user preference? Check Level 3.

Memory Management

We implement both short-term and long-term memory:

• Short-term (Session Memory): Tracks the current workflow, decisions made, and progress

• Long-term (Cross-Session Memory): Learns user patterns, common workflows, and domain-specific knowledge

The AI gets smarter with each interaction, like an experienced colleague!

Our SELECT Strategy

Intelligent Context Retrieval

We don't load everything at once. Instead:

1. Analyze the task → Determine what information categories are needed

2. Fetch relevant subset → Use semantic search and metadata filtering

3. Load minimal context → Only what's necessary for current step

4. Cache for reuse → If the same information is needed again

Example Workflow:

User Task: "Process this technical document"

Step 1: Identify document type
  Context loaded: Document type classifiers (lightweight)

Step 2: Extract key information
  Context loaded: Extraction patterns for this type

Step 3: Generate structured output
  Context loaded: Output schemas and examples

Each step: Different focused context!

Results We're Seeing:

While we can't share exact numbers, we're experiencing:

• Dramatically improved accuracy across workflows

• Significant reduction in processing time

• Better cost efficiency per operation

• More consistent, reliable outputs

• Enhanced user satisfaction

The investment in context engineering has fundamentally transformed our AI systems from "useful" to "production-grade"”

---

Key Takeaways from Part 1

The Four Failure Modes (What Goes Wrong):

1. Context Poisoning → Mistakes get saved and reused

2. Context Distraction → Too much info, AI loses focus

3. Context Confusion → Irrelevant info influences decisions

4. Context Clash → Contradictory info causes inconsistency

The First Two Solutions (What to Do):

WRITE:

• Use scratchpads for temporary notes

• Build structured long-term memory (semantic, episodic, procedural)

• Organize information hierarchically

• Tag and timestamp everything

SELECT:

• Use RAG for large knowledge bases

• Apply hybrid search (semantic + keyword)

• Grab context around relevant chunks

• Rerank for precision

• Load dynamically by priority

• Filter by metadata first

The Core Insight:

Don't try to load everything into the AI's "backpack" at once. Instead:

• WRITE it down in organized notes

• SELECT only what you need right now

This is like the difference between:

• ❌ Carrying all your textbooks to every class

• ✅ Bringing only today's math book to math class

---

What's Coming in Part 2

In the next installment, we'll explore:

• Pillar 3: COMPRESS – Making information smaller without losing meaning

  • Hierarchical summarization

  • Sliding windows for conversations

  • Tool output compression

  • When to use lossy vs lossless compression

• Pillar 4: ISOLATE – Separating concerns for clarity

  • Multi-agent architectures (90.2% performance improvement!)

  • Sandboxed execution

  • State-based isolation

• Production Challenges – Real engineering lessons

  • Handling errors that compound

  • Debugging non-deterministic systems

  • Deployment strategies

  • Anthropic's hard-won lessons from production

• Advanced Patterns – Taking it to the next level

  • Context tiering strategies

  • Long-horizon conversation management

  • Smart caching techniques

Don't miss Part 2!.

---

Further Reading

Essential for Understanding Part 1:

1. How Long Contexts Fail (and How to Fix Them) - Drew Breunig's essential guide covering the four context failure modes with extensive research citations

2. Context Engineering for AI Agents – A comprehensive guide covering the four pillars (WRITE, SELECT, COMPRESS, ISOLATE)

3. Anthropic's Multi-Agent Research System – Deep dive into building production multi-agent systems

Research Referenced:

• DeepMind Gemini 2.5 Technical Report: Context poisoning in game-playing agents

• Microsoft/Salesforce Sharded Prompts Study: 39% performance drop from context clash

• Berkeley Function-Calling Leaderboard: Every model performs worse with more tools

• Daffodil Software Best Practices: Hierarchical organization patterns