Why your AI agent needs a context graph, not just more tools
October 16, 2026 · 7 min read
Most teams start their AI work the same way. They connect an agent to Zendesk, Jira and Stripe through MCP servers and call it done. The agent can now "see" the company. Then someone asks it a question that spans a week of data, and it falls over.
Gil Feyg, co-founder and CTO of Merge, gave a talk on exactly this gap: why a company needs a context graph and how to structure one. He kept it vendor-neutral on purpose. What follows is our write-up of the ideas, plus where we've landed on them while building the Canon in dpaperwork.
What is a context graph?
Forget nodes and edges for a moment. In this framing a context graph is the Canon, everything an agent can draw on to answer a question the way your company would. It has four parts:
- Third-party data from the systems you already use, like NetSuite or Jira.
- Structured context, the documents and reference material you always want an agent to have.
- Memories the agent forms as it works, either because someone told it to remember something or because it decided to.
- Skills, the written-down processes for how your company does things.
Skills are the part people skip, and Feyg argued they matter most. More on that below.
Third-party data
NetSuite, Jira, Zendesk, the CRM
Structured context
Docs every agent should always have
Company
brain
Memories
Facts the agent was told or chose to keep
Skills
How your company does things, written down
Where live lookups break
Take a simple question: "Why was customer A upset last week?"
An agent with a ticketing MCP server handles this fine. It pulls that customer's tickets from last week, sees three were filed about a broken API sync, and answers.
Now change one word. "Which customers were upset last week?"
The agent now has to load every ticket from the past week. Stretch it to a year and it's paging through thousands of tickets, a hundred at a time, until the request times out. The question is reasonable. The agent just can't answer it.
The problem isn't MCP. It's the APIs underneath. They were built for fetching records by ID, not for questions asked in English across a whole dataset. Feyg's point was that vendors have no reason to fix this. Vectorizing your data costs them money, and a good semantic API means you visit their product less.
So you need a local copy.
Live API only
“Which customers were upset last year?”
…
Synced copy
“Which customers were upset last year?”
Sync what you need to search, look up the rest
The rule is simple. If a question is a single lookup or a single action ("what plan is this customer on in Stripe", "refund this charge"), call the live API. Live calls are cheaper and easier to build.
If a question needs semantic search or analysis across lots of records, sync a copy of that data into a vector database you control, and query it there. That costs more. You have to run the database, keep the sync going, and decide how data gets stored and retrieved. But it's the only way to answer the year-long question.
Most real systems need both: live MCP lookups for simple reads and actions, and a synced cache for anything deep.
Router in, summarizer out
Once you have two sources of data, you have a new problem. You can't dump everything you retrieved back into the agent's context.
Feyg described two pieces that sit around the retrieval layer. A router reads the incoming prompt and decides which agent or path should handle it. A summarizer collects what came back from the synced store and the live calls, condenses it, and returns that to the agent that asked.
Skills decide where to look
"Which customers were upset last week?" means different things at different companies. At one, it's a count of open tickets. At another, ticket volume is noise because many tickets are positive. A third company doesn't use a ticketing tool for this at all and always checks survey results in Qualtrics first.
An agent can't guess that. It has to be written down as a skill: how do we decide whether a customer is happy? That's why skills belong inside the context layer and not off to the side. Getting context is itself a company process.
So the full flow looks like this:
- A prompt comes in and the router sends it to the right agent.
- The agent checks for a matching skill or memory.
- The skill tells it where to look, in the synced store, through live lookups, or both.
- The summarizer condenses the results and the agent answers.
Prompt
“Which customers were upset last week?”
Router
Picks the agent that should handle it
Skills and memory
“Customer happiness = Qualtrics first, then tickets”
Synced store
Semantic, cross-record
Live MCP
Single lookups and actions
Summarizer
Condenses results for the agent
Answer
With sources attached
With the synced store, the "upset customers" question becomes easy. Gather every customer with tickets last week, summarize each customer's tickets, give a status per customer. Nothing clever, just the right data in the right place.
Four tiers of context
Feyg laid these out as tiers, roughly from cheapest to most involved:
- Prompt, skills and memory. These drive the decision about where to fetch data from.
- Live API calls. Cheap and easy. Prefer them for simple lookups.
- Cached, synced data. Needed for semantic and cross-record questions. More expensive to run and harder to keep fresh.
- Derived context. New context you create while pulling data in. If the agent summarizes a customer's full ticket history, store that summary. Next time someone asks a common question, you don't re-fetch and re-summarize everything.
Derived context is the tier I'd underline. It's what turns repeated work into a stored answer.
Prompt, skills, memory
Decides where to fetch from
Live API calls
Simple lookups and actions
Cached, synced data
Semantic and cross-record questions
Derived context
Stored summaries, so work is not repeated
The decision flow
When a prompt arrives:
- Is there a matching skill or memory? Run it. If it doesn't need outside data, answer.
- If it needs data, or there was no match, check the synced store. Is the data fresh enough to meet your SLA? Use it.
- If not, call the live API.
- Loop until you have enough context.
- Optionally save what you learned as a memory, then answer.
Matching skill or memory?
Yes run it
No go to data selection
Does the skill need outside data?
Yes go to data selection
No answer now
Is synced data fresh, within SLA?
Yes use the cache
No call the live API
Enough context?
Yes save a memory, answer
No fetch more
Keep the receipts
The last section of the talk covered traceability, and it's the part teams regret skipping. Never drop where a piece of data came from. MCP responses carry it. For anything you sync into a vector database, put it in the metadata next to the data.
For every fact an agent uses, you want to know:
- Where it was fetched from, and when.
- Which identity and scopes fetched it, and who asked.
- Whether it was transformed, and what the original looked like.
- What should invalidate it, for example "ignore anything older than a month".
- source
- zendesk / ticket 48213
- fetched_at
- 2026-10-14T09:12:00Z
- fetched_by
- service identity, scope tickets:read
- requested_by
- priya@acme.example
- transformed
- summarized from 6 comments
- invalidate_after
- 30 days
Chat tools that list sources under an answer are doing a lighter version of this. For business decisions it matters more, because data goes stale and versions change. You don't want an agent deciding something today based on a record it synced a year ago. And when an answer does go wrong, and one will, provenance is how you find out why.
A worked example
"Give me a one-paragraph status update on Acme. Include renewal risk and the last support interaction."
- Check the cache for an Acme account summary. If it's stale or missing, call the CRM live.
- Get the last support interaction from ticketing. Use the cache if it's within SLA. If you have to go live, pull only the last five tickets. Pulling full history live is how you hit timeouts.
- Write a derived summary, store it for next time, and return it.
What we're taking from this
Three things stuck with us.
Tools are not context. An agent with a dozen integrations still can't reason about your company. It needs your data, a synced copy where search matters, your processes written as skills, and the tools to act.
More data is not better data. Pull what you need, keep it fresh, and structure synced data around how it will be queried.
Every answer has to be traceable. If you can't explain why the agent said something, you can't trust it with a decision.
This lines up closely with how we think about the Canon in dpaperwork. Agents read from a shared, permissioned Canon instead of hitting every tool cold, they show what they changed, and nothing they produce reaches a client until a person accepts it.