Why AI agent performance degrades over time, and how to keep it in check
Why AI agents can drift, four signals to watch, and a six-step review with a named owner and a recorded baseline.
Written by the Problem Solver Team with AI assistance.
The short answer
AI agent performance can degrade when a long task accumulates context or when inputs, policies, connected systems and models change between runs. These mechanisms need different fixes. Record a baseline, compare a fixed test set and recent live cases, and assign a person to review speed, volume, errors and capacity. Decline is possible, not inevitable.
Key takeaways
- An agent that keeps adding to one long conversation carries more text into every later step. That can mean more cost and, according to Anthropic, lower accuracy.
- Inputs, rules, connected systems and the AI model itself change over time. An agent that was right at launch can drift without anyone touching it.
- You cannot see drift without a baseline. Record how the agent did in its first weeks and compare every month.
- Watch four things: speed, volume, errors and fullness. They come from a well-known monitoring practice and need no technical skill to read.
- Give the review a named owner and a fixed day. Without both, it does not happen.
Many teams expect an AI agent to behave the same in month six as in week one. It may not. Answers can get slower, costs can rise, and results can become less reliable. This explainer is for an operations leader who is not a technical specialist. It covers why AI agent performance degrades over time and what you can check this week.
01Why does an AI agent get slower and costlier over time?
A long-running task can accumulate more history at each step. This is different from drift across months: independent tasks may start with fresh context, while inputs and systems can change over time.
Anthropic's documentation explains how conversation history accumulates and how longer context can reduce accuracy and recall. It also describes context management and caching. More text does not translate into a fixed cost increase: the bill depends on retained history, cached input, model and pricing.
Ask your supplier: "What does the agent carry into each step, what gets dropped, and what is cached?" Compare your own timings and cost per completed case before drawing conclusions.
02Why can the same agent get worse when nothing was changed?
Because the things around it change. An agent is judged against inputs, rules and systems, and each can move without anyone telling the agent.
Four common sources of drift:
- Inputs change. Customers write differently, a new product line arrives, a supplier changes its invoice layout.
- Rules change. A policy, a limit or an approval path is updated, but the agent's instructions still describe the old one.
- Connected systems change. A field is renamed, a form gets a new step, a tool is retired.
- The model changes. A provider may update or replace the AI model behind a service. OpenAI's documentation says that writing evaluations, meaning tests against your own expectations, matters especially when upgrading or trying new models.
These changes can expose weaknesses in the agent or its setup. They are reasons to look at the whole process regularly.
Composite example (illustrative, not a client). An agent checks incoming supplier invoices for missing fields and sends the incomplete ones back. At launch, a reviewer corrects a handful of its results each week. Over the following months a supplier changes its invoice layout and a new approval step is added. Nobody changes the agent. Reviewers start correcting more, then stop reporting it because it is routine. Months later someone notices that the "sent back" count has crept up and that the monthly bill is higher. Because no baseline was recorded, nobody can say when it began.
03What should you watch to catch the slide early?
Watch four signals, adapted from the "four golden signals" in Google's Site Reliability Engineering book: latency, traffic, errors and saturation. The book is about monitoring services, not AI, so we adapt the idea rather than claim it covers agents. In plain words for an agent, they are speed, volume, errors and fullness.
| Signal | What it means for your agent | Question to ask |
|---|---|---|
| Speed | How long a case takes from arrival to finished | Is a typical case taking longer than in the baseline? |
| Volume | How many cases the agent handles, and how many steps each takes | Did the load or the steps per case grow? |
| Errors | Results a person corrected, rejected or sent back | Is the correction rate rising, and in which category? |
| Fullness | How close the agent is to its limits: context size, usage caps, queue length | Is anything near a limit or growing steadily? |
The same chapter lists "comparing over time" and "analyzing long-term trends" among the reasons to monitor. That is the point of the baseline: one month's figure says little, and the change between months says a lot.
04How do you run a focused monthly check?
Pick one process, one owner and one fixed day each month. We suggest a 30-minute review time box after the baseline, test cases and reporting are prepared; setup and investigation can take longer.
- Write down the baseline. Record the four signals for the first weeks after launch, using your own figures. Done when the page has a date and an owner's initials.
- Choose a test set. Pick real past cases that cover normal and tricky situations, with the correct result noted by a person. Done when a colleague can rerun them without asking you.
- Run the test set and compare. Done when you have this month's results beside the baseline for every signal.
- Sample live results. Read a small random handful of recent real cases. Done when you have written one line on each: right, wrong or unclear.
- Look for changes. Ask what changed in inputs, rules, systems and the model since last month. Done when the answer is written, even if it is "nothing".
- Decide and record. Choose one: leave it, adjust the instructions, trim what the agent carries, or pause it. Done when the decision and its owner are on the page.
OpenAI's documentation describes an evaluation as test data plus grading criteria. Your test set and the right-or-wrong notes are a simple version of that.
05What does a person do, and what does the agent do?
The agent runs the process and can help monitor changes it has access to. A named person owns the review and checks that the monitoring covers changes in policy, inputs and connected systems.
- The agent does the work and, if you set it up, logs speed, volume and the cases it escalated.
- The process owner reads the four signals, runs the test set and decides.
- The sponsor hears about any pause or change in scope.
Plan the response in advance as well. For example, if the correction rate rises for two checks in a row, who looks into it, and by when? Treat this as an illustrative trigger, not a safety threshold; investigate serious errors immediately. Write that rule on the same page.
06Which three questions can you ask this week?
Three questions show whether the slide is under control: is there a baseline, who owns the check, and what does the agent carry forward?
- Do we have a baseline for our running agent, and where is it written?
- Who owns the monthly check, and on what day?
- What does the agent carry into each step, and what is dropped? (Ask your supplier.)
If you cannot answer the first two, that is your starting point. If you are earlier in the journey, our guide on where to start with AI in operations covers the success criteria that become your baseline. Book a 30-minute call if you would like a second pair of eyes on a running agent.
07Frequently asked questions
Why does AI agent performance degrade over time?+
Performance can change as a task's context grows or as inputs, policies, connected systems and models change between runs. These are different mechanisms, and decline is not inevitable. Compare a fixed test set and recent live cases against a recorded baseline, then investigate which part of the process changed.
Why do AI agents get more expensive to run?+
Longer retained history can increase input usage, but the effect on your bill depends on caching, the model and your pricing plan. More cases or more steps per case can also raise costs. Track cost per completed case alongside volume, and ask your supplier what is retained, trimmed and cached.
How often should you check a running AI agent?+
Monthly is a sensible default for a process that runs every day, and the first weeks after launch deserve closer attention. Fix the day and the owner in advance. Check sooner after any change to inputs, rules, connected systems or the underlying model. This cadence is our suggestion, not a standard.
What is context rot in AI agents?+
Context rot is the term Anthropic's documentation uses for a drop in accuracy and recall as the amount of text in a model's context grows. For an agent that carries its whole history into each step, later steps can become less reliable as well as more costly. The practical defense is to ask what the agent carries forward and what gets trimmed or summarized.
Can an AI agent monitor its own performance?+
An agent can log results and detect changes when it has the necessary data and access. That does not establish that its checks are complete or correct. A named person should review the signals, compare a test set with the baseline, inspect live cases and decide what action to take.
Sources and further reading
- Context windows. Anthropic (Claude Platform docs), 2026.
- Monitoring Distributed Systems. Google, 2017.
- Working with evals. OpenAI (API docs), 2026.