How Revefi Reduced Its Claude Code Data Agents Spend by 90%

Earlier this year, we had to quickly reduce our Claude spend. We had inadvertently introduced a change that made our costs spiral out of control!

As we dug deeper, we realized that we are not the only ones making this mistake! While working with large enterprise customers on optimizing their AI workflows, we found this repetitive mistake in prompt caching across many AI agents. Hence, this blog. We hope to share our learnings with the AI developer and engineering community.

Prompt caching is quite misunderstood, especially the practical aspects of how to leverage it. Mistakes with prompt caching are costly to the tune of seven figures and more for large spenders.

We identified ~$300K in annual savings on a single internal agent for one of our customers.

Addressing Enterprise AI Cost & Performance Challenges

As an agentic AI platform, Revefi helps customers understand and maximize the ROI of their data and AI spend. Raden, our AI agent for data, takes control and manages your data warehouses (e.g. tasks like optimizing warehouses, auto detecting and root-causing data quality issues, performance tuning to name a few). Raden uses Large Language Models (LLMs), in addition to some of our own algorithmic optimizations, and unsupervised machine learning models at scale.

Our customers are Fortune 500, and Global 2000 enterprises with large data footprints and, unsurprisingly now, very large & fast growing AI footprints. Given our scale and customer base, we needed to answer these questions:

• Is Raden spending $100 on LLM requests to solve a $1 problem?
• Is it delivering quality outputs to our customers?
• What is Raden’s end-to-end latency and performance for different customers?

We quickly realized that the same problems plagued our customers’ agents as well. Late in 2025, we started investing in AI observability & optimization with Raden as the immediate beneficiary. This has since become the foundation of our AI FinOps, observability & optimization offering.

Evolution of Revefi’s Agentic AI

In June 2026, we revamped Raden’s architecture to support shared streaming sessions, tiered memory for enterprises, and a continuous feedback loop. We moved to the latest LangChain libraries and middleware architecture, rebuilt prompt assembly, and added memory & context injection. The new features were a hit.

In parallel, we shipped prompt-caching metrics and alerts in our AI FinOps offering. To our surprise, Raden’s cache hit rate was abysmal.

We found the ‘why’ and fixed it. But then we asked another question: This seemed like an innocuous change - if our team could ship this, how many of our customers made the same mistake? Keep reading for more.

Brief Explanation of Prompt Caching

Consider an AI agent or chatbot that calls the LLM each time you (the user) post a message. Every call to an LLM re-sends the entire conversation, such as your tool definitions, your system prompt, and every message so far. The model re-processes all of it before generating a single output token.

Most of that input is identical from one call to the next, so the providers (Anthropic, OpenAI, Google) let you cache it. Caching is prefix-based, that is when a request’s prefix matches a recently cached one, the provider skips re-processing those tokens and charges a fraction of the normal input price. Generally, cached input tokens cost ~0.1x the price of regular input tokens (0.025x on Claude Fable 5.1). Here’s a graphic showing how the pricing works:

That makes caching a big lever for agentic systems, and a huge one for long running agents like Raden (where one user request triggers dozens of LLM turns). As the agent executes, the cached proportion of every request climbs as the session goes on. Anthropic’s caching launch post quotes 53% to 90% in cost savings depending on the workload.

Long agentic sessions sit at the top of that range.

The Bug: Our Volatile System Prompt

As part of the upgrade, our system-prompt gained two additions:

• The current time
• The user’s context from our UI

Both were crucial for our agent’s quality. Knowing the current time is critical for a data agent, LLMs are trained on historical data and will happily assume it’s still 2024. The user context, like the URL of the page currently open, lets Raden provide answers that are context-aware.

These changes, however, made the system prompt volatile. The user context changes occasionally, where the timestamp changes on every single LLM call. With prefix caching, only the tool schemas and the stable part of the system prompt ever hit the cache. The conversation history, the bulk of every request, was re-billed at full price every time.

The Fix

We needed to keep both the additions, especially the current timestamp. We learnt from our peers in the industry, and made changes to our prompt generation layer where the least volatile content comes first, and the most volatile last.

For Raden that meant a “system reminder” appended to the end of the request, alongside the latest message in the conversation, carrying the user context and then the current timestamp. Everything above it (tools, instructions, the whole history) stays byte-identical between calls and hits the cache. Even the slow-changing user context is cached most of the time, because it sits above the timestamp.

Here’s the impact as seen on our Anthropic Console the week it shipped. Cache reads climbed to 99% of input tokens.

A note on latency: Anthropic’s benchmarks report 31% to 79% lower latency on long cached prompts. We didn’t notice this benefit with Raden. Further, tool calls dominate our wall clock time, meaning that our end to end latency is dependent on other factors.

Taking the Learnings to Our Customers

Our customers are already pushing their agents' telemetry to Revefi to track their LLM spend and dollar efficiency.

To bring this to the forefront, we added cache efficiency analysis to the product: per service, per agent, per model. Here’s what we found:

• The top agents by spend, and the external tools, were fine.
• The fifth, sixth or seventh agents were not. These agents lower down the pecking order had cache read ratios near zero, burning money as silently as we had been.
• Breaking it down by model sharpened the picture. At one customer, Anthropic traffic specifically wasn’t cached. In other words, cache markers were not set. Unlike OpenAI and Gemini, Anthropic makes you opt in, either with a single top-level cache_control field (automatic breakpoints) or by placing “cache markers” (cache_control: {"type": "ephemeral"}) on specific blocks of the request.

Impact: For a single customer, this resulted in approx. $300K annualized savings on a single internal agent.

If there’s one thing you should take from this blog, it’s this

Prompt caching failures don’t throw exceptions. They throw invoices.

What Should You Do?

If you’ve gotten here, you’re clearly interested in acting on this knowledge.

Start with the provider dashboards:

• Anthropic: the Console has a Caching view under Usage with basic hit rate metrics. Docs.
• OpenAI: the usage dashboard breaks out the caching metrics. Caching is automatic on prompts over 1,024 tokens (GPT-5.6 and later). Docs.
• Google: Gemini reports usage.total_cached_tokens in usage metadata. Implicit caching is on by default for Gemini 2.5 and newer. Docs.

Of course, you can take the easy route where Revefi monitors and alerts on prompt caching at every granularity, and provides many recommendations out of the box (TTL recommendations, unnecessary caching etc).

Reach out to contact@revefi.com if you’re interested as it’ll take just a few minutes to start pushing your telemetry to Revefi and get the full benefits.

Other Key Learnings

Here are some additional things to know:

There are a additional cases beyond “missed” caching to think about:

• Unnecessary caching. If you only use an LLM for a single request/response pair and never continue the chat, you’re just paying extra for caching.
• Cache TTL. Not every agent needs the 1 hour TTL at 2× write cost. The 5-minute option at 1.25× is often enough. Autonomous agents (with no humans in the loop) make their next call within seconds.

• Don’t abuse the ephemeral content in LLM calls. The system reminder suffix is a great idea but don’t overuse ephemeral content that disappears across LLM calls. The model can find it hard to reason why a tool call was requested by it previously. This can lead to very weird behavior.

References

• Anthropic, Prompt caching (developer docs): cache_control, breakpoints, TTLs, invalidation rules.
• Anthropic, Pricing: prompt caching: 1.25× / 2× write and 0.1× read multipliers.
• Anthropic, Prompt caching with Claude (launch post): 53-90% cost and 31-79% latency reductions by workload.
• Anthropic, Effective context engineering for AI agents.
• OpenAI, Prompt caching (API guide): automatic caching, 1,024-token minimum, cached_tokens reporting.
• Google, Gemini API context caching: implicit vs. explicit caching.
• LangChain, Agent middleware.
• Anthropic, Claude Code.
• Manus, Context Engineering for AI Agents: Lessons from Building Manus (Yichao “Peak” Ji, July 2025): KV-cache hit rate as the key metric for a production agent; a timestamp at the start of the system prompt named as a common mistake.