Do you know what your Claude Code is doing? (Part 1)

Key Takeaways

• The blog describes how AI agents can perform tasks in inefficient ways. When these tasks are repeated, they can be extremely costly and make the agent slow.
• In our example, we ask Claude Code to address comments on pull-requests. Each time Claude deals with a large GitHub API response and then writes the same incomplete Python script to process it.
• A short skill and checked-in script that builds the unresolved comment threads makes the task 8x cheaper.
• In Part 2, we cover Revefi’s solution to identifying and addressing these agentic inefficiencies.

The utopia

In January 2026, our team - like much of the software world - was on a mission. Use Claude Code for everything. As part of this effort, we added a Claude skill to our mono-repo that automatically addresses Github pull-request (PR) comments! To us, this was a step towards agentic utopia: A reviewer bot leaves comments on the pull-request, then Claude Code uses our skill to address the comments, and pushes fixes. Repeat.

Engineers only need to review the final improved pull-request.

Some of you may not agree with this version of “utopia”. You’re in luck! This blog isn’t about the goodness of the utopia. It’s about what was happening behind the scenes.

An observation

Soon after, we noticed a pattern. Claude kept creating a python script to parse the API response from Github. Similar script every time. It felt like the agentic equivalent of code smell. One day we decided to ask why.

What GitHub gives you

To understand what was going on, let’s look at the Github API response when querying for the comments on a pull-request:

You’ll notice that the API response has a few issues:

1. It is big and almost all of it is noise for our purpose. As an example, for one of our PRs, we saw 6.9 KB per comment, 264 bytes of which is the comment. 31 comments on that PR, 215 KB (63k tokens) in total.
2. It does not say what is already resolved. Resolution state lives in a different GitHub API.
3. A little less obvious, the response is a flat list. You’ll notice the field in_reply_to_id in the response. This field captures the nested comment thread structure. The response has to be parsed and traversed to view the comment threads correctly.
4. To add more comments to an existing thread, you need the root comment id - requiring parsing and traversal.

As we mentioned, Claude wrote python scripts to handle the Github response. The script trimmed the response and invoked the additional API to identify resolved comments. But it did not build the comment threads! Instead, the thread building was happening in the model's head.

The language model was basically doing graph traversal.

The “fix”

We seriously questioned this behavior and immediately extended our skill with a checked-in python script. The script fetches the comments, builds the threads, drops the resolved ones, and hands Claude the built comment threads that need to be addressed.

The experiment

Of course, we wanted to know if the script was actually better. Is it faster? More importantly, does it consume fewer tokens?

To answer these questions, we designed a small experiment. We picked a representative pull-request and ran multiple headless Claude Code sessions for three different scenarios:

• Raw JSON. Claude code is not allowed to write scripts. The gh API response is all that it can use.
  • Note that this category is actually quite relevant to SaaS agents. These agents usually operate without VM sandboxes to execute scripts, so tool responses can end up directly in the context window.
• Ad-hoc script. In this scenario, Claude is free to write a jq or Python to process the output.
• Checked-in script. Our checked-in skill + script. The script builds the tree and filters resolved threads; Claude reads the script’s output.

Here are the results. (You can also see the source code for this experiment at: github.com/revefi/sample-code-blogs)

The checked-in script was 8x cheaper (at API pricing) than raw JSON, 3.8x cheaper than the ad-hoc script, and at least 5x as fast as both.

If you consider a 20-developer team shipping say, 2,000 PRs a month, with a review bot leaving inline comments on nearly all of them.

The checked-in script can save $800 to $1,000 a month in Opus tokens against the raw output scenario.

Even if you’re on a team plan and don’t care about API pricing, you probably care about usage limits and the waiting time.

The good news

The good news however is that a lot has changed since January.

GitHub's MCP server

• Github MCP server is much better now - with greater feature parity to the APIs. (One of the reasons why we chose the CLI + API over using the MCP server).
• The MCP tool returns the trimmed comment threads, not a flat list like the API.

Claude Code

• The models have gotten better. They don’t spend as much time verifying their scripts.
• Claude Code now has much better handling of large tool results.

At this point you might ask, “Is this still a problem?”. GitHub and Claude Code have already fixed most of the issues here.

Now, for the bad news….

The bad news

Similar problems still exist with other GitHub MCP tools and APIs - CI run statuses, bloated job failure logs, approvals on stale commits etc. But this isn’t a GitHub problem. Here is how Claude Code creates jira tickets:

This isn’t just a Claude Code problem either. Any MCP server or tool you use, any AI orchestrator that you use or agent that you build - these problems exist.

What can you do?

There are simple learnings from this blog itself:

• Don't just throw an MCP server on top of your existing APIs. Design MCPs to be natural for LLMs to understand.
  • A special case is graph/tree structured data (as we saw earlier). Modeling such data as a flat list is great for an API but not for an LLM.
• Think twice about tool responses directly ending up in the LLM’s context.
• Use the latest models where possible - they are actually better.

But the biggest change you can make, both as agent builder and consumer, is giving the agent better instructions - skills, skills with scripts and instructions (think CLAUDE.md files etc.). That brings us to the real question:

What are my repeated tasks and what skills do I write?

We’ll cover these in Part 2 🙂.