Introduction
In July 2026, a developer and entrepreneur published a startling analysis on Habr that challenges how we think about AI agent costs. After building a custom CLI tool to trace every API call across his multi-agent workflow, he discovered that 89% of his spending on AI agents went to cache retrieval, not actual generation. The finding upends the common assumption that most costs come from model inference. Instead, the majority of money is eaten by redundant data fetching—repeatedly pulling the same context, logs, and intermediate results from memory stores. This article unpacks that news, explains the technical breakdown, and offers actionable insights for anyone running AI agents in production.
The Discovery: A CLI That Exposed the Truth
The developer, running a stack of autonomous agents for tasks like code review, data extraction, and report generation, noticed his monthly bills were climbing faster than expected. He assumed the culprit was expensive LLM calls (e.g., GPT-4o or Claude 3.5 Opus). But when he wrote a lightweight command-line tool to tag every API request with a category—"generation" vs. "cache"—the data told a different story.
| Category | Percentage of Spend | Description |
|---|---|---|
| Generation | 11% | Actual LLM inference (prompt + completion) |
| Cache reads | 89% | Retrieving cached prompts, vector DB lookups, session state, and intermediate outputs |
Over a 30-day period, the tool logged 340,000 cache reads against 12,000 generation calls. The cache operations were cheaper per call, but they dwarfed generation in volume. The result: 89% of the total dollar spend went to cache, not to the AI models themselves.
Why Cache Costs So Much
Cache is often overlooked because it's not "visible" in the same way as an LLM API bill. But in a multi-agent system, cache isn't just a simple key-value store. It includes:
- Vector database queries (e.g., Pinecone, Weaviate) for retrieving relevant context chunks.
- Session state caches (e.g., Redis) that store conversation history and agent memory.
- Intermediate result caches for pipeline stages—like parsed JSON outputs or tool call results.
- Prompt template caches that store precomputed embeddings or formatted strings.
Each cache call might cost only $0.0001, but when agents make 10,000 cache reads per day, the bill adds up to $1 per day—or $30 per month—just for cache. Meanwhile, generation might cost $0.50 per day. The developer's CLI revealed that his agents were re-fetching the same vector embeddings dozens of times per conversation, often for redundant context that never changed.
Practical Implications for AI Practitioners
This finding has direct consequences for anyone building AI agents in production:
- Don't assume LLM cost is the bottleneck. Monitor your entire stack, including vector DBs, session stores, and middleware.
- Cache aggressively—but smartly. Use TTLs (time-to-live) to avoid stale data, but also implement deduplication to prevent repeated identical fetches.
- Profile before optimizing. The developer's CLI is a reminder that intuition is often wrong. Instrument every API call with a simple tag (generation/cache) before deciding where to cut costs.
- Consider local caching layers. For high-frequency reads, a local in-memory cache (e.g., SQLite or Redis on the same node) can slash latency and cost.
Real-World Example: The Redundant Context Problem
One concrete case from the developer's logs: his code-review agent fetched the same 50-KB chunk of a repository's README file 47 times in a single session. Each fetch cost $0.0002 from a vector DB. Total: $0.0094—small, but multiplied across 30 agents over a month, it became $8.50 in wasted spend. After implementing a simple local LRU (Least Recently Used) cache, those reads dropped to 3 per session, saving 94% on that line item.
The Tool Itself
The CLI, released as open source on the developer's GitHub, is a simple Python script that wraps any API client (OpenAI, Anthropic, Pinecone, Redis) with a decorator that logs latency, cost, and category. It outputs a CSV file that can be loaded into any analytics tool. The developer suggests that similar tools exist (e.g., LangSmith, Helicone), but his approach is lighter and more focused on cache vs. generation breakdown.
For teams using platforms like LangChain or AutoGPT, the same instrumentation can be added by monkey-patching the HTTP client. The key insight is not the tool itself, but the mindset: always measure before you optimize.
Conclusion
The Habr article is a wake-up call for the AI engineering community. The assumption that "most spend is on LLM inference" is, at least for this developer, completely backward. 89% of his AI agent costs went to cache—meaning that optimizing generation alone would barely move the needle. The real savings lie in reducing redundant data retrieval, smarter caching strategies, and better pipeline design. As AI agents become more ubiquitous, cost observability tools like this CLI will become standard. The lesson: measure everything, trust nothing, and always question where your money is really going.
Comments