AI Token Optimization
Photo by Alexander Mils on Unsplash
Exploring technologies for optimizing token usage, because AI can get very expensive very fast
Quick Note
I originally wrote this to explore optimizing my GitHub Copilot subscription, which has shifted to usage-based billing, meaning higher token consumption accelerates the depletion of your pre-paid balance. However, my approach has evolved since moving to Google's Anti-Gravity CLI. This shift also meant moving away from OpenCode, except when using locally hosted models, as Anti-Gravity is unsupported in those tools due to its restrictive terms of service.
While I frequently mention Copilot and Anti-Gravity, the tools I discuss also support popular platforms like Claude Code and Cursor. I enjoy exploring niche tools to see what is available; they often implement similar features, and when I notice a trend, it usually indicates that the concept has “caught on.” Smaller tools often push distinct use cases to differentiate themselves—for instance, Copilot’s tendency to leverage the GitHub ecosystem versus Google’s focus on the agent experience. I won’t dive into those specifics here; I will save them for a future post.
Agentic AI development is gaining traction, yet many companies remain in the “figure it out” phase, investing heavily in employee exploration. This trend often triggers Jevons Paradox, where technological advancements that increase resource efficiency ultimately lead to an increase in total resource consumption. Consequently, companies use AI to automate tasks, realize efficiency gains, and end up consuming more tokens because the technology proves effective. That said, not all usage is equal, and analyzing its effectiveness is a complex topic in itself.
As software engineers, we are taught to optimize inefficient processes. Relying solely on additional computational resources is often less effective than optimizing the resource utilization itself. I believe this same logic should—and will—be applied to token utilization. Deliberately optimizing token usage allows users and companies to maximize their value.
But first, let's establish a baseline: what exactly is a token?
What is a Token?
“AI tokens are tiny units of data that come from breaking down larger chunks of information. AI models process tokens to learn the relationships between them and unlock capabilities including prediction, generation, and reasoning.” (Ref: NVIDIA Blog) In the context of this blog, tokens represent the unit of 'information' consumed by AI agents. This unit underpins most pay-as-you-go cost structures, where increased AI usage—and the resulting computational resource consumption—directly correlates with higher costs. This structure reflects the provider's operational costs, which scale with compute intensity, data center utilization, and power consumption.
In the consumer AI space, most models are subscription-based rather than pay-as-you-go. This also applies to several developer tool subscriptions, such as Claude Code Pro, where the most common entry-level price is around $20 per month. These plans often come with usage restrictions and background throttling to prevent abuse. This model suits end-consumers who use AI sparingly for complex reasoning tasks or light automation, as the limits prevent power users from exceeding budgets.
While providers might lose money on a monthly subscription model, encouraging AI adoption offers other benefits. For example, the Gemini AI Pro subscription includes YouTube Premium Lite and expanded Google Drive and Photos storage alongside improved usage quotas. This integration provides better value as a consumer and deepens your adoption of the Google ecosystem.
Conversely, developers often have complex use cases that require more extensive AI compute than a web browser chatbox can provide. Consequently, most providers offer pay-as-you-go models, which are favored by enterprise companies to avoid developer productivity bottlenecks caused by throttling. In these instances, enterprises pay for additional tokens to ensure they have the necessary compute power.
Regardless of the model, you are ultimately tied to your token usage. Even for consumers with quotas, those limits are typically derived from token consumption. Therefore, to maximize your quota, the most effective strategy is to reduce token usage—whether you are on a pay-as-you-go plan or allotted a specific window of time.
There is a clear financial incentive to optimize token usage. As the software industry increasingly shifts toward agentic workflows, optimizing how we pay for and utilize these agents will become a significant concern. While the potential optimizations are numerous, they are also highly non-deterministic; optimizing AI usage is often fuzzy and situational.
As this is a developer-focused blog, I will only cover tools for improving developer-centric token usage. I am neither aware of nor interested in consumer-level tools.
Like any optimization project, you need a baseline, and a baseline requires monitoring.
How to Monitor Token Usage
If you are working in an enterprise environment, AI providers such as Anthropic provide tools and utilities to understand costs and high-level usage patterns. Again, because AI usage is flexible and highly dynamic, consumption is likely to rise as teams test the technology against new use cases.
I will focus on individual developer options rather than enterprise tools. For consumer-level AI, providers typically offer dashboards to help you monitor usage and track how much time remains before your rate limits reset. For developers, however, their needs are more nuanced, as you likely use multiple providers and tools. Here are the options I have been using:
Personal Pick: Codeburn
codeburn Codeburn is a CLI tool that tracks usage across multiple providers, including Claude Code, Cursor, Codex, GitHub Copilot, and OpenCode.
It works entirely locally by leveraging each tool's internal tracking data. I recommend it because it is easy to set up, supports most tools, and provides a simple CLI interface with cost estimates. While I am waiting for better support for Google’s Anti-Gravity CLI, the GitHub Copilot support has been excellent. It is particularly useful now that GitHub Copilot has shifted to a usage-based model rather than a per-request model; each request draws against my subscription, effectively meaning I pay for each token in advance.
Personal Pick 2: Tokscale
tokscale I am new to Tokscale, having started using it primarily because Codeburn does not currently support the Anti-Gravity CLI.
This tool is similar to Codeburn but features a different UI that focuses more on graphs over time and model usage. It also provides more aggressive cost calculations, including estimates for LM Studio-powered usage, which is effectively free aside from energy costs.
Improving Token Usage
Once you have insight into your token usage, you can begin optimizing. Many developers have noted that AI-consumed tokens can be reduced by "offloading" data into other tools. While passing plain text to agents works, structured information can often be compressed or improved without losing context.
For example, a long log file with repeating characters can be condensed into a single line noting the repetition count. Some tools do this automatically, but others go further to save tokens.
Tool: RTK
rtk RTK is my go-to choice for token optimization. It compresses common tasks before sending them to the AI by utilizing the agent harness's hooks API to intercept and modify the data. By compressing the data between the tool and the agent, the agent consumes less compute, saving you tokens and money.
A full list of examples is available on GitHub. RTK provides insights into savings, but support varies; while it has first-party support for tools like Copilot, newer tools like Anti-Gravity often receive secondary support.
Tool: Headroom
headroom Headroom is similar to RTK as a "compression" tool that uses the hooks API to intercept data. However, Headroom covers more use cases. While RTK focuses on shell commands, Headroom manages more unstructured input and output, which can lead to greater overall cost savings than RTK alone.
Tool: Token-Optimizer
token-optimizer This is the "better" option according to its documentation. I haven't had the opportunity to utilize this extensively, but it is the starting point I would use if I were returning to Claude Code or Copilot.
Going Beyond Optimization: Self-Hosting
If you want to bypass external providers entirely, you can run and host your own models locally. There are three main caveats to this approach:
- The models you can run will likely not perform as well as cloud-based models.
- Performance depends heavily on your hardware; dedicated GPUs or Macs with unified memory allow for higher-end performance, provided you stay within your machine's limits.
- You must manage the setup and maintenance yourself.
Running the Model
What model can I run?
Depending on your architecture, you might see great performance for basic tasks, or you might struggle to run even a small model. I highly recommend checking CanIRun.ai to see what your hardware can handle.
In short, what you can run depends largely on what fits into your GPU’s VRAM or, if applicable, your machine's integrated memory. You want a model that performs decently for coding, otherwise, you will spend significant time waiting for computations.
Ollama
ollama Ollama is the most common tool for this; it provides a high-level interface that allows you to browse Hugging Face and run models. Its interface is primarily terminal-based, though GUI versions exist. I recommend Ollama if you want a simple, effective, and well-documented setup.
Ollama allows you to pick hugging face models and run them, just like the other options in this list. Its very easy to use and very commonly supported even if it has some extra overhead that not everyone appreciates, hence the alternatives I listed below.
LM Studio
Lm studio LM Studio is a more robust GUI option that performs the same role as Ollama. Use this if you prefer a visual, out-of-the-box experience. There may be a performance hit from running the UI, but this is usually negligible unless you are on an under performing machine. I usually go with this, primarily so I have the easiest time setting this up on my desktop where it just runs LM Studio and task manager to observe VRAM usage while I use another machine entirely.
Or on my MacBook pro, I run an instance of LM Studio on the same machine, this works but isn’t as fast or as nice as just having it run on another machine.
Llama.cpp
llama.cpp Llama.cpp is the "lean" option used under the hood by LM Studio. I don't use this directly but had to mention it, but it is the best choice if you want to squeeze maximum performance out of your setup.
What Models to Run?
I am not an expert in model performance, but there are a few concepts worth considering based on your available hardware.
Quantized Models
Quantization is an optimization technique that adjusts model weights so they consume fewer resources. A higher quantized model fits into less VRAM/RAM, making it easier to run, though it may be less “accurate” because it approximates more.
Mixture of Experts (MoE)
A Mixture of Experts splits a model into different "experts." A larger model might have 8 billion parameters, but only 4 are active at any given time. This keeps only a portion of the model in VRAM, while the rest resides in system RAM and is loaded as needed. While naming conventions vary, common ones like "Mixtral 8x7B" indicate 8 experts, each with 7 billion parameters.
Tool Calling Support
For coding agents, you need a model that supports tool calling. This allows tools like OpenCode to execute file changes or terminal interactions. Without this, your model will seem "lazy"—it will provide information but never execute actions.
Using the Model as a Coding Agent
OpenCode
opencode OpenCode attempts to be an "open" version of Claude Code. It supports similar configuration files and general capabilities but is compatible with multiple providers and models, including Anthropic. It is a popular choice due to its out-of-the-box functionality and support for cloud-based features. While I used this extensively instead of the Copilot CLI, I now use it strictly for local model usage since I moved to the Anti-Gravity CLI. Now that you have a model running, the next step is leveraging a tooling harness. There are many options, but I will focus on two mainstream picks.
Pi.dev
pi.dev Pi.dev is a leaner harness compared to OpenCode. It supports fewer standards but is highly extensible with plugins. It is an excellent choice if you have lower-end hardware or prefer a minimalist setup. I personally haven’t used this too much besides installing it and checking it out a bit, but again I wanted to list out alternative options.
Using Models for Other Tasks
While this post focuses on coding agents, you could theoretically use a similar setup for other agentic, non-coding tasks like Hermes Agent or Open Claw. I do not cover them here because I do not use them (or fully trust autonomous agents yet). These local setups are helpful, but I believe they are best reserved for smaller tasks where their lower "intelligence" can be offset by the fact that they are free to use repeatedly. Better hardware leads to better results, but unless you have invested in a large GPU setup, you will likely miss the raw performance of cloud-based models.
In Summary and my setup
My personal setup relies on using my desktop computer, which was built for flight simulation and Kerbal Space Program 2 years ago. It has a decent GPU capable of running some Gemma 4 models with moderate quantization. It is quite fast, often hitting 100 tokens per second for simple tasks. However, its "intelligence" is limited, often requiring significant hand-holding or repeated retries for basic tasks. I plan to upgrade my graphics card in the future, but until prices stabilize, I will wait.
This is the longest post I have written and covers one of the most extensive topics I have explored over the last few months. I hope it has been worth the wait!
Bonus
If everything you just read sounds great, and you want to see how your device runs AI directly yourself but are on the fence. I’ll be posting about something shortly that solves this problem. It’s one of the projects I’ve been working on behind the scenes, along with this post. It’s also one of the reasons why I stopped updating my blog. But this is for next time, until then, keep learning, keep growing and keep asking questions!