How to Check if Your Code Fits an LLM Context Window

Guessing whether your codebase fits into a context window is a special kind of torture. You spend five minutes meticulously copying files into Claude or ChatGPT, only for the AI to tell you it's "too long" or, even worse, start hallucinating because it silently truncated your most important logic.

I got tired of this guessing game, so I built a CLI tool called Tokenazire to stop the madness.

The Workflow

Instead of playing "Token Roulette," the tool handles the math for you:

1. Scanning: It rips through a local folder or clones a GitHub repo if you provide the URL.
2. Counting: It uses tiktoken to count tokens. Since that's what OpenAI uses, it's a close enough approximation for most LLMs to prevent a total meltdown.
3. Visuals: It uses color-coded warnings (green to red) to highlight which files are the bloated culprits.
4. Capacity Check: It calculates the exact percentage of a model's window (defaults to 200k, but you can tweak it) that your project occupies.
5. Noise Reduction: It automatically ignores the usual suspects like .git, venv, and node_modules so you aren't counting 50,000 lines of library code.

The real winner here is the --export flag. Once you confirm the project actually fits, this flag bundles the entire directory structure and file contents into one single text file. You just copy the whole thing and dump it into the chat without manually opening twenty different tabs.

Technical Breakdown

The stack is intentionally boring because it actually works:

• Language: Python
• Tokenization: tiktoken
• UI: rich (for the colors and progress bars that make it feel like a real app)

If you want to stop manually trimming files like it's 2023, you can find the source here:

https://github.com/DeKlain4ik/token-counter

It's an MIT-licensed project, so feel free to tear it apart or contribute.