I built a small CLI, tokcost, that counts tokens and estimates cost for
LLM text — files, stdin, or a live-wrapped command. Cargo.toml has an
empty [dependencies] section: the BPE encoder, JSON output, arg parsing,
and the live ticker's ANSI are all hand-rolled.
Why zero deps
Mostly because the alternative for exact token counts today is
pip install tiktoken or shelling out to Python, and I wanted something
that's a single static binary with no runtime to bring along. The two
real tiktoken vocabularies (cl100k_base, o200k_base, ~300k merge rules
combined) are embedded at build time from the canonical rank files and
checked in CI against real tiktoken output.
The one design choice worth discussing
The pretokenizer doesn't reproduce tiktoken's split regexes boundary-for-
boundary — it's deliberately coarser in one place. Real o200k_base cuts
a word at every case transition (myVariable → my | Variable); mine
keeps the whole alphabetic run as one piece. That's safe rather than
approximate: BPE vocabularies are trained on text that's pretokenized by
this same regex first, so no merge rule in the shipped rank tables can
ever bridge a boundary the real splitter always draws. Coalescing across
it can't change the merge sequence — there's nothing there to merge.
Splitting finer than the real regex would be unsafe (it can suppress a
real merge); I only ever go coarser. Golden tests check the final token
IDs against real tiktoken output on ~40 cases including camelCase and
case-folded contractions.
Repo (MIT): GitHub - abhiprd2000/tokcost · GitHub , crate : crates.io: Rust Package Registry
cargo install tokcost
Genuinely interested in pushback on the pretokenizer approach above, and
in clippy/idiom nits — this is a smaller codebase than most things I post
about, so it's a good one to get right.