Announcing Tokei 11 
Thanks to some major internal refactoring, Tokei has received significant performance improvements, and is now one of the fastest code counters across any size of codebase. With Tokei 11 showing up to 40–60% faster results than tokei's previous version.
Check out the full release post for benchmarks on running tokei and other code counters across different codebases.
Awesome work! What were some of the changes that led to the performance improvement?
Well a lot of the performance came from replacing slice.iter().any(|i| line.starts_with(i)) with aho_corasick::AhoCorasick in a hot loop, as well adding per language memoization for things like AhoCorasick. I also added a heuristic where Tokei essentially splits your file from the start until there's "important" syntax that needs to be tracked across lines, and divides that work into two separate parallel tasks. This means if your file has simple enough syntax it can be counted entirely in parallel.
It's hard to list all or tell which made the most impact on performance, as I essentially tried to make every piece that could be concurrent, concurrent, and then repeatedly benchmarked to see if it actually improved it.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.