Cai - The fastest CLI tool for prompting LLMs

I set myself the goal to build the fastest AI for the command line!

Of course by first using the fastest language and also the fastest LLM right now:
Mixtral powered by Groq at up to 500 tokens per second!

And the first results are already quite impressive! :sunglasses:

You can find the installation instructions here:


Fast, however, means to get the final result fast and sometimes you need more than one model. So today I added support for explicitly specifying a provider:

Which additional features would you like to see? :grinning:

2 Likes

And I just released a new version of cai! :tada:

Including a feature I wanted for a long time:
Prompting several LLMs at once! :sunglasses:

v0.6.0 is out! :tada:

Now with support for all Groq and Ollama models!