Rust for AI Agents

code review
1

I've been working on ADK-Rust - a production-ready framework for building AI agents in Rust. After a year researching frameworks (LangChain, LangGraph, OpenAI SDK, Autogen, Google ADK) and writing books about them, I found they all compromise between simplicity and power and tilt towards vendor lock-in.

ADK-Rust combines the best patterns into one open source modular framework:

image
image1675×961 54.1 KB

Check out the GitHub repo and let me know how i can improve it or any serious design issues from an architecture perspective. GitHub - zavora-ai/adk-rust: Rust Agent Development Kit (ADK-Rust): Build AI agents in Rust with modular components for models, tools, memory, realtime voice, and more. ADK-Rust is a flexible framework for developing AI agents with simplicity and power. Model-agnostic, deployment-agnostic, optimized for frontier AI models. Includes support for real-time voice agents.

4 Likes
4

I am quite interested in what you are doing with an ADK implementation in Rust. It would be great if there was support for local models though. I have successfully integrated local models with Google's ADK using the LiteLlm support.

1 Like
5

Thank you for the feedback and we very much appreciate it. Local models now supported in new release v0.2.0

We have implemented local models through two main approaches:

1. Ollama Integration

The easiest way to get started with local models. ADK-Rust has native Ollama support, so you can run any model available through Ollama (Llama, Mistral, Gemma, etc.) locally.

See: Ollama Documentation

For Ollama (no key, just run: ollama serve) and simple project like the one below:

use adk_rust::prelude::*;
use adk_rust::Launcher;

#[tokio::main]
async fn main() -> AnyhowResult<()> {
    dotenvy::dotenv().ok();
    // Requires: ollama serve && ollama pull llama3.2
    let model = OllamaModel::new(OllamaConfig::new("llama3.2"))?;

    let agent = LlmAgentBuilder::new("assistant")
        .instruction("You are a helpful assistant.")
        .model(Arc::new(model))
        .build()?;

    Launcher::new(Arc::new(agent)).run().await?;
    Ok(())
}

2. mistral.rs Integration

For high-performance local inference, we've integrated mistral.rs directly. This gives you native Rust performance for local model execution without external dependencies.

See: mistral.rs Documentation

For native local inference without external dependencies, use the adk-mistralrs crate:

use adk_mistralrs::{MistralRsModel, MistralRsConfig, ModelSource, QuantizationLevel};
use adk_agent::LlmAgentBuilder;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Load model with ISQ quantization for reduced memory
    let config = MistralRsConfig::builder()
        .model_source(ModelSource::huggingface("microsoft/Phi-3.5-mini-instruct"))
        .isq(QuantizationLevel::Q4_0)
        .paged_attention(true)
        .build();

    let model = MistralRsModel::new(config).await?;

    let agent = LlmAgentBuilder::new("local-assistant")
        .instruction("You are a helpful assistant running locally.")
        .model(Arc::new(model))
        .build()?;

    Ok(())
}

Note: adk-mistralrs is not on crates.io due to git dependencies.
Features: ISQ quantization, PagedAttention, multi-GPU splitting, LoRA/X-LoRA adapters, vision/speech/diffusion models, MCP integration.

2 Likes