Hi Rustaceans,
I’m the maintainer of Ferrum, an MIT-licensed local LLM inference runtime
written in Rust. Version 0.8.4 is now available.
Ferrum aims to make running and serving local LLMs feel like installing a
normal CLI:
- One Rust binary
- No Python, PyTorch, or vLLM runtime
- Metal acceleration on Apple Silicon
- CUDA support on NVIDIA GPUs
- An OpenAI-compatible chat completions API
Apple Silicon quick start:
brew tap sizzlecar/ferrum
brew install ferrum
ferrum doctor qwen3.5:4b-q4_k_m
ferrum run qwen3.5:4b-q4_k_m --disable-thinking
The first run downloads approximately 2.55 GiB of model files.
Start the API server:
ferrum serve --model qwen3.5:4b-q4_k_m --served-model-name ferrum --disable-thinking --port 8000
Call the OpenAI-compatible endpoint:
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"ferrum","messages":[{"role":"user","content":"Hello from Ferrum"}]}'
Links:
The project is still early, so I’d especially appreciate feedback about
installation, the first model download, Metal/CUDA compatibility, and API
behavior.
v0.8.6 is out: Qwen XML tool arguments now preserve indentation and trailing newlines, and structured values are restored using the declared JSON schema. This matters for coding tools that send code or nested objects as arguments; I'd welcome small failing request examples from anyone testing the release.
Release notes: Release Ferrum v0.8.6 · sizzlecar/ferrum-infer-rs · GitHub
Maintainer update: Ferrum v0.8.8 is available.
Fresh run and serve downloads now select the shards named by a root SafeTensors index instead of also pulling alternate weight formats. This release also fixes standalone Hugging Face template loading for Qwen packages and keeps download progress off stdout when using run --output-format jsonl, so scripts can parse the output from the first run.
For new Homebrew 6 installations, the updated quick start includes the formula-trust step before installation. If you try the release, a small failing request or the first unclear installation step would be useful feedback.
Ferrum 0.8.9 is available: local text generation through a Rust CLI, an OpenAI-compatible API, or an engine embedded in your application.
I maintain Ferrum. The CLI and HTTP server share the same engine. If your application should own the model lifecycle, the complete in-process Rust example shows construction, generation, streaming, and shutdown using the v0.8.8 library API.
This release adds a native Windows CUDA installer for NVIDIA RTX 40-series GPUs. It bundles CUDA and VC runtimes, adds Ferrum to PATH, and requires a compatible NVIDIA driver. Apple Silicon Macs and Linux x86_64 also have one-command installation. Re-running the installer upgrades Ferrum while existing processes keep their loaded version; new launches use the update.
Applications can execute model-requested tools, return the results, and receive a final answer constrained by their JSON schema. Reasoning controls follow the loaded model’s capabilities and templates. Incomplete model caches recover through both run and serve.
Installation instructions · Release notes and downloads
I’d welcome feedback on integration and first-prompt behavior.
Disclosure: drafted with AI assistance and checked against the release.