New week, new Rust! What are you folks up to?
Currently taking a small break from my main project 8080_msg by trying to build a simple linear equation solver.
First step is thinking about how to translate the "human" algorithm of solving linear equations to what a computer can execute.
Just released transactional outbox addition for the historia - event-driven framework I'm working on. It's a separate crate which might run as either a binary (if you're fine with how it works out of the box) or a library (if you'd like to modify some behavior or just plug it into the main service).
Been experimenting with Ollama's local server and doing json tool calls with serde and reqwest. Not intended as a full agent harness, but small models for 4GB GPU to read docs and answer math / protocol questions, or organize data etc
I'm not someone that uses AI to write code, but I do use it as a tutor or research assistant. Ollama and local models in the 30b parameter range.
I currently have an extra 6gb GPU that I'd like to experiment with using small models. What's been your experience so far with that 4gb GPU?
It's hard to find concrete information about the model designs sometimes, so I've mostly experimented with ollama's available models in that range. IBM granite4.2:3b is a good standard to start with, I was getting about 21 t/s (faster than reading speed) and I like how the training for the model was described. The default running context was 4k tokens and could expand to 12k and still run on the GPU 4gb at that rate, and 16k it shifted some of the layers to CPU and dropped slightly to 19-20 t/s.
The lfm2.5-thinking:1.2b was interesting for me because you could add much more context, 48k didn't even change the starting rate at 55-60 t/s, so this is like the "doc reader" option, also tool capable with json
And then gemma4:e2b was pleasantly confusing in that it's the largest model, but still runs faster than the granite4.2 despite the fact that more layers are offloaded to the CPU. It was running about 26 t/s. With these smaller models though, personally I don't use them for the quantized training weights. I'm an amateur of course but think these < 12B models are better for situational "1-off" questions based on tool-call searches. For 6GB GPU I'd recommend trying gemma4:e4b
Awesome! Thanx for the input!!!
I initially was surprised to find that the granite and gemma models you mentioned are not MoE. OTOH, PLE and the IBM Hyrbrid Mamba approaches sound amazing.
I'm an amatuer also, but some writing is on the wall. Local AI is likely to become more of a question over time. If/When the AI bubble burst completes, and hopefully memory prices return to something reasonable, local AI setups using small models and "smaller" gpus are likely to blossom.
Thanx again for the input.
Work on my fan controller has essentially stabilized. Before moving on to support for multiple GPU's AND GPU specific load states, I need to clean up the abstractions and create an install script.
Also trying to update functionality in Blackbox Background (bb_bg). Who remembers Blackbox and Windowmaker? Pepperidge Farms does. LOL.
Anyway, it supports Cinnamon, KDE Plasma, and anything (mostly) that runs wmsetbg to set backgrounds. I started trying to update it to support multiple displays (put a different image on each) when using wmsetbg. Turns out that may not be straight forward. Ooops!
Will likely step away from both for a short bit to do some Bevy WASM stuff and work in areas of the language I'm weak. Generics.
Try rig, it really levels up that use case.
Getting the Rust side of ArrowMetal in front of people. ArrowMetal runs Apache Arrow compute on the
Apple silicon GPU through Metal, and the arrowmetal crate (0.1.0 on crates.io) takes an arrow-rs
ArrayRef, hands it across the C Data Interface without a copy when the buffer is page aligned, and
gives an ArrayRef back. Reductions, comparisons, filter, take, sort, cast and a group-by are
wrapped and tested against arrow::compute on the same data. The honest number on 10M Int64 rows:
a lone sum loses to arrow-rs by 2.3x, because mapping the buffer costs more than the kernel, and
compare + filter wins by 1.5x. It needs an Apple silicon Mac and a dylib the build script has to
find, which is the part of the API I'd most like opinions on. docs.rs is red for 0.1.0 (a build
script bug), so the docs live in the repo:
ArrowMetal/docs/RUST.md at main · singhpratech/ArrowMetal · GitHub
If you have an Apple silicon Mac and give it a try, I'd like to hear how it went, good or bad:
what was awkward to set up, what you expected the API to do that it didn't, and what your own
timings look like.
Done a lot of clean-up and feature work for uutils awk. Around a third of GNU tests pass now, but I don't have concrete numbers since there are many false negatives and all of them need to be triaged and patched.
The plan now is to do some polishing, iron out as many GNU tests as possible and do an alpha release over the coming weeks. Stay tuned!
just found this option too