µTate is a music visualizer aiming to leverage modern, post-attention heuristics. This will achieve a beat prediction rather than just beat recognition and will also enable modulating pipeline composition and inputs instead of being limited to manually created presets.
Can Wayland users even cargo run this yet? Is it clear that Windows and MacOS users will need to integrate CPAL? If you started digging into this project to figure out how to run it, which pieces are helpful and did you find them easily? I want to support your platform, and I need some test pigs! Are you loud? Do you complain well? Please come!
Of course, make it have an awesome video. I'll get right on it.
What I'm working on next is having the frontend dispatch, which operates on the presentation clock, catch a smoothly sliding slice of the upstream output rings from audio processing. I'm using Kalman style filters on both clocks so that the feed rate is smoothly aligned in time.
A lot of the schlep is done, and I've gotten up to speed on the kinds of Rust needed for this project and Vulkan 1.4, so jump right in. There's absolutely a business end to this and I'm looking for co-founders. If you can intuitively understand the Gabor limit, you will have a good time.
I know nothing in music software from programming point of view, however I use FLStudio and I may ask you if your project is just a music visualizer or making a DAW later ?
I am interesting for learning only and I can't contribute to your project, I am learning first to Rust programming.
It's a visualizer and a context for developing post-attention machine learning.
The DAW market is pretty saturated and the barrier to get in is a zillion features. While I'm putting a lot of work into DSP to come up with a commercially viable spectrogram, the techniques are focused on monitoring and may not be great for audio processing (64bit operations are limited on the GPU's SFU unless you have a "workstation" card that is a bit rare). There are also times where it's better to get information onto the screen and kind of correct it later, but "kind of" will bite for serious DAW work. Tradeoffs for DAW and visualization are at odds.
Think more like fancy eye candy for live shows and casual home visualization.
The 2017 paper on attention can be seen more or less as a milestone. Pre-attention, we see deep learning, CNNs, stacked Boltzmann machines etc. Post-attention means more sophisticated than any pre-attention techniques, but is expressly not limited to LLM style solutions.
The on-device DFT is working at a basic level and needs some implementation correctness and extra features for improved low-pitch resolution and more level response and prevention of far-field bin leakage.
µTate is a box for others to put ML inside. The DFT is one of the most crucial inputs. Inputs like the DFT are hand rolled because it would be very inefficient for ML to try to learn the equivalent behavior. With a good DFT available on the GPU, extracting musical awareness is basically a seq2seq problem.
The black boxes will be able to articulate pipelines and manipulate data within the buffers. That is where the "mutation" will happen. The job of µTate is to expose the knobs that machine learning can operate to modulate the output. A five-minute old model can't know Vulkan, Slang, or how to emit valid procedures to call in Rust. It needs to be able to use a simple vocabulary of pipelines and buffers.
I will be providing one reference implementation using intentionally bizarre, non-formalized machine learning in order to provoke other development.
edit: I forgot to add, but since this is an engineering forum, I'll explain why this situation is forced upon us. "Post-attention" is a bit of an evasion of people who don't know anything about machine learning but feel entitled to derail conversations about it. I used "heuristics" in the sense I got from Skiena. I encourage all ML developers to use "post-attention heuristics" as a catch-all for "things that are better than LLMs" without exciting the ignorant masses.
Are you loud? Do you complain well? Please come!