10 minutes for a clean dev build is a bit too much, but expected for complex apps. How long do your incremental builds take? They should normally be measured in seconds. If incremental builds take more than a minute, there is some issue.
I don't think you can cross-compile Go to MacOs easily.
Go tries to boil the ocean and performs the entire compilation end-to-end with its own custom toolchain. It also obsessively avoids dynamically linking anything, including libc (they have walked back that position on the most stubborn OS's, including MacOS). This makes the process easier for the end user when it works, but very non-scalable (how many platforms exist?) and impossible to fix when it doesn't. Rust integrates with existing toolchains, including the linkers and dynamic libraries. This means that whatever you can build with C, you likely can build with Rust, but the process has more moving parts. Have you tried cross?
EDIT: Oh, you're using C++ dependencies. I don't know why you would expect easy cross-compilation or good compilation performance in that case.
Imho you should seriously consider adding a feature which allows to use pre-built Whisper. You really shouldn't be recompiling C++ if you want good build performance.
EDIT2: Build scripts are also a drag on compilation performance. I think your scripts are misconfigured. You don't emit rerun-if-env-changed for all of your used environment variables (e.g. BLAS and CUDA paths), and you also don't emit rerun-if-changed. I believe this means basically any change causes the build scripts to be rerun. Besides wasting time on building and running build scripts, this may also have downstream effects of invalidating your compilation caches. See rerun-if-changed and FAQ.
EDIT3: You are also building the same packages multiple times. Your Cargo.lock contains many duplicate packages which differ only in version. Windows builds seem particularly affected, there are many duplicated crates. E.g. windows-core is built 4 (!) times with different versions. Windows API crates are notoriously huge, this can be multimillions lines of code, with obvious detrimental effects. You really should understand why you have duplicate dependencies, and prune them aggressively, particularly windows ones. Manually pin minor versions of dependencies if need be.
Hell, you depend on 2 different versions of itertools! I get how someone can pull in multiple winapi crates, those are updated often and almost always incur an incompatible version bump. But itertools? Your dependencies are just a mess.
A simple deduplication of name = ".." strings in your Cargo.lock shows me you have 73 (!) dependencies with multiple versions. 17 of those are winapi-specific, with about half of those probably irrelevant (different targets). That's a lot!