It doesn't have exactly the same semantics as your code (no existential type stuff here), but would it be acceptable to use generics for your use case?
trait MatVecMul {}
fn solve<F, M>(a: Vec<f64>, mul: F) -> f64
where
F: FnOnce(i32) -> M,
M: MatVecMul
{ ... }