The difference is that the first is using the trait as a trait, and the second is using it as a type. When you use a trait as a trait, the compiler will generate separate versions of the function for every choice of types you call it with. When you use it as a type, a single version of the function is compiled that can accept any value of any type that implements the trait.
When traits are used like types, they become something called a trait object, which is a specific type. The trait object type is not the same as the underlying trait that it corresponds to as traits and types are different things.
The reason it failed is that the trait object types can only exist behind a pointer because they could have any size. There could be a type of size one million bytes that implement the trait, and that type has to be supported. Supporting this only works if the value is behind a reference.
You may be looking for this syntax:
fn inline<T>(mut f: impl FnMut(&T)) {
}
The impl Trait syntax is a short-hand for generics, and this is equivalent to the first version.