It's possible to override the profile configuration for specific packages in your own Cargo.toml. Note that this applies to specific packages, not their own dependency trees. So, for example, here's my own configuration for ensuring that any png decoding is fast in all build profiles:
Seeing 50x different is not uncommon at all. LLVM is really good at optimizing, and because of that Rust isn't all that careful about what it emits, since LLVM can clean it up. But that means that if the optimizer isn't running at all, the generated code is very slow.
Making code faster is computationally expensive task. During development it's common that every new run requires recompiling as the source code is changed since last compilation. In this case optimizations usually slow down the total compile+run cycle. But after released the code compiled once tends to keep running for a long time. In this case the benefits of fast code outweighs the one-shot cost of the optimization.