That is correct. I am not suggesting use of catch_unwind. In fact what I suggest is that the client code should not come to a point to use catch_unwind.
If the proper way of error handling in Rust is using Result/Option, then there should not be any call to panic! (directly or indirectly) in a library.
The problem with unwrap is that if it is not examined carefully to make sure it would not result in panic, then it would decrease robustness/resiliency of the whole application. I case of you code, you are correct and as you explained panic would not happen in to_u32, simply it was a bad example that I picked.
Using unwrap is an easy way to deal with errors and it is very tempting to use it everywhere. If you look at sample codes provided by many library authors, unwrap is used for dealing with errors which makes sense since that sample code is supposed to show the concept and using unwrap makes the code simpler to understand and less verbose. However if you are a new user of Rust, it gives you impression that is the way of dealing with errors in Rust.
Essentially unwrap is a footgun. Rust has made a good reputation that its compiler stops you from doing anything unsafe (unless code author explicitly indicates that with unsafe keyword).
So what I suggest here is that Rust compiler could go further and not only improves safety but also robustness/resiliency and I am not sure if “unwrap” helps in that direction.