Of course, internal server errors should never happen, but they occasionally do anyway. There's a difference between "user pasted his thesis into the input field instead of an expression" and "programmer forgot to check the size and indexed into an empty slice". The first kind of error is not exceptional, it is expected that users will occasionally do dumb stuff accidentally or on purpose, and the user should expect some feedback about what they did wrong. The second kind of error is exceptional, its existence should have the least possible impact on happy path performance, and whatever internal server details are relevant should be logged and the user shown a 500 page.
Of course you can distinguish between kinds of panics. That means you must be doing some checking at catch_unwind, which means you're either using panic_any with custom types and downcasting it (ew) or parsing the string representation (double ew). I guess I can see how not having to use ? or -> Result<T> might add up over a deep call stack but... in terms of error handling, I guess I think of most of the boilerplate being in creating custom error types, messages and conversions, which just surfaces in another form if you are doing matching on panic types at the top level.
Worth it? 