To my mind, the point of writing safe code isn't to avoid bugs. Bugs are inevitable. There are certain kinds of buggy behavior (panics, memory leaks) about which Rust explicitly says "yep, that's fine". There are other more subtle bugs that Rust allows (like changing the hash of a value in a HashMap via internal mutability, or writing an asymmetric PartialEq impl) that are clearly bad but still can't lead to memory unsafety or undefined behavior.
What's special about UB is that it attacks your ability to find bugs, like a disease that attacks the immune system. Undefined behavior can have arbitrary, non-local and even non-causal effects that undermine the deterministic nature of programs. That's intolerable, and that's why it's so important that safe Rust rules out undefined behavior even if there are still classes of bugs that it doesn't eliminate.
I also think you are overestimating the complexity of the safe solution compared to the unsafe-using one; or, at least, you're not accounting for the difficulty of proving the unsafe solution correct because it's a cost you're used to paying. In most cases the difficulty of writing correct unsafe code is no more than that of convincing the compiler to accept a safe one, and in a lot of cases the unsafe route turns out to be harder. Of course, it is always easy to write unsafe code that may be wrong.
(Also I believe the program can be further simplified, but that is neither here nor there.)