My code relies on bitwise operations on pointers, so I used unsafe code extensively. I am seeking a delicate code review for safety, and any other feedback is welcome too!
I suggest you test your code with cargo +nightly miri test. Perhaps you have done so already, but it isn’t done in your GitHub CI. Testing with Miri is an important step that is highly valuable for non-FFI unsafe code. Miri cannot check that your code is sound, but it can check that for the tests that you do have, it is not executing UB.
Glancing at your code, it seems that you are using a NonZeroUsize value for storing the modified pointer. This is problematic, because it discards provenance. You should use a pointer type to store any pointer you are going to dereference later. It’s okay to do arithmetic on the address of that pointer.