You need to use an Arc<UnsafeCell<...>> if you want shared mutability without any overhead. But unless you're very confident something else is already doing synchronization you can run into problems even if it's true that the threads are never running at exactly the same time.
UnsafeCell is the only type that's allowed to mutate through a shared reference, and all the other types like Mutex that do so use an UnsafeCell internally. As the name implies though, you have to use unsafe to do anything useful with it.