Is it possible to unsafely ignore a generic's lifetime bound?

help
1

I'm trying to implement an OwnedMutexGuard on top of std::sync::{Arc, Mutex, MutexGuard}. I know about the prior art of tokio::sync::OwnedMutexGuard, and lock_api::ArcMutexGuard. But I'm using this for a data structure that is complex enough that I want to test it with loom, and loom only mocks std::sync types. And I'd prefer to use standard library types anyway, for maintainability.

Recap of what an "owned mutex guard" is: Have a Arc<Mutex<T>>, clone the Arc, lock the Mutex<T> to get a MutexGuard<'a, T>, and store the Arc along with the MutexGuard. Then as long as your struct only ever drops the MutexGuard before dropping the Arc, the guarded T is guaranteed to live as long as the guard, so lifetime bounds are unnecessary.

In theory.

In practice though, this is my best attempt:

pub struct OwnedMutexGuard<'a, T: 'a> {
    guard: std::sync::MutexGuard<'a, T>,
    handle: std::sync::Arc<std::sync::Mutex<T>>,
}

It needs an unsafe std::mem::transmute call when constructing, which is expected; to change the MutexGuard's lifetime to 'a, but it works.

It just really sucks to use, because of the lifetimes, when the whole point of this was to avoid having to write down a lifetime! I wish I could just have it be OwnedMutexGuard<T>, with the 'a silently swallowed because it's unnecessary.

All the 'a is caused by MutexGuard<'a, T>, which has a T: 'a bound, which "infects" the surrounding struct, which when used infects the surrounding context, and so on, until every generic thing in my codebase is <'a, T: 'a>.

I understand why this happens, and why a 'self lifetime doesn't exist: it's usually helpful to know when a struct contains a reference, and that reference's lifetime is usually relevant. But in this case it's pure noise. There is no clarity or correctness gained from including it. tokio::sync::OwnedMutexGuard doesn't have such a bound either.

Is there some unsafe workaround to this problem? What sins can I commit to get a clean API?

I want to tell Rust to "sudo ignore this T: 'a bound, and let me write just OwnedMutexGuard<T>; I promise I know what I'm doing".

Solutions I've considered:

  • Some std::mem::transmute incantation? I don't know how to phrase it, because I still need to at some point state the type I'm transmuting to, like MutexGuard<'?, T>, and that '? needs to come from somewhere, and that somewhere is now T: '?, and so is everything that touches it. Same problem.

  • Omit the lifetime by making it 'static. But then MutexGuard<'static, T> requires T: 'static. Which is equally "infectious" in the codebase, and restricts use.

  • Give up on my dreams of std::sync-purity and easy maintenance. Use lock_api and replace this with lock_api::ArcMutexGuard. Then vendor loom as a git submodule, and manually merge this old PR to support lock_api. And maintain that, forever. :sob:

2

You can use ouroboros (or maybe yoke?) to build the self-referential struct for you; it does the required lifetime-mucking under the hood.

Cavears:

  • it might turn out that ouroboros's technique is unsound (but so far, all known flaws have been fixed by the maintainer)
  • it won't return an &mut reference to the contents of the lock guard; you have to use a callback like with(|data| { ... }) to manipulate it.
3

I could be wrong, but I think that under the hood ouroboros replaces all lifetimes called 'this with 'static, so this would also end up with T: 'static bounds due to the MutexGuard<'a, T: 'a> signature if you're trying to avoid (@anko). I don't know a better solution though

4

Despite yoke's focus on the zero-copy deserialization use case and general bias towards immutable usage, yoke does have Yoke::with_mut which should mostly work for this application, although again it only offers a with-callback API.

Generally, if you can't create the lifetime-encapsulating API you desire with either yoke or ouroboros or yoke, it's because of some fundamental bit of complexity you've overlooked which if not addressed, can be turned into a soundness hole. In this case, without a T: 'static bound or very careful callback bounds, it would be possible to mutate T to introduce a borrow of data not actually owned by the 'self construct, to outlive that borrow, and to create a use after reference invalidation UB.

IIRC the use of 'static here for this purpose could actually be eliminated, but it would require introducing a different 'bottom lifetime to the type to use as a bound instead.

Your best bet probably is to use the lock_api versions. Loom is generally used behind a cfg(loom) gate for testing rather than a more public-exposed feature anyway, so using a semi-maintained patch to loom. It's honestly quite small, just exposing the already existing internal raw mutex.

5

AFAIK the answer is that there isn't a great way right now.

Some people are exploring a feature that would allow

pub struct OwnedMutexGuard<T> {
    guard: unsafe<'a> std::sync::MutexGuard<'a, T>,
    handle: std::sync::Arc<std::sync::Mutex<T>>,
}

as a way to do this in future, so that safe code could never use the thing.

2 Likes
6

I'm probably misunderstanding something, but why do you think this lifetime is avoidable? After all, you could have sync::Mutex<SomeStructWithReference<'struct>>, which will not ever allow soundly getting a MutexGuard<'static, _> - otherwise this guard could outlive the value inside the Mutex.

1 Like
7

I presume for the same reason that Box and Rc don”t have lifetime parameters: By existing at all, they ensure that a value of type T also exists, and their owners are allowed to keep them alive as long as it would be legal to keep an unwrapped T alive.

In this situation, the extra lifetime parameter is redundant at best and artificially limiting at worst (if the lifetimes get desynced).

2 Likes
8

Not if they literally have a MutexGuard<'a, T> field, which carries with it a well-formedness constraint. To get rid of the lifetime, you'd have to erase that type somehow, and reform it with some short-lived lifetime on drop... or the like.

9

To clarify, I was only talking about why the API interface would ideally not have the lifetime parameter, based on the type's intended behavior.

Whether or not it's possible to write an implementation that conforms to that ideal interface is the main question of this thread, and I have no particular insights to add regarding the implementation of such an interface.

The well-formedness of the type of a private field is certainly relevant to the discussion, but only insofar as it is an implementation detail that must be accounted for.

10

Thank you everyone. I got flustered by the all-star response to this thread, so I put off replying. But I figure I should review the alternatives presented since I've had time to mull them over.

I've been working on the code that uses the useless-lifetime-containing OwnedMutexGuard shown in my first post. I have ended up just tolerating it, because the alternatives haven't felt better:

Self-referential struct helpers (ouroboros, yoke)?

Their .with_mut(|x| ...)-callback interface yields less clear code than what I already have. I much prefer lifetime-noise in type definitions and using DerefMut, over clean type definitions but callback-noise at every use site.

Type erasure?

I got that to work if I introduce an additional Box. Here's a proof-of-concept playground.

It has comments, but in short:

  • To avoid stating the MutexGuard<'_, T>-type, lock the mutex, Box the guard, Box::into_raw, cast the box pointer to a *mut u8, and store that in OwnedMutexGuard.
  • In Deref and DerefMut, cast the pointer to a MutexGuard<'_, T>-pointer again, dereference, and reborrow, constraining the lifetime to the elided one, which gets inferred to match &self/&mut self.
  • In Drop, Box::from_raw the same cast pointer.

So, the same thing as before, but boxed, and casting the box pointer to *mut u8 to avoid mentioning the MutexGuard<'_, T> type, except in contexts where the implicit lifetime '_ is inferrable.

It passes the same tests, also under Miri.

However, it does introduce a Box, which means a heap allocation on every mutex lock. In my case, I have lots of mutexes needing to be locked this way, so I don't want to touch the heap more than necessary. But for someone who needs this kind of thing and who only needs it rarely on a slow path, this might be useful.

Type-erasing MutexGuard, without the Box?

Rust really doesn't want you to do this. I tried, but even the maybe-possible hacky approaches feel like hammering in screws.

Collapsed section containing notes about sharp explodey unportable hacks that I could not finish enough to demo them before giving up.

If you wanted to avoid the Box, you'd have to store not just a pointer, but the MutexGuard itself, as an "erased" type, like a byte array. That means you'd have to communicate to the compiler that you want a byte array with a compatible size and alignment. Neither is directly possible at compile-time on stable Rust:

So either you have to—

  • —do it at run-time: Have OwnedMutexGuard contain a probably-large-enough byte array, then manually allocate the MutexGuard in that byte array. Get the MutexGuard's layout at runtime using std::alloc::Layout::for_value, and find an aligned space for it in the array using std::ptr::align_offset, then transmute_copy the guard in there, and mem::forget the original so its Drop doesn't run.

  • —or use a build script to find the size and alignment of MutexGuard at "compile-run time" using std::alloc::Layout::for_value. Generate code that makes those into compile-time constants, and insert them into the declaration of OwnedMutexGuard. (Which has all the caveats.)

    Maybe surprisingly, transmute allows casting if the compiler can prove the two types have the same size, which it can do here, even though std::mem::size_of doesn't work. This should be safe if the storage space for the MutexGuard within OwnedMutexGuard is declared with #[repr(C, align(_))] that specifies enough alignment, but a macro could generate that too.

Then, similarly to the "boxed" playground, transmute the byte array into an appropriate MutexGuard<'_, T> or reference inside Deref, DerefMut and Drop, and look ma, no heap!

All of this is probably an awful idea.

The future?

This would directly address the problem, and would have saved me a lot of sanity. Is there a draft RFC or internals thread where I can read more?

11

Or simply erase to MaybeUninit<MutexGuard<'static, [()]>> (together with compile-time size&alignment assertions, since it's an internal implementation detail that they're compatible)? Playground.

1 Like
13

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.