Is it UB to transmute fn(&mut T) to fn(*mut T) and then call them?

help
1

Does the following code contain any kind of UB? Suppose that the caller of g will manually guarantee the exclusiveness of its argument.

use std::mem::transmute;
use std::ptr::NonNull;

struct Foo(i32);

impl Foo {
    fn foo(&mut self) -> Option<NonNull<i32>> {
        return None;
    }
}

fn main() {
    unsafe {
        let f = Foo::foo as fn(&mut Foo) -> Option<NonNull<i32>>;
        let g = transmute::<_, fn(*mut Foo) -> Option<NonNull<i32>>>(f);
        let mut x = Foo(0);
        g(&raw mut x);
    }
}
1 Like
2

Why not just reborrow as a &mut when you need to guarantee anyways that all it's invariants are upheld (uniqueness and non-null). It makes it very clear that what you assume is, that it is safe to use this mutable raw pointer as a mutable reference.

fn main() {
    let mut x = Foo(0);
    let x_ptr = &raw mut x;

    // Safety: x_ptr is the only thing pointing to x and it is not null.
    Foo::foo(unsafe{&mut *x_ptr});
}
1 Like
3

&mut T and *mut T are intended to be fully ABI compatible. Also, your code passes Miri. So I'd expect it to be OK. That said, do you really need to do a transmute? If you just need to pass fn(&mut T) where fn(*mut T) is expected, you could always write a simple ad-hoc closure

fn takes_callback(f: fn(*mut T)) {..}

takes_callback(|t: *mut T| Foo::foo(unsafe { &mut *t }));

Note that a closure without captures coerces to a simple function pointer.

11 Likes
4

The linked playground is simplified. In fact I was implementing sort of vtable mechanism, which requires to cast some safe function pointer types to their "looser" version (with lifetime or even type erased). But I'm not quite sure if this would cause UB.

5

IIRC, the answer is "maybe". I did a very similar thing myself, and I believe the answer was that while it isn't guaranteed to be the same, it works, and is very unlikely to ever change.

1 Like
6

yep, this is documented in the Unsafe Code Guidelines

https://rust-lang.github.io/unsafe-code-guidelines/layout/pointers.html

1 Like
7

If you use extern "C" fn:s it should be fine, without extern (aka extern "Rust") it might be ub.

1 Like
8

So casting "Rust" convention func ptrs is not a good idea. What about the workaround mentioned by @afetisov ?

9

This is not final yet though so I wouldn't rely on it.

Also, I think you maybe didn't mean to answer to my reply since I don't see how it is related tk what I wrote.

10

It's in fact also documented in the stdlib docs:

8 Likes
11

It's probably clear, but just for completeness since most of the answers focused on the specific aspect of calling a function through a different signature. There are 3 possible ways to get UB when transmuting from fn(&mut T) to fn(*mut T):

  • The transmute itself. But in this case, there's no UB (same validity invariant).
  • Calling an fn(&mut T) through a fn(*mut T). That's what was focused on in this thread. Not a problem either (ABI compatible).
  • Exposing the transmuted value outside the scope of unsafe (the safe abstraction around the transmute). When transmuting to a safe type, you must make sure that either the value is within the safety invariant of that type (it's not the case here since the safety invariant of fn(*mut T) is much smaller than fn(&mut T) because the safety invariant of &mut T is much smaller than *mut T and by contra-variance of function arguments[1]) or that this value is only used within the scope of unsafe (here making sure it's always called with a *mut T that satisfies the safety invariant of &mut T[2]). Alternatively you could transmute to unsafe fn(*mut T) and document the pre-condition if you need to expose the result of transmute to your public API.

  1. Actually that's not the only reason, calling a function through a different signature transmutes all arguments so you must also make sure your *mut T is actually valid for &mut T (not only safe[2:1]). ↩︎

  2. Note that the *mut T being safe for &mut T is not strictly necessary if the initial function is robust. Only the validity is really required. ↩︎ ↩︎

1 Like