FFI: Converting function/closure to isize and back

I have a rust function that uses the windows crate to enumerate all currently open windows. Depending on the window class and title I want to sort them into different containers, which i want to accomplish by using a selector function.

The enumerate windows function looks like this:

pub(crate) struct Window { /* ... */ }

pub(crate) fn enumerate_windows<F>(selector: &F) -> bool
where
    F: FnMut(Window) -> bool,
{
  unsafe extern "system" fn enum_windows_proc_wrapper(hwnd: HWND, param: LPARAM) -> BOOL {
    let window_proc: &fn(Window) -> bool = ptr_to_type_ref(param.0);
    
    let Ok(window_information ) = get_window_information(hwnd) else {
      return false.into();
    };
            
    let Some(window) = window_information else {
      return true.into();
    };
      
    let proc_result = !window_proc(window);
    proc_result.into()
  }
  let selector_ptr = LPARAM(type_to_isize(selector));
  unsafe { EnumWindows(Some(enum_windows_proc_wrapper), selector_ptr).into() }
}

the selector function usually is a closure that takes the window struct and adds it to a list of windows.

As you can see, the EnumWindows function takes the wrapper function and an (optional) parameter for the wrapper function (MSDN). I convert the parameter to LPARAM / isize with

pub(crate) fn type_to_isize<T>(ptr: &T) -> isize {
    (ptr as *const _) as isize
}

and back with

pub(crate) unsafe fn ptr_to_type_ref<T>(ptr: isize) -> &'static T {
    let ptr = ptr as *const T;
    &*ptr
}

However, this gives me an access violation. I also tried

let fn_ptr = unsafe {
    std::mem::transmute::<isize, fn(Window) -> bool>(ptr)
};

to no avail.

I want to add, that when I the above functions with structs everything works as I expect. What did I miss?

Casting a pointer to isize is unsound. Cast to usize instead.

The problem is, enumerate_windows() takes an &F, where F is a FnMut closure, but window_proc expects F to be a fn pointer. Most closures are not function pointers: they are effectively structs containing the values that they capture from their environment. Thus, when you read the F as a fn pointer, this results in an access violation. The solution is to either take selector as a fn(Window) -> bool in the first place, or to read window_proc as a &mut F. (Since F is a FnMut, you must also use &mut F in place of &F.)

Also, it would be a very good idea to use catch_unwind() around the function. This prevents any unwinding panics from escaping the extern "system" function, which currently results in UB.

This is incorrect; both isize and usize are large enough to hold any exposed pointer value.

Both isize and usize have the same size as a pointer, that’s right. These types are not the same tho. They have one big difference - range of values. isize’s range of values includes signed integers, while values of usize are unsigned. It means that an isize cannot contain all the values that usize or a pointer can, doesn’t it?

Sounds reasonable. To understand it a bit further, I created this MWE which just takes a closure, converts it to isize and then back to a closure. However I still get an access violation. This is the code (playground)

let foo = || 0;

let ptr = dbg!(type_to_isize(&foo));
let function: &fn() -> i32 = unsafe {
  ptr_to_type_ref(ptr)
};

assert_eq!(function(), 0);

I assume this is the same problem you described, that I can't simply cast to fn because I have a closure? If so, how would I annotate a closure? If not, what is the error here?

@Miiao: In addition to what @LegionMammal978 said, I can't use usize since the windows crate expects isize in 99% of all cases, like struct LPARAM(isize);
Also while in theory correct that isize only holds 2^63 as opposed to usize's 2^64, the memory limits on the machine imposed by the OS are far smaller. IIRC Windows and linux currently only use 48bits.

Nope- values between isize::MAX and usize::MAX are simply stored as negative numbers when converting from usize to isize directly. On most modern systems, usize, isize, and pointers are all 64 bits, and you can losslessly convert between them.

Well, I actually wanted to say that a negative number cannot be a correct address since a pointer is always a non-negative number. My general idea was something like “a pointer can only be transmuted to usize because a pointer is usize”. Similar to answers to “why do we need unsigned integers, why not just have signed?”. Guess I cannot help @Ruhrpottpatriot solve this case, so I’m gonna leave you.

1 Like

Closures that don't capture their environment can be coerced or cast into a function pointer of the appropriate signature:

let f: fn() -> i32 = ||0;

However, you'll still have problems, because of mismatched calling conventions- the rust calling convention is undefined, and winapi is going to expect a function with a different abi, likely either "system" or "C". You'll have to write a proper function for that:

unsafe extern "C" fn callback(_param: LPARAM) -> i32{
    0
}

Any state that you need must be passed in via the provided parameter, probably via a type erased pointer to some data somewhere. If you ever find yourself in an unfortunate situation where an api requires a callback but doesn't allow for state you need to be passed to it, libffi can be used to close the gap.

However, you'll still have problems, because of mismatched calling conventions

Which is what I'm using. If you look at the first post I'm passing unsafe extern "system" fn enum_windows_proc_wrapper(hwnd: HWND, param: LPARAM) -> BOOL as the callback. The closure is then the parameter to the callback.

Sorry, brain fart on my part. You aren't performing the type erasure properly. To safely type erase a closure and recover it, you'll need to convert it into a pointer to a trait object, and pass a pointer to that object to your context. Something like

let mut f:Box<dyn Fn() -> i32> = Box::new(f)
let ptr = &mut f as *mut _ as isize;

on the caller side and

let func = unsafe { &mut*(param.0 as *mut Box<dyn Fn() -> i32>) };
let result = func();

on the callback side.

Bits are bits, and specifically signed vs unsigned is just a question of interpretation on two’s complement systems (which is all of them essentially). as casting between signed and unsigned is a no-op and always valid, and it’s equally valid to transmute a pointer to isize and usize. Half the address range being represented by negative integers may feel weird, but the hardware doesn’t care.

1 Like

This is not necessary, if a generic function is used.

The error is that you are still taking the foo closure (a size-0 Fn() -> i32) and interpreting it as a fn() -> i32. You'd need to explicitly convert foo into a fn() -> i32 (either with an as cast or a type annotation) to get the actual function pointer.

Here's how I'd write enumerate_windows() to use a function pointer:

pub(crate) fn enumerate_windows(selector: fn(Window) -> bool) -> bool {
    unsafe extern "system" fn enum_windows_proc_wrapper(hwnd: HWND, param: LPARAM) -> BOOL {
        let window_proc: fn(Window) -> bool = isize_to_fn(param.0);

        let Ok(window_information) = get_window_information(hwnd) else {
            return false.into();
        };

        let Some(window) = window_information else {
            return true.into();
        };

        let proc_result = std::panic::catch_unwind(|| window_proc(window))
            .unwrap_or_else(|_| std::process::abort());

        (!proc_result).into()
    }
    let selector_ptr = LPARAM(fn_to_isize(selector));
    unsafe { EnumWindows(Some(enum_windows_proc_wrapper), selector_ptr).into() }
}

pub(crate) fn fn_to_isize(ptr: fn(Window) -> bool) -> isize {
    ptr as isize
}

pub(crate) unsafe fn isize_to_fn(ptr: isize) -> fn(Window) -> bool {
    unsafe { std::mem::transmute(ptr) }
}

(fn(Window) -> bool is already a pointer, so there is no need to put it behind another reference.) And here's how I'd write enumerate_windows() to use a FnMut closure:

use std::panic::AssertUnwindSafe;

pub(crate) fn enumerate_windows<F>(selector: &mut F) -> bool
where
    F: FnMut(Window) -> bool,
{
    unsafe extern "system" fn enum_windows_proc_wrapper<F>(hwnd: HWND, param: LPARAM) -> BOOL
    where
        F: FnMut(Window) -> bool,
    {
        let window_proc: &mut F = isize_to_ref(param.0);

        let Ok(window_information) = get_window_information(hwnd) else {
            return false.into();
        };

        let Some(window) = window_information else {
            return true.into();
        };

        let proc_result = std::panic::catch_unwind(AssertUnwindSafe(|| window_proc(window)))
            .unwrap_or_else(|_| std::process::abort());

        (!proc_result).into()
    }
    let selector_ptr = LPARAM(ref_to_isize(selector));
    unsafe { EnumWindows(Some(enum_windows_proc_wrapper::<F>), selector_ptr).into() }
}

pub(crate) fn ref_to_isize<T>(ptr: &mut T) -> isize {
    ptr as *mut T as isize
}

pub(crate) unsafe fn isize_to_ref<'a, T>(ptr: isize) -> &'a mut T {
    &mut *(ptr as *mut T)
}

Feel free to ask any questions.

The error is that you are still taking the foo closure (a size-0 Fn() -> i32) and interpreting it as a fn() -> i32. You'd need to explicitly convert foo into a fn() -> i32 (either with an as cast or a type annotation) to get the actual function pointer.

Ah! Now I got it to work. So closures are essentially anonymous types, which I cannot annotate, right? I found THIS in the rust reference.

So, to recap and see if I understood you correctly.

  • A closure is an anonymous type that, could it be annotated would be a struct containing the captured variables and the possibility to call the closure with the struct fields in addition to any other parameters available in the body.
  • Because of this I can't simply cast to isize and back because, in very rough terms and probably wrong, I would "land" in the struct part (or somewhere else entirely) instead of the" callable part".
  • Casting a closure to a function pointer will give me the "callable part" of the closure which I then can pass everywhere a function pointer is expected.
  • Because of the aforementioned possibility I can also pass the closure to a generic function where the generic type argument is Fn (for &fn) or FnMut (for &mut fn). What would be the equivalent for FnOnce?

Perfect.

For any closure not capturing any variables, yes. Closures with captures can't be coerced into function pointers though, for the same reason a live object's method can't.

I'm having a little trouble following you here, is it maybe Box<dyn FnOnce()> you're looking for?

1 Like

You can't cast a closure to a pointer and back for an even more fundamental reason: they are not auto-boxed; they have varying sizes.

You can only cast a non-capturing closure to a function pointer.

Function pointers are always Fn as opposed to only FnMut or FnOnce, as they have no state (nothing to mutate or consume). However, perhaps it will help to point out that

  • If you implement Fn, you implement FnMut (FnMut can just call Fn::call(&*self))
  • If you implement FnMut, you implement FnOnce (FnOnce can just call FnMut::call_mut(&mut self))

And therefore function pointers do implement all of Fn, FnMut, and FnOnce.

More documentation can be found here.

Named functions and methods also have their own unnameable types.

Yes, the basic idea of closure types is correct.

Really, the unnameable closure type doesn't even have a "callable part": for Fn closures without captures, coercing them to fn() pointers creates the callable pointer on the spot. In other words, a closure type remembers its function pointer as part of its type, without explicitly storing it in its values. A closure without captures does not store any data (it's 0 bytes long), since the type itself acts as a marker for which function to call.

In the generic F: FnMut(Window) -> bool version, we don't pass around any fn pointer. Instead, we pass around the &mut F reference converted to a pointer, which is a closure to the "struct part". (We don't need to remember the "callable part", since that can be derived from F itself.)

The difficulty with FnOnce is that it cannot be called behind a reference. Therefore, either the value must be moved out from a location which lives long enough, or it must be placed in a Box which is converted to a pointer. To illustrate the two methods:

use std::mem::ManuallyDrop;

// works only if `inner` is called before `outer` returns
pub fn outer_1<F>(f: F) -> i32
where
    F: FnOnce(i32) -> i32,
{
    unsafe fn inner<F>(value: i32, ptr: usize) -> i32
    where
        F: FnOnce(i32) -> i32,
    {
        // this copies the closure out of `storage`
        // since `storage` is a `ManuallyDrop` and we never touch it again,
        // we have effectively moved the value
        let f = (ptr as *mut F).read();
        f(value)
    }

    let mut storage = ManuallyDrop::new(f);
    let ptr = &mut *storage as *mut F as usize;
    unsafe { inner::<F>(42, ptr) }
}

// works no matter when `inner` is called, as long as it is only called once
pub fn outer_2<F>(f: Box<F>) -> impl FnOnce() -> i32
where
    F: FnOnce(i32) -> i32,
{
    unsafe fn inner<F>(value: i32, ptr: usize) -> i32
    where
        F: FnOnce(i32) -> i32,
    {
        let f = Box::from_raw(ptr as *mut F);
        f(value)
    }

    let ptr = Box::into_raw(f) as usize;
    move || unsafe { inner::<F>(42, ptr) }
}