I'm working on function hooking library (mainly for native game modding) that builds on my closure-ffi crate to allow capturing closures to be used as hooks. At the moment I'm just trying out different user-facing APIs for constructing the hook, to see what plays nicely with type inference.
One thing that is very useful to be able to do when writing hooks is to access a function pointer to that calls the original function (usually via a JIT'd trampoline) from within the hook body itself. I attempted to do this by having the user provide a "getter" FnOnce impl that is passed the hook context, and returns their hook closure. This way they can move it into their hook body.
I particularly like this solution because it doesn't require further macro-implementing traits for functions of different argument counts in the hooking crate (beyond what is already done in closure-ffi). However, this plays very poorly with type inference as can be showed in the example below (cut down to only the necessary parts to show the problem).
Code
use std::marker::PhantomData;
/* rough layout of the closure-ffi API */
// Calling convention marker types; only C provided for brevity
mod cc {
#[derive(Default)]
pub struct C;
}
/// Used to support an API that doesn't require passing the calling convention marker type
/// when the function type is known
pub unsafe trait FnPtr: Copy {
type CC: Default;
}
unsafe impl<R> FnPtr for unsafe extern "C" fn() -> R {
type CC = cc::C;
}
unsafe impl<T, R> FnPtr for unsafe extern "C" fn(T) -> R {
type CC = cc::C;
}
// Has an associated const with a compiler-generated thunk function that gets a pointer to the closure
// and calls it.
//
// CC is included as a generic parameter instead of just B to allow downstream crates to implement
// FnThunk for higher kinded bare function types through a macro as needed
// For the same reason, we don't bound B with `FnPtr`
unsafe trait FnThunk<CC, B: Copy> {
}
// We implement for (CC, F) instead of just F to allow downstream crates to implement
// FnThunk for higher kinded bare function types through a macro as needed
unsafe impl<R, F> FnThunk<cc::C, unsafe extern "C" fn() -> R> for (cc::C, F) where F: Fn() -> R {}
unsafe impl<T, R, F> FnThunk<cc::C, unsafe extern "C" fn(T) -> R> for (cc::C, F) where F: Fn(T) -> R {}
// Higher parameter counts omitted for brevity
/* hooking library */
struct HookCtx<B: Copy> {
original: B
}
struct Hook<'a, B: Copy>(PhantomData<(B, &'a ())>);
impl<'a, B: Copy> Hook<'a, B> {
fn new<F: 'a + Sync>(hook: F) -> Self
where
B: FnPtr,
(<B as FnPtr>::CC, F): FnThunk<<B as FnPtr>::CC, B>
{
todo!()
}
fn with_cc<CC, F: 'a + Sync>(cc: CC, hook: F) -> Self
where (CC, F): FnThunk<CC, B>
{
todo!();
}
fn with_context<CC, F: 'a + Sync, P>(cc: CC, hook: P) -> Self
where
P: FnOnce(HookCtx<B>) -> F,
(CC, F): FnThunk<CC, B>
{
todo!();
}
}
fn test_inference() {
// Case 1: properly infers F
let hook: Hook<'_, unsafe extern "C" fn(usize) -> usize>;
hook = Hook::new(|arg| arg + 42);
// Case 2: properly infers B
let hook = Hook::with_cc(cc::C, |arg: u32| arg + 42);
// Case 3: also properly infers B
let hook = Hook::with_context(cc::C, |ctx| move || false);
// Case 4: also properly infers B
let hook = Hook::with_context(cc::C, |ctx| move |arg: u32| {
println!("trampoline to original: {:016x}", ctx.original as usize);
arg + 42
});
// (FAILS) Case 5: fails to infer B
let hook = Hook::with_context(cc::C, |ctx| move |arg: u32| -> u32 {
let _ = unsafe { (ctx.original)(arg) };
arg + 42
});
// (FAILS) Case 6: also fails to infer B, even when it's in the type of the assigned variable
let hook: Hook<'_, unsafe extern "C" fn(u32) -> u32>;
let hook = Hook::with_context(cc::C, |ctx| move |arg: u32| -> u32 {
let _ = unsafe { (ctx.original)(arg) };
arg + 42
});
// Case 7: Works when the turbofish operator is used
let hook = Hook::<unsafe extern "C" fn(u32) -> u32>::with_context(cc::C, |ctx| move |arg| {
let _ = unsafe { (ctx.original)(arg) };
arg + 42
});
// Case 8: Works when the outer closure arg is typed
let hook = Hook::with_context(cc::C, |ctx: HookCtx<unsafe extern "C" fn(u32) -> u32>| move |arg| {
let _ = unsafe { (ctx.original)(arg) };
arg + 42
});
}
What's particularly surprising to me is that case 4 in the above code type infers successfully, but 5 doesn't. Yet both require knowing the generic parameters of ctx to some extent (for the cast to be valid in case 4 and the function pointer call in case 5).
Is there a general "rule" of how closure type inference works that prevents cases 5 and 6 from working no matter what, and does anyone know if the API or trait definitions could be modified to allow the same thing to be done, while allowing the failing cases to type infer successfully?