Create a callback inside a function using another function, passed as argument, inside the callback itself

help
1

Hi again :slight_smile:

Sorry for the title, but it is the most precise I was able to find.

I found myself tricked in a problem quite interesting and I would like your help.

I have an external "C" library that accept a callback.
Since I (should be) am writing an abstraction over this library I want to write a safer and more ergonomic interface.

The overall goal is to have a wrapper where I can pass rusty parameters, make some magic with these parameters inside the body of the wrapper and transforming them in something more C-like, and finally invoke the extern C function.

The function I am invoking is this one:

extern "C" {
    #[link_name = "RedisModule_CreateCommand"]
    pub static mut RedisModule_CreateCommand:
               ::std::option::Option<unsafe extern "C" fn(
                    ctx: *mut RedisModuleCtx,
                    name: *const ::std::os::raw::c_char,
                    cmdfunc: RedisModuleCmdFunc,
                    strflags: *const ::std::os::raw::c_char,
                    firstkey: ::std::os::raw::c_int,
                    lastkey: ::std::os::raw::c_int,
                    keystep: ::std::os::raw::c_int)
                                         -> ::std::os::raw::c_int>;
}

where

pub type RedisModuleCmdFunc =
    ::std::option::Option<unsafe extern "C" fn(
           ctx: *mut RedisModuleCtx,
           argv: *mut *mut RedisModuleString,
           argc: ::std::os::raw::c_int)
                              -> ::std::os::raw::c_int>;

Finally the function I am trying to implement looks like this:

type RedisModuleCmdFuncSafe = Fn(Context, Vec<String>) -> i32;

fn CreateCommand<F>(ctx: Context,
                    name: String,
                    f: &F,
                    flags: String,
                    first_key: i32,
                    last_key: i32,
                    key_step: i32)
                    -> i32
    where F: Fn(Context, Vec<String>) -> i32
{
    let c_name = CString::new(name).unwrap();
    let ptr_name = c_name.as_ptr();
    let c_flags = CString::new(flags).unwrap();
    let ptr_flags = c_flags.as_ptr();
    let function = |ctx: *mut ffi::RedisModuleCtx,
                                  argv: *mut *mut ffi::RedisModuleString,
                                  argc: i32|
                                  -> i32 {
        let context = Context { ctx: ctx };
        let argvector = parse_args(argv, argc).unwrap();
        f(context, argvector)
    }; 
    let option_function = Some(function);
    unsafe {
        ffi::RedisModule_CreateCommand.unwrap()(ctx.ctx,
                                                ptr_name,
                                                option_function,
                                                ptr_flags,
                                                first_key,
                                                last_key,
                                                key_step)
    }

}

Clearly this doesn't work since I am using a closure instead of a function and, however if I use a closure f should be somewhere accessible after the completion of the function.

I believe I kinda get why it is not working, but I am not really able to see a solution myself.

Could you guys help?

2

Not clear what you are asking,
if question is "how to fix your code",
then obviously you need pass fn instead of Fn to CreateCommand,

if question how to use C API in such wayt that pass closure instead of
function pointer,

then there is no clear way to do it, if authors not design API for that.
I mean, for example register callback function should accpet void *opaque_addon_data
argument, and should be such argument in callback by itself.

There is of course one not clear way to do it: generate code on the fly,
move closure to Box, box into raw, and then generate machine code that accept
callback params and use aboslute adress from raw to call closure.
But, almost all modern OSes should prevent code execution from heap,
so to make it works, you should generate code into .so or .dll and load it into your address space.

3

Fundamentally, closures (Fn types generally) are strictly more capable than simple callbacks: they have state, in the form of the closed-over variables. This is more obvious looking at the types of the various Fn* traits: the value being borrowed in a FnMut call or moved in a FnOnce call is exactly the parameter holding the closed-over state. The closure cannot meaningfully be separated from its state, so you can't convert an arbitrary closure into a C-compatible function pointer.

It's possible in principle to generate a unique function pointer for a given Fn-typed value, but you need run-time code generation to make it work, and Rust doesn't have any built-in facilities for that. There might be a crate… Otherwise, you'll have to live with the limitations of the fn family of types (bare function pointers).

4

Hi @Dushistov, Hi @derspiny,

sorry, for the question not being so clear.

The point of the whole post is to understand first how the mechanism works and then to finally have the code working properly :slight_smile:

The author of the C library exposed this function RedisModule_CreateCommand that need a pointer to function of type RedisModuleCmdFunc.
RedisModuleCmdFunc takes in input 3 parameters and provide an output of 1.

What I am doing is to create a safer and more rusty interface.

The real problem is that I don't want a closure, I need a

unsafe extern "C" fn(
           ctx: *mut RedisModuleCtx,
           argv: *mut *mut RedisModuleString,
           argc: ::std::os::raw::c_int)
                              -> ::std::os::raw::c_int

The simplest way to illustrate my problem is showing that this code doesn't compile:

fn CreateCommand(ctx: Context,
                 name: String,
                 f: RedisModuleCmdFuncSafe,
                 flags: String,
                 first_key: i32,
                 last_key: i32,
                 key_step: i32)
                 -> i32 {
    let c_name = CString::new(name).unwrap();
    let ptr_name = c_name.as_ptr();
    let c_flags = CString::new(flags).unwrap();
    let ptr_flags = c_flags.as_ptr();
    unsafe extern "C" fn function(ctx: *mut ffi::RedisModuleCtx,
                                  argv: *mut *mut ffi::RedisModuleString,
                                  argc: i32)
                                  -> i32 {
        let context = Context { ctx: ctx };
        let argvector = parse_args(argv, argc).unwrap();
        f(context, argvector)
    }

    let option_function = Some(function);
    unsafe {
        ffi::RedisModule_CreateCommand.unwrap()(ctx.ctx,
                                                ptr_name,
                                                option_function,
                                                ptr_flags,
                                                first_key,
                                                last_key,
                                                key_step)
    }
}

There are two problem here:

  1. The function f is not on the local scope
    error[E0434]: can't capture dynamic environment in a fn item; use the || { ... } closure form instead
    │ --> src/lib.rs:248:9
    │ |
    │248 | f(context, argvector)
    │ | ^

However I cannot simply use a closure since I actually need a function to pass to the C code.

  1. The type of the function that I generate in this way still is not correct ( I believe the problem are quite correlated ).

    error[E0308]: mismatched types
    │ --> src/lib.rs:255:49
    │ |
    │255 | option_function,
    │ | ^^^^^^^^^^^^^^^ expected fn pointer, found fn item
    │ |
    │ = note: expected type std::option::Option<unsafe extern "C" fn(*mut ffi::RedisModuleCtx, *mut *mut ffi::RedisModuleString, i32) -> i32>
    │ found type std::option::Option<unsafe extern "C" fn(*mut ffi::RedisModuleCtx, *mut *mut ffi::RedisModuleString, i32) -> i32 {Creat$Command::function}>

Are there any way around this?

5

If simplify our answers, you have two option:

  1. Do exactly what interface of C library want: pass not closure,
    but pure function

  2. Generate code on the fly

For prototype (2) you can write several pure function that work with global variables,
each function with unique one, and use this cache of pure functions to work with closures,

like

lazy_static! {
    static ref F1: Option<Box<Fn>> = { ... };
    static ref F2: Option<Box<Fn>> = { ... };
}

unsafe extern "C" fn f1(ctx: *mut ffi::RedisModuleCtx,
                                  argv: *mut *mut ffi::RedisModuleString,
                                  argc: i32)
                                  -> i32 {
 //work with F1
}

unsafe extern "C" fn f2(ctx: *mut ffi::RedisModuleCtx,
                                  argv: *mut *mut ffi::RedisModuleString,
                                  argc: i32)
                                  -> i32 {
 //work with F2
}

optional/additional variant would be modify C library that you are trying to work with.

And note, that this is not specific for Rust issue, in all other language,
even in C and assembler you have the same issue, if you pass context to callback,
while callback have no arguments for this.

6

Hi,

hum, still I don't get it.

Look at this other approach I am considering.

fn CreateCommandIntern2(f: (Box<fn(Context, Vec<String>) -> i32>))
                        -> (Box<unsafe extern "C" fn(*mut ffi::RedisModuleCtx,
                                                     *mut *mut ffi::RedisModuleString,
                                                     i32)
                                                     -> i32>) {
    unsafe extern "C" fn fn1(ctx: *mut ffi::RedisModuleCtx,
                             argv: *mut *mut ffi::RedisModuleString,
                             argc: i32)
                             -> i32 {
        let context = Context { ctx: ctx };
        let argvector = parse_args(argv, argc).unwrap();
        f(context, argvector)
    }
    Box::new(fn1)
}

Still it says:

  │error[E0434]: can't capture dynamic environment in a fn item; use the || { ... } closure form instead
  │   --> src/lib.rs:241:9
  │    |
  │241 |         f(context, argvector)
  │    |         ^

But now f shouldn't be a simple pointer to the heap? What it can't capture?

What is the difference with this python script?

def foo(f):
    def g(a, b):
        def h(c):
            return f(a + 1) + b + c
        return h
    return g
        
def j(x):
    return x + 1

> foo(j)(1, 2)(3)
8
7

The issue here isn't f; it is f1 (which tries to capture f).

What is the difference with this python script?

def f(x): in Python is far closer in semantics to let f = |x| {...} in Rust than it is to fn f(x) {...}

8

There is difference: function object and pure function.
In our context function object - some structure with data, for which exists implemntation of std::ops::Fn. While pure function is just pointer to code generated by compiler.

The important difference that to call Fn(i32) you need actually give it two arguments: i32 and self, while to call pure function fn(i32) you need just i32. Thus if there is already compiled code that work with fn, it can not works with Fn, you need modify and recompile it.

Because of Fn accept self, you can attach context to it, this context allocate on the heap,
so for example with API that accept Fn you can take one function, wrap it with closure,
and pass closure, instead of function parameter,

but you have existing C API, that not accept Fn, so you have to give it pure function,
that not depend on context, only on input arguments and may be global variables,
not local variables for current function.

In your python code you actually create Fn in terms of Rust, not pure function,
if you modify your python example in such way:

x = foo(j)
print(x.__call__)

then you see that this is object, with some address in the heap.

So in python, you actually pass around objects allocated in the heap with
attached integer arguments. You can do the same thing in Rust, and pass around Box<Fn>,
but if your Rust/C function accept only fn, then you should pass only pure fuction without attached context.

9

To put it another way, an fn is a plain function pointer (void*), whereas an Fn is a function pointer and some context; it might like like this in C:

struct Fn {
    void *fn_pointer;
    void *context;
};

And then calling a given Fn would look like fn.fn_pointer(fn.context, args...);.

If your C function takes a single pointer (void*), you need a single value to give it. An Fn is never that, because there's no one location (void*) that sums up the content of the closure.

When people are suggesting you generate code, they're saying you could generate (perhaps at compile time) a separate fn with all the available context — that way, you'd have a single function pointer, i.e. an fn, not an Fn.

Often C APIs will allow you to register some void* "context" along with a function pointer, which is passed opaquely through to your callback along with other arguments. If the RedisModulesSDK API did so, you could manually construct the closure out of a struct yourself, but it doesn't look like it does.

10

Wooow, thanks everybody for the feedback, it was extremely valuable.

I actually got the difference between fn and Fn and it makes sense.

It is still a little weird because at the very end what I am doing is simply shuffle some arguments and then call from a function another function.
Given two function:

h(C)->B the C API
g(A)->C my wrapper

I would simply like to obtain:

f(A)->B which is nothing more than the composition of g() and h().

I actually see where is the problem, I believe that I also understand it; it just feel weird...

Thanks everybody who comment in this thread, the help of all of you was extremely valuable.

11

A trick to do this alternate to @Dushistov global static suggestion is to;
Move your rust closure into the rust function.
The function then creates a thread, moves the closure to it and stores it in local storage.
A utility function for c callback gets and uses the rust closure.

Downside; Assumes that c calls should be safe in alternate thread. Slow if it is intended to be called a lot. May need extra effort making closed variables pass across to other thread.

Also have to pick which closure to box Fn, FnMut, FnOnce

p.s. not tired this.