Callbacks and external libraries

help
1

Hey folks,

I'm building a library that wraps cross-platform text-to-speech APIs. I'm trying to support callback mechanisms for when utterances begin and end speaking, but each library does things slightly differently and I'm having lots of trouble creating a generic interface. Here, for instance, is how one lower-level library implements its callbacks:

pub fn on_begin(&self, f: Option<fn(u64, u64)>)

In that instance, the second u64 is the ID of the synthesizer. So I can do a lookup in a lazy_static CALLBACKS HashMap keyed off of the ID.

Unfortunately, not all of these libraries pass a client ID. WinRT, for instance, seems like it just passes the tracks that started and stopped playing. Seems like this means I have to arbitrarily assign a unique ID to the synthesizer, but I can't seem to pass variables into closures. I.e. if I do:

        let id: u64 = 0; // Assigned elsewhere
        sd.0.on_begin(Some(|_msg_id, client_id| {
            let callbacks = CALLBACKS.lock().unwrap();
            let cb = callbacks.get(&client_id); // works
            let callbacks = CALLBACKS.get(&id); // doesn't
        }));

Is there any way to get a primitive from outside into a closure? Failing that, is there some other approach I should be using? I know I have a working solution here, but that's only because this particular synth is nice enough to assign and pass its own client IDs, and I don't want to go too far down this rabbit trail if I'm just going to hit a brick wall with the 5 others I need to cover. I think this is the only one that actually assigns an ID to itself, in fact.

Thanks for the help.

2

What you need here is the move keyword:

        let id: u64 = 0; // Assigned elsewhere
        sd.0.on_begin(Some(move |_msg_id, client_id| {
            let callbacks = CALLBACKS.lock().unwrap();
            let cb = callbacks.get(&client_id); // works
            // id is moved into the closure
            let callbacks = CALLBACKS.get(&id);
        }));
3

Sorry, maybe should have included the full error:

error[E0308]: mismatched types
  --> src/backends/speech_dispatcher.rs:27:28
   |
27 |           sd.0.on_begin(Some(move |_msg_id, client_id| {
   |  ____________________________^
28 | |             let mut speaking = SPEAKING.lock().unwrap();
29 | |             speaking.insert(client_id, true);
30 | |             let callbacks = CALLBACKS.lock().unwrap();
31 | |             let cb = callbacks.get(&id);
32 | |         }));
   | |_________^ expected fn pointer, found closure
   |
   = note: expected fn pointer `fn(u64, u64)`
                 found closure `[closure@src/backends/speech_dispatcher.rs:27:28: 32:10

So the problem seems to be that using something from the environment prevents it from being cast to an fn(u64, u64). My question is whether there's a way around this limitation either at the callsite, or in this particular library? (In this instance, I control the fn(u64, u64) implementation.) move didn't seem to do it.

Last night it occurred to me that, as long as I can access statics and use lazy_static, maybe I can maintain and clean up mappings from whatever I get in the closure to a global ID. So in this case it's fairly easy since the client generates them, but in the case where a client doesn't, I can map the data it passes in to the universal ID, look that up when said data goes into the callback, then more aggressively clean up. Another solution would be better, but that's the best I came up with.

Thanks for the help.

4

That looks problematic, as you can't store any additional data. So you have to use the provided arguments to somehow get the data you need.

5

So I got things sort of working using fn, but every time I try accessing anything from outside, even via move, I get an error like:

   | |_________^ expected fn pointer, found closure

Now I'm trying to rewrite things to use FnMut to see if that helps and am hitting other issues. Specifically, I'm trying to convert:

#[derive(Clone, Copy)]
struct Callbacks {
    begin: Option<fn(u64, u64)>,
    end: Option<fn(u64, u64)>,
    index_mark: Option<fn(u64, u64, String)>,
    cancel: Option<fn(u64, u64)>,
    pause: Option<fn(u64, u64)>,
    resume: Option<fn(u64, u64)>,
}

to:

#[derive(Default)]
struct Callbacks {
    begin: Option<Box<FnMut(u64, u64)>>,
    end: Option<Box<FnMut(u64, u64)>>,
    index_mark: Option<Box<FnMut(u64, u64, String)>>,
    cancel: Option<Box<FnMut(u64, u64)>>,
    pause: Option<Box<FnMut(u64, u64)>>,
    resume: Option<Box<FnMut(u64, u64)>>,
}

I have code like:

unsafe extern "C" fn cb_im(msg_id: u64, client_id: u64, state: u32, index_mark: *mut i8) {
    let index_mark = CStr::from_ptr(index_mark);
    let index_mark = index_mark.to_string_lossy().to_string();
    let state = match state {
        SPDNotificationType_SPD_EVENT_INDEX_MARK => Notification::IndexMarks,
        _ => panic!("Unknown notification received in IM callback: {}", state),
    };
    if let Some(c) = callbacks.lock().unwrap().get(&client_id) {
        let f = match state {
            Notification::IndexMarks => &c.index_mark,
            _ => panic!("Unknown notification type"),
        };
        if let Some(mut f) = f {
            f(msg_id, client_id, index_mark);
        }
    }
}

After the transition to MutFn, this now throws:

error[E0507]: cannot move out of `f.0` which is behind a shared reference                                                            
   --> speech-dispatcher/src/lib.rs:143:30                                                                                           
    |                                                                                                                                
143 |         if let Some(mut f) = f {                                                                                               
    |                     -----    ^                                                                                                 
    |                     |                                                                                                          
    |                     data moved here                                                                                            
    |                     move occurs because `f` has type `std::boxed::Box<dyn std::ops::FnMut(u64, u64, std::string::String)>`, which does not implement the `Copy` trait

I've read the description for E0507, and am not sure which of the three paths are open to me, if any. Not sure if a RefCell would help either, but nested smart pointers confuse me because sometimes I don't know what should wrap what. I.e. should my entire struct be in a RefCell, or the individual members to which I need access? CALLBACKS is:

lazy_static! {
    static ref callbacks: Mutex<HashMap<u64, Callbacks>> = {
        let m = HashMap::new();
        Mutex::new(m)
    };
}

So maybe I need an Arc somewhere in there? :confused: Not sure anymore, and I've spent the past week implementing callbacks from half a dozen different libraries so my head is starting to spin.

Thanks for any help.

6

If the API takes a fn pointer, then there's no way to store environment data: it's just a code reference. The only way to inject your own data into one is to store something like a HashMap in a static variable, and use the parameters you're given to look up the correct entry.

7

Right, I get that. What I'm saying here is that I control the API, so
I'm looking for help to make it better.

8

If you control the C-side of the API as well, the best thing to do is to store a void* user_data alongside the callback function, and give it back as an argument. Then you can store a pointer to the closure object there (probably via Box::into_raw()) and write a stateless function that passes the real parameters through to the closure as the callback function.

9

I don't control the C side, but if I'm wrapping it, can't I just do what
I'm trying to here? That is, take the parameters, look up an FnMut
from a lazy_static, then be back in Rust land? If so, that's what I'm
asking about today.

To be succinct, are you telling me that what I'm trying here is
impossible? I thought maybe some combination of Arc and RefCell
would work around this issue, but I'm not sure what that combination
might be.

Thanks.

10

Sorry; I didn't really look at the code you posted. The general approach should work, but you'll need to get a mutable reference out of the Mutex. Something like this (untested):

    if let Some(c) = callbacks.lock().unwrap().get_mut(&client_id) {
        let f = match state {
            Notification::IndexMarks => &mut c.index_mark,
            _ => panic!("Unknown notification type"),
        };
        if let Some(mut f) = f.as_mut() {
            f(msg_id, client_id, index_mark);
        }
    }
11

Thanks, I'll give that a shot and report back. Do I still need a
RefCell? Got impatient, started down that rabbit hole, and hit another
dead end. Wondering if I should undo that change.

12

You shouldn't: the Mutex is taking care to ensure there's only one access at a time, which is the same job that RefCell does.

13

Oh wow, it was the as_mut() in:


         if let Some(mut f) = f.as_mut() {

that unblocked me. Thanks so much. Now I'm getting errors about static
lifetimes in dependent code, but that's a problem I can work around,
potentially with another solution.

Thank you!

14

If that's the callback API you've got to work with then you're out of luck. That Callbacks struct has no way for the user to provide state for use with the callbacks (usually you'll have a void *user_data field and a destructor), so you are forced to use global static variables.

This is pretty typical for C libraries, but Rust will force you to do some sort of synchronisation (e.g. via lazy_static!() and a Mutex) to let the callbacks communicate with each other and the outside world.

I wrote an article about passing callbacks to native code a while back...

Don't forget that a function pointer and a closure are two very different things from a memory perspective. A function pointer is just a pointer to some machine code in your program binary, while
a closure is more akin to a custom struct with a std::ops::Fn() impl.

You can't convert a Box<dyn Fn(...)> to a function pointer because:

a) it's a trait object, so there are actually two pointers (see fat pointer)
b) even if you got a pointer to the Fn impl's call() method, you wouldn't have a pointer to the closure's data (because the Callbacks struct has way to provide user-defined data) that can be passed to the method call, and
c) A closure's call() method uses the "rust-call" calling convention, so it's not safe to call the function from non-Rust code

Also, don't forget to specify a calling convention when passing function pointers to C code! (i.e. f: Option<unsafe extern "C" fn (...)>)

If you don't specify a calling convention then the compiler will generate a function that passes arguments around and sets up the stack in a way Rust expects. This may or may not match the calling convention used by the library (Rust's ABI is unspecified and subjecet to change), and if there's a mismatch you'll run into funny crashes because arguments aren't where the code expects or you mess up your stack.