Saving state over dll calls to Rust dll

I'm changing my application to a lib so I can build my GUI in another language. I can't figure out how to hold state over the calls to the Rust dll. In an app main() can hold onto state until the app exits. In a dll the call has to return which is fine as it just needs to start the main thread of the library. All I'm trying to do at the moment is get it to start up and then close it down. To close() down at the very least it needs the sending end of the channel which was created in the start() function. How can I store this so the close() function can get hold of it. I'm not calling this from a Rust client, it will probably be Python.
I'm probably missing something very simple here!

Often times libraries return some sort of context pointer from their startup function, which can then be passed to subsequent calls into the library (like the close function).

Sometimes libraries just choose to use global variables instead, but that can be a pain to work with in some circumstances.

3 Likes

I don't think I can pass back something like a channel or some other object ref so not sure what context I could use. This is as close as I've got but its not working because the send and receive are not the same channel and can't seem to set both in the same static call.

lazy_static! {
    pub static ref r: crossbeam_channel::Receiver<i32> = unbounded().1;
    pub static ref s: crossbeam_channel::Sender<i32> = unbounded().0; 
}

#[no_mangle]
pub extern "C" fn sdrlib_run() {

    // Start library
    let handle = app::app_start(r.clone());
    println!("Started Rust SDR Server");
}

#[no_mangle]
pub extern "C" fn sdrlib_close() {
    // Close library
    println!("\n\nRust SDR Server shutdown...");
    s.send(0);
    // Need to wait for handle here
    thread::sleep(Duration::from_millis(1000));
    println!("Rust SDR Server closing...");
    thread::sleep(Duration::from_millis(1000));
}

Found the right syntax. Not pretty but can't think of any alternative.

lazy_static! {
    pub static ref CH: (crossbeam_channel::Sender<i32>, crossbeam_channel::Receiver<i32>) = unbounded();
}

#[no_mangle]
pub extern "C" fn sdrlib_run() {

    // Start library
    let handle = app::app_start(CH.1.clone());
    println!("Started Rust SDR Server");
}

#[no_mangle]
pub extern "C" fn sdrlib_close() {
    // Close library
    println!("\n\nRust SDR Server shutdown...");
    CH.0.clone().send(0);
    // Need to wait for handle here
    thread::sleep(Duration::from_millis(5000));
    println!("Rust SDR Server closing...");
    thread::sleep(Duration::from_millis(1000));
}

Trying to get this a bit more sensible but falling over on the usual
error[E0596]: cannot borrow data in dereference of init as mutable. Do I have to implement this defefMut or is that a red herring.
--> src\lib.rs:69:5
|
69 | init.set_handle(handle);
| ^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable
|
= help: trait DerefMut is required to modify through a dereference, but it is not implemented for init

error[E0596]: cannot borrow data in dereference of init as mutable
--> src\lib.rs:79:22
|
79 | if let Some(h) = init.handle.take() {
| ^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

pub struct InitData {
    // Channel
    pub sender : crossbeam_channel::Sender<i32>,
    pub receiver : crossbeam_channel::Receiver<i32>,
    pub handle: option::Option<JoinHandle<()>>,
}

impl InitData {
    pub fn new() -> InitData {
        let (s, r) = unbounded();
        InitData {
            sender: s,
            receiver: r,
            handle: None,
        }
    }
    pub fn set_handle(&mut self, h:JoinHandle<()>) {
        self.handle = Some(h);
    }
}

lazy_static! {
    pub static ref init: InitData = InitData::new();
}

#[no_mangle]
pub extern "C" fn sdrlib_run() {

    // Start library
    let handle = app::app_start(init.receiver.clone());
    init.set_handle(handle);
    println!("Started Rust SDR Server");
}

#[no_mangle]
pub extern "C" fn sdrlib_close() {
    // Close library
    println!("\n\nRust SDR Server shutdown...");
    init.sender.clone().send(0);

    if let Some(h) = init.handle.take() {
        h.join().expect("Failed to join application thread!");
    }
    
    println!("Rust SDR Server closing...");
    thread::sleep(Duration::from_millis(1000));
}

Red herring. Lazy static is for something that you are initializing once and never change.

What you need it just simple Mutex<Option> or something like that.

That will work, but if you want to pass a context on this or other methods; it depends on the FFI particulars, but generally you will have a way of wrapping up a pointer. Then you can use apis like Box::into_raw and Box::from_raw to create and consume the pointer respectively, and dereferencing the pointer for non consuming use.

If you haven't already checked it out, the rustonomicon has a section on FFI which should help you avoid many of the crashier mistakes. Unfortunately, while it has a simple example of using a context pointer which might be useful for reference, it depends on the stack owning the lifetime (Rust calling C), rather than explicitly showing handing out pointer ownership.

Thanks but I'm just floundering now. I don't know if this is even close to right. Can you help me with handling this poison error. I just don't understand the error message.
error[E0308]: match arms have incompatible types
--> src\lib.rs:66:31
|
64 | / match h.lock() {
65 | | Ok(h) => h.join(),
| | -------- this is found to be of type Result<(), Box<dyn Any + Send>>
66 | | Err(e) => e.into_inner(),
| | ^^^^^^^^^^^^^^ expected enum Result, found struct MutexGuard
67 | | }
| |_________________- match arms have incompatible types
|
= note: expected enum Result<(), Box<dyn Any + Send>>
found struct MutexGuard<'_, JoinHandle<()>>

pub struct InitData {
    // Channel
    pub sender : crossbeam_channel::Sender<i32>,
    pub receiver : crossbeam_channel::Receiver<i32>,
    pub handle: option::Option<Mutex<JoinHandle<()>>>,
}

impl InitData {
    pub fn new() -> InitData {
        let (s, r) = unbounded();
        InitData {
            sender: s,
            receiver: r,
            handle: None,
        }
    }
    pub fn set_handle(&mut self, h:JoinHandle<()>) {
        self.handle = Some(Mutex::new(h));
    }

    pub fn wait_handle(&mut self) {
        match &self.handle {
            None => (),
            Some(h) => {
                match h.lock() {
                    Ok(h) => h.join(),
                    Err(e) => e.into_inner(),
                }
            }
        }
    }
}

lazy_static! {
    pub static ref init: InitData = InitData::new();
}

#[no_mangle]
pub extern "C" fn sdrlib_run() {

    // Start library
    let handle = app::app_start(init.receiver.clone());
    init.set_handle(handle);
    println!("Started Rust SDR Server");
}

#[no_mangle]
pub extern "C" fn sdrlib_close() {
    // Close library
    println!("\n\nRust SDR Server shutdown...");
    init.sender.clone().send(0);
    init.wait_handle();
    println!("Rust SDR Server closing...");
    thread::sleep(Duration::from_millis(1000));
}

You have to laugh after Rust send you round in circles for 2 hours suggesting things that never work. I don't know how something so simple can be made so complex. I just need to wait for the handle and I've tried it a hundred ways and failed every time. I did look at the rustonomicon which is where I got the info to get something working but it's a very simple example they give. I don't know how to wrap a pointer and pass it back to the C level but maybe that's the only way to achieve this.

I think you are overestimating the ability of compiler to write programs. If it would have included sufficient AI to always give you correct suggestions then you wouldn't have been needed.

Globals are never simple. But if you understand what the problem with globals solution is 100% obvious.

Okay, you need to keep some global into… what does that mean? Right: you need some kind of lock to guarantee that you don't have concurrency issues.

So we would have something like:

use std::sync::Mutex;

static GLOBAL_DATA: Mutex<Option<MyData>> = Mutex::new(None);

struct MyData {
    pub foo: String,
    pub bar: i32,
    pub baz: Vec<i32>,
}

pub fn init() {
    let mut guard = GLOBAL_DATA.lock().unwrap();
    let my_data = &mut *guard;
    my_data.replace(MyData { foo: "Hello".to_string(), bar: 42, baz: vec![1, 2]});
}

pub fn inc_bar() {
    let mut guard = GLOBAL_DATA.lock().unwrap();
    let my_data = guard.as_mut().expect("Call init before using inc_bar");
    my_data.bar += 1;
}

Easy, simple and something we would have to do in C or C++ anyway if we want reliable code.

Except in Rust compiler asks you to do you homework upfront and not rely on the advices of experiences developers in review.

Don't consider Rust compiler as an adversary which you have to conquer. Rust compiler is your friend.

Reliable but dumb. It may tell you that something that you are trying to invent would never work (reliably at least), but it couldn't invent the solution. It's your job.

Here's how you might use a context pointer instead of a global, just for the sake of completeness.

#![allow(clippy::all)]

use std::{
    thread::{self, JoinHandle},
    time::Duration,
};

use crossbeam_channel::unbounded;

pub struct InitData {
    // Channel
    pub sender: crossbeam_channel::Sender<i32>,
    pub receiver: crossbeam_channel::Receiver<i32>,
    pub handle: Option<JoinHandle<()>>,
}

impl InitData {
    pub fn new() -> InitData {
        let (s, r) = unbounded();
        InitData {
            sender: s,
            receiver: r,
            handle: None,
        }
    }
    pub fn set_handle(&mut self, h: JoinHandle<()>) {
        self.handle = Some(h);
    }
}

mod app {
    use std::thread::JoinHandle;

    use crossbeam_channel::Receiver;

    pub fn app_start(_receiver: Receiver<i32>) -> JoinHandle<()> {
        todo!()
    }
}

// Using the strategy from https://doc.rust-lang.org/nomicon/ffi.html#representing-opaque-structs for creating opaque types
// This allows us to avoid exposing any details about InitData to the C API
#[repr(C)]
pub struct AppContext {
    _data: [u8; 0],
    _marker: core::marker::PhantomData<(*mut u8, core::marker::PhantomPinned)>,
}

#[no_mangle]
pub extern "C" fn sdrlib_run() -> *mut AppContext {
    let mut init = Box::new(InitData::new());
    // Start library
    let handle = app::app_start(init.receiver.clone());
    init.set_handle(handle);
    println!("Started Rust SDR Server");

    Box::into_raw(init).cast()
}

#[no_mangle]
pub extern "C" fn sdrlib_close(ctx: *mut AppContext) {
    let mut init = unsafe { Box::from_raw(ctx.cast::<InitData>()) };
    println!("\n\nRust SDR Server shutdown...");
    init.sender
        .clone()
        .send(0)
        .expect("Channel should not be closed yet");

    if let Some(h) = init.handle.take() {
        h.join().expect("Failed to join application thread!");
    }

    println!("Rust SDR Server closing...");
    thread::sleep(Duration::from_millis(1000));
}

Also note you're currently calling methods that may panic (unwrap and expect) inside functions that are going to be called from another language. It's currently undefined behavior to allow an unwinding panic to cross an FFI boundary[1]


  1. though this may change in the near future, see the C-unwind tracking issue ↩︎

1 Like

Thanks. There is a lot of information there which I need to digest. I had most things figured out when it was an application but using it as a lib was a lot harder than I thought. I fully admit my understanding is lacking somewhat but I will persevere. I like the idea of passing back a context because its a lot neater than the alternative so I will give that a go.

And I probably should apologize and clarify a bit.

Situation with globals is complicated because they are really unsafe unless you would protect them with mutex (it's not Rust specific, as a “rule of thumb” globals must be protected by mutex in C, C++ and other languages… but they wouldn't tell you that) — yet clear solution in Rust only became availabel in august.

That explains both why tutorials are so vague and also why compiler doesn't help much: it would be stupid for a compiler to suggest some third-party crate but before Rust 1.63 it really had no other choice.

I guess it may be time to file bugs about diagnosis, though: since there are an std-based solution these days maybe it's time to start recommending it.

1 Like

I was a little frustrated and didn't do my usual of sitting on a reply for a while before posting. I think in general the compiler actually does a pretty good job. I re-implemented a large C program in Rust and all I had was a few panics on buffer overrun. In contrast when I did the C program some years ago it took me weeks to get it stable. When I can get into flow its a nice language, feels a bit Python like at high level and a bit C like at low level and reminds me a bit of Pascal in that if it compiles you pretty much know it will work. The reason I'm doing the lib is because I'm not happy with any of the GUI implementations at present. I got quite a long way with egui but the compromises were too much so its back to PyQt for the GUI.

1 Like

Also, there's a reason the documentation for unsafe code is named after the Necronomicon, anything to do with it, including FFI, is well into expert territory (even though it's the standard state of affairs in C) - don't feel bad for finding this hard (even after about two years I still screw it up all the time)

Regarding the state of GUI, yeah it's not amazing right now. There's lots of experimental native Rust crates that look like they could be amazing, but nothing I would ship on right now. I feel best about Tauri right now, but I haven't had the chance to put it into real use yet, and if you're more familiar with Qt than HTML it makes sense to stick with that.

You're probably thinking about this already, but one recommendation I would make is, if possible, to copy Deno and others and try to minimize your FFI interface, building something like a simple message-passing API to minimize the things that aren't self describing. Code generation is another approach that might work better with Qt and C++ in general, but it's a bit messy to integrate with building.

1 Like

The context pointer works nicely and is a much nicer solution so thanks for posting that.

I tried FLTK and got a fair way before realising it had no alpha channel. The egui stuff was much easier to work with but as I say too many compromises. I might have a look at some of the web based stuff again but I would really be looking for something as functional as Qt but in pure Rust so its built to fit with the language but I guess that's a fair way off. My C implementation would work as a lib or a server with Json over UDP which I may well replicated later on.

Looks like I still have a problem here. So I added one interface.

#[no_mangle]
pub extern "C" fn sdrlib_freq(ctx: *mut AppContext, freq: u32) {
    println!("f is: {:?}, {}", ctx, freq);
    let mut init = unsafe { Box::from_raw(ctx.cast::<InitData>()) };
    let msg = messages::AppMsg {msg_type: messages::AppMsgType::Frequency, param1: freq};
    init.sender
        .clone()
        .send(msg)
        .expect("Channel rejected frequency command");
}

and this is the receiver

pub fn app_process(&mut self, receiver: crossbeam_channel::Receiver<messages::AppMsg>) {

        loop {
            // Check for messages
            let r = receiver.try_recv();
            println!("Got {:?}", r);
            match r {
                Ok(msg) => {
                    match msg {
                        messages::AppMsg {msg_type: messages::AppMsgType::Terminate, param1: _} => {
                            break;
                        },
                        messages::AppMsg {msg_type: messages::AppMsgType::Frequency, param1: freq} => {
                            println!("Setting: {}", freq);
                            self.i_cc.lock().unwrap().cc_set_rx_tx_freq(freq);
                        },
                        _ => (),
                    };
                },
                // Do nothing if there are no message matches
                _ => (),
            };
            thread::sleep(Duration::from_millis(100));
        }
    }

and this is the trace after calling this new function twice. It looks like sender goes out of scope when the fn exits so I can only send one message. Not sure if there is a way to stop that happening as Rust isn't holding this reference although it still exists in memory.
Got Err(Empty)
Got Err(Empty)
Got Err(Empty)
Got Err(Empty)
etc
f is: 0x11d6db0, 7150000
Got Ok(AppMsg { msg_type: Frequency, param1: 7150000 })
Setting: 7150000
Got Err(Disconnected)
Got Err(Disconnected)
etc
f is: 0x11d6db0, 7180000
thread '' panicked at 'Channel rejected frequency command: "SendError(..)"', src\lib.rs:104:10

I've gone back to the static so maybe not as clean but it's a working solution.

You should only use Box::from_raw in the close function. The box will be dropped when the variable goes out of scope which will deallocate it, obviously that's not what you want anywhere but the close function.

Generally you should just work with the raw pointer directly[1] everywhere except close. It will minimize the extra requirements placed on how the pointer is used.


  1. as opposed to trying to convert it to a reference or Box ↩︎

Thanks, didn't realise that.