How do I implement an external trait for an external struct?

That's part of the issue though; I'd like the DB connection struct itself to live until I call close (at which point rusqlite destroys its associated native resources).

Is what you are suggesting only going to create / destroy containers carrying the connection struct between the language barriers, or is it going to create / destroy it itself as well? Wasn't sure from your answer.

You create the DB connection itself when you create a box, and when you turn it back into a box on close, dropping the box will also drop the DB connection.

Yeah, no good.

Could this be the answer, how do you think? rustler::resource::ResourceArc - Rust

1 Like

Yep, that looks good. You use the ResourceArc instead of the box I proposed. It looks like you'll need to use the resource_struct_init! macro to be allowed to pass the data struct to the ResourceArc, and the documentation seems rather sparse ... :sweat_smile:

That's my biggest issue so far. :frowning: The documentation is sparse and I have to rely on parsing the Rust code myself which I am still pretty bad at.

Last thing and I hope you don't mind. Could you post [pseudo-]code on how would you go about using that struct and its procedural macro in the case of the open and close functions above? I'd like to produce a working example for opening and closing an sqlite3 connection while passing the connection back to Elixir and then use Elixir again to call close with that serialised connection as a parameter.

EDIT: Even if you don't -- and you really don't have to -- I am very grateful for the discussion.

I have not tested this, but just based on the previous snippet with add:

#[macro_use]
extern crate rustler;

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler_atoms! {
        atom ok;
    }
}

rustler::rustler_export_nifs! {
    "Elixir.Xqlite.RusqliteNif", // Name of the Elixir module
    [
        ("new_db", 1, new_db),
        ("use_db", 1, use_db),
    ],
    None
}

struct RustqliteData {
    conn: rusqlite::Connection
}

fn into_rustler_err(err: rusqlite::Error) -> Error {
    ...
}

fn new_db<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let path: String = args[0].decode()?;
    
    let conn = rusqlite::Connection::open(&path).map_err(into_rustler_err)?;
    let data = RustqliteData {
        conn
    };
    
    resource_struct_init!(RustqliteData, env);
    
    let arc = ResourceArc::new(data);

    Ok((atoms::ok(), arc).encode(env))
}

fn use_db<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let data: ResourceArc<RustqliteData> = args[0].decode()?;

    // use data here

    Ok((atoms::ok(), 0).encode(env))
}

You do not need to explicitly provide drop code in this case — when the garbage collector collects the last instance, the connection will be automatically dropped. That said, you can replace the conn field with an Option<rusqlite::Connection> and replace it with None in close, if you wish to provide an explicit close method.

Additionally if you wish to modify anything inside the Data struct, you will need to wrap it in a Mutex, as Erlang may call it from multiple threads simultaneously.

3 Likes

Erlang will indeed call it from multiple threads but I'm pretty sure it won't modify it so I should be fine.

I can't say I understood everything that you said but you've given me a ton of stuff to think about and try in code.

Thanks a lot! I'll be trying things and will report back with findings and what worked in the end.

1 Like

I've just seen another guy use serde_rustler to deal with serialisation but he wasn't actually having my scenario (namely working with native resources). Still, it's one more venue I'll likely explore.

Your problem is that serializing the Connection object tries to turn a live database connection — with lots of internal state, pointers, locks, open file handles — into a flat bunch of dumb bytes that do nothing, and then Rust would probably close the actual connection for you (because the serialized copy doesn't keep ownership of the original). So you'd be left with a xeroxed copy of what used to be an internal state of a database connection, and no db connection.

What you need to do is to keep the Connection object on the Rust side and never send it to the Erlang side. The only thing Erlang needs to know is which connection object it's talking about, and the "which" question can be answered with a raw pointer value, which you can get from a Box or Arc or similar. It'll be just a 64-bit number, which is trivial to serialize and won't destroy the connection. You could also have a global array of Rust connections and send an integer to Erlang that is an index in that array (but that's just an example, in practice sending pointers is easier to manage).

3 Likes

Yes, this is what you want for the BEAM VM to hold onto arbitrary data and call a destructor on it when the BEAM VM is done with. On the BEAM VM a resource is an opaque value, only useful to be passed back through NIF's to perform work on them.

What @alice put in their recent post with the sinppet looks correct on an initial look.

The BEAM VM can and will call things from many threads as it's task style actor system distributes workers with impunity. But no, it knows nothing about resources beyond "It's a pointer to data and I should call this callback function on it, passing it in along with my own environment, when I am done with it (GC'd)".

One note, just to be on the same page. The lifecycle of that rusqlite::Connection struct object should be as follows:

  • It is created via rusqlite::Connection.open_* (there's several functions to open an sqlite3 db).
  • It is then passed back to Erlang/Elixir contained in rustler::resource::ResourceArc. It should not be GC'd at that boundary!
  • The Erlang/Elixir code should then be free to happily pass around this opaque resource to all sorts of Erlang/Elixir Rustler wrappers of rusqlite functions that query the DB, insert or update stuff etc. The container with the rusqlite::Connection resource in it should be kept alive during all of that as well (no GC or implicit destruction).
  • Eventually, Erlang/Elixir code will want the underlying Rust code to call conn.close() (a method of the rusqlite::Connection struct). At that point I want to explicitly destroy the connection. I am still reading on Rust's own Arc -- so I can understand and use rustler::resource::ResourceArc in an educated manner -- and would appreciate a quick example on how do you explicitly destroy the contents of such an Arc-wrapped resource.

So this is the lifecycle I am looking for. Is that what's going to happen with your proposed code?


/cc @kornel & @OvermindDL1, I hope they don't mind.

This sounds more or less like the life cycle. Regarding the close function, the main thing to notice is that you want to be able to close it before it is garbage collected. The way you can do this is to store an option inside the data, and when close is called, you simply replace the option with None.

2 Likes

I expect rusqlite to make sure the contents of the Connection struct become worthless after I call .close() on it. But I suppose you are talking about destroying / freeing the struct itself after that? Apologies, I haven't learned Arc yet and I am very likely coming across as clueless. :expressionless:

The .close() function takes the connection by self, so calling it will destroy the value completely. There won't be a value of type connection for you to leave in the Arc after calling close, however the Arc still exists, so you have to leave something which is why I suggest leaving an empty option instead.

1 Like

The database connection will be closed automatically when it is Dropped, and it will be dropped when Arc refcount goes to 0.

If you need to .close() manually, then there's the question: where are your remaining Arc copies hiding? They will end up holding a defunct connection. You should probably prevent them from existing past the useful lifetime of the connection object instead, and then you won't need to do anything other than drop your Arc (e.g. reclaim a pointer with Arc::from_raw).

Can't say I am yet grasping all of that but I am working on it.

One thing that's still a bit bewildering for me is: why do I need to wrap the rusqlite::Connection in a struct of my own if I am not going to serialise it but only pass around a pointer to it?

You don't need to wrap it, it's just in case you wanted extra fields or some such.

2 Likes

I ended up having to wrap the rusqlite::Connection because the compiler yelled at me that I can't implement traits for structs not in the current crate.

Some progress afterwards:

struct XqliteConnection {
    conn: rusqlite::Connection
}

fn open<'a>(env: Env<'a>, args: &[Term<'a>]) -> NifResult<Term<'a>> {
  // ...
  resource_struct_init!(XqliteConnection, env);
}

This blows up with:

error[E0277]: `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>` cannot be shared between threads safely
  --> src/lib.rs:51:13
   |
51 |             resource_struct_init!(XqliteConnection, env);
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>` cannot be shared between threads safely
   |
   = help: within `XqliteConnection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>`
   = note: required because it appears within the type `rusqlite::Connection`
   = note: required because it appears within the type `XqliteConnection`
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

I'm lost (for now).


Maybe I should go out there and find usages of rusqlite; I can't be the only person who wants a lock-free read-only reference to the underlying sqlite3 DB being passed around while only locking the Arc when calling .close()!

The connection Send, so just put it in a mutex.

So the wrapping structure definition should be this?

struct XqliteConnection {
    conn: Mutex<rusqlite::Connection>
}
1 Like