How do I implement an external trait for an external struct?

One note, just to be on the same page. The lifecycle of that rusqlite::Connection struct object should be as follows:

  • It is created via rusqlite::Connection.open_* (there's several functions to open an sqlite3 db).
  • It is then passed back to Erlang/Elixir contained in rustler::resource::ResourceArc. It should not be GC'd at that boundary!
  • The Erlang/Elixir code should then be free to happily pass around this opaque resource to all sorts of Erlang/Elixir Rustler wrappers of rusqlite functions that query the DB, insert or update stuff etc. The container with the rusqlite::Connection resource in it should be kept alive during all of that as well (no GC or implicit destruction).
  • Eventually, Erlang/Elixir code will want the underlying Rust code to call conn.close() (a method of the rusqlite::Connection struct). At that point I want to explicitly destroy the connection. I am still reading on Rust's own Arc -- so I can understand and use rustler::resource::ResourceArc in an educated manner -- and would appreciate a quick example on how do you explicitly destroy the contents of such an Arc-wrapped resource.

So this is the lifecycle I am looking for. Is that what's going to happen with your proposed code?


/cc @kornel & @OvermindDL1, I hope they don't mind.

This sounds more or less like the life cycle. Regarding the close function, the main thing to notice is that you want to be able to close it before it is garbage collected. The way you can do this is to store an option inside the data, and when close is called, you simply replace the option with None.

2 Likes

I expect rusqlite to make sure the contents of the Connection struct become worthless after I call .close() on it. But I suppose you are talking about destroying / freeing the struct itself after that? Apologies, I haven't learned Arc yet and I am very likely coming across as clueless. :expressionless:

The .close() function takes the connection by self, so calling it will destroy the value completely. There won't be a value of type connection for you to leave in the Arc after calling close, however the Arc still exists, so you have to leave something which is why I suggest leaving an empty option instead.

1 Like

The database connection will be closed automatically when it is Dropped, and it will be dropped when Arc refcount goes to 0.

If you need to .close() manually, then there's the question: where are your remaining Arc copies hiding? They will end up holding a defunct connection. You should probably prevent them from existing past the useful lifetime of the connection object instead, and then you won't need to do anything other than drop your Arc (e.g. reclaim a pointer with Arc::from_raw).

Can't say I am yet grasping all of that but I am working on it.

One thing that's still a bit bewildering for me is: why do I need to wrap the rusqlite::Connection in a struct of my own if I am not going to serialise it but only pass around a pointer to it?

You don't need to wrap it, it's just in case you wanted extra fields or some such.

2 Likes

I ended up having to wrap the rusqlite::Connection because the compiler yelled at me that I can't implement traits for structs not in the current crate.

Some progress afterwards:

struct XqliteConnection {
    conn: rusqlite::Connection
}

fn open<'a>(env: Env<'a>, args: &[Term<'a>]) -> NifResult<Term<'a>> {
  // ...
  resource_struct_init!(XqliteConnection, env);
}

This blows up with:

error[E0277]: `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>` cannot be shared between threads safely
  --> src/lib.rs:51:13
   |
51 |             resource_struct_init!(XqliteConnection, env);
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>` cannot be shared between threads safely
   |
   = help: within `XqliteConnection`, the trait `std::marker::Sync` is not implemented for `std::cell::RefCell<rusqlite::inner_connection::InnerConnection>`
   = note: required because it appears within the type `rusqlite::Connection`
   = note: required because it appears within the type `XqliteConnection`
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

I'm lost (for now).


Maybe I should go out there and find usages of rusqlite; I can't be the only person who wants a lock-free read-only reference to the underlying sqlite3 DB being passed around while only locking the Arc when calling .close()!

The connection Send, so just put it in a mutex.

So the wrapping structure definition should be this?

struct XqliteConnection {
    conn: Mutex<rusqlite::Connection>
}
1 Like

This is getting a bit ridiculuous. :confused:

error[E0308]: mismatched types
  --> src/lib.rs:52:13
   |
52 |             resource_struct_init!(XqliteConnection, env);
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected enum `std::result::Result`, found `bool`
   |
   = note: expected enum `std::result::Result<rustler::Term<'a>, rustler::Error>`
              found type `bool`
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

Full code:

fn open<'a>(env: Env<'a>, args: &[Term<'a>]) -> NifResult<Term<'a>> {
    match rusqlite::Connection::open_in_memory() {
        Ok(conn) => {
            resource_struct_init!(XqliteConnection, env);
            let mutex = Mutex::new(conn);
            let xconn = XqliteConnection { mutex };
            let wrapper = ResourceArc::new(xconn);
            Ok((atoms::ok(), wrapper).encode(env))
        },
        Err(err) => {
            Ok((atoms::error(), format!("{}", err)).encode(env))
        },
    }
}

I should probably contact Rustler's author at this point. :frowning:

Try inserting the expanded version (copy from the source) and posting the error message then?

You reminded me to RTFM again:

fn on_init<'a>(env: Env<'a>, _load_info: Term<'a>) -> bool {
	resource_struct_init!(XqliteConnection, env);
	true
}

:point_up: This is something that Rustler is supposed to detect and run for me on initialisation (only once, not every time like in my previous function code).

Putting that function in the file and removing the previous invocation of the procedural macro fixed my previous problem.

Then the open function compiled (with a small change when constructing the wrapper struct; I used the anonymous field syntax and I had to use the named field syntax).

Then this is the close function:

fn close<'a>(env: Env<'a>, args: &[Term<'a>]) -> NifResult<Term<'a>> {
    match args[0].decode::<ResourceArc<XqliteConnection>>() {
        Ok(wrapper) => {
            let xconn: XqliteConnection = wrapper.unwrap();
            let conn: rusqlite::Connection = xconn.lock().unwrap();
            match conn.close() {
                Ok(()) => {
                    Ok(atoms::ok().encode(env))
                },
                Err((conn, err)) => {
                    Ok((atoms::error(), wrapper).encode(env))
                }
            }
        }
        Err(err) => {
            Ok((atoms::error(), err).encode(env))
        }
    }
}

Errors with:

error[E0599]: no method named `unwrap` found for type `rustler::ResourceArc<XqliteConnection>` in the current scope
  --> src/lib.rs:72:51
   |
72 |             let xconn: XqliteConnection = wrapper.unwrap();
   |                                                   ^^^^^^ method not found in `rustler::ResourceArc<XqliteConnection>`

error[E0599]: no method named `lock` found for type `XqliteConnection` in the current scope
  --> src/lib.rs:73:52
   |
29 | struct XqliteConnection {
   | ----------------------- method `lock` not found for this
...
73 |             let conn: rusqlite::Connection = xconn.lock().unwrap();
   |                                                    ^^^^ method not found in `XqliteConnection`

As said above, I still haven't learned the intricacies of working with Arc structs. Working on finding the proper API to pull data out of the wrapper => mutex => Arc.

I am aware the code is wrong and I am working on it. Just posting progress. :slight_smile:

You use the arc like it was the value inside, e.g. if you have an Arc<MyType> and MyType has a foo() method, you can call it on the Arc with my_arc.foo(). Note also that you will not be able to obtain a XqliteConnection inside the close() function — you'll just be able to get a &XqliteConnection.

Fair, but I am not sure rustler::resource::ResourceArc works like that:

struct XqliteConnection {
    conn: Mutex<rusqlite::Connection>
}

let wrapper = args[0].decode::<ResourceArc<XqliteConnection>>()?;
let xconn = wrapper.lock().unwrap();

Error:

error[E0599]: no method named `lock` found for type `rustler::ResourceArc<XqliteConnection>` in the current scope
  --> src/lib.rs:71:33
   |
71 |             let xconn = wrapper.lock().unwrap();
   |                                 ^^^^ method not found in `rustler::ResourceArc<XqliteConnection>`

This seems to imply that Rustler's ResourceArc doesn't transparently pass the methods down to the contained object? Am I misunderstanding it?

Have you added a lock() method to XqliteConnection? Try calling it on the mutex instead.

Oops, you're right (and I forgot to reference the field):

let xconn = wrapper.conn.lock().unwrap();

This compiles.

You can see that it transparently passes references through by noticing it has a Deref impl. Additionally since it does not have a DerefMut impl, you can't mutate through the Arc. (although the mutex let's you get around that)

Yes. Just read about that an hour ago and promptly forgot it. :man_facepalming:

Apparently now I have a std::sync::MutexGuard<'_, rusqlite::Connection> on my hands. How do I get the true native resource that I am after? (The rusqlite::Connection)

Through the Deref and DerefMut impls on MutexGuard. You can use the guard as if it was a Connection.