Help with Elixir NIF for table data structure

help
#1

Hello rustaceans. Would somebody be kind enough to help me with a NIF problem? I'm using the rustler library to write a NIF for elixir, and I'm getting stuck. The purpose of my library is to implement a table data structure for time series analysis. Each column works like a circular queue. In elixir we define our table dimensions and column labels to instantiate a table, and in rust the table is defined as:

struct SlidingWindow<'a> {
    data: Mutex<HashMap<&'a str, Vec<f32>>>,
    insertion_index: usize,
    length: usize,
    width: usize,
    labels: &'a [&'a str],
}

I've tried passing this to the rustler::resource! macro so I can generate references to my table instances for elixir, but this doesn't compile and I suspect this isn't the right way to do what I want (I think this comment describes my situation).

fn load(env: Env, _info: Term) {
    rustler::resource!(SlidingWindow, env);
}

It seems that rustler::resource! is incompatible with structs that require a named lifetime. The macro code is here. When we compile, rust says:

error[E0726]: implicit elided lifetime not allowed here
   --> src/lib.rs:177:24
    |
177 |     rustler::resource!(SlidingWindow, env);
    |                        ^^^^^^^^^^^^^- help: indicate the anonymous lifetime: `<'_>`

...and if we try the suggestion...

error[E0478]: lifetime bound not satisfied
   --> src/lib.rs:177:5
    |
177 |     rustler::resource!(SlidingWindow<'_>, env);
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: lifetime parameter instantiated with the lifetime `'_` as defined on the impl at 177:38
   --> src/lib.rs:177:38
    |
177 |     rustler::resource!(SlidingWindow<'_>, env);
    |                                      ^^
    = note: but lifetime parameter must outlive the static lifetime
    = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

I get the feeling that the solution will ultimately be simple, as I'm aiming to give elixir only a reference to my data, not serialize the data into elixir terms, but I don't know how to do this. I think I need to somehow make an ErlNifResourceType (see erlang.org/doc/man/erl_nif.html), and implement ResourceTypeProvider and maybe Encoder.

#2

You can't safely store temporary references in a static. I'm not familiar with NIFs or rustler at all, but this doc comment:

//! A NIF resource allows you to safely store Rust structs in a term, and therefore keep it across
//! NIF calls. The struct will be automatically dropped when the BEAM GC decides that there are no
//! more references to the resource.

implies there's no way to soundly give Elixir temporary references to your data with rustler::resource "manually" either. Indeed, looking a little further, we see:

pub trait ResourceTypeProvider: Sized + Send + Sync + 'static

So if there's a way to pass temporary-holding data structures to Elixir, this isn't it. You'll have to clone data (or leak it) in some sense to make it 'static.

1 Like
#4

Okay. Another approach would be to instantiate tables in a global Vec and return the index number to elixir, so it has an id it can use to access each respective table. But I still can't figure out how to satisfy rust lifetimes for this to work:

lazy_static! {
    static ref TABLES: Mutex<Vec<SlidingWindow< LIFETIME_NEEDED >>> = {
        Mutex::new(Vec::new())
    };
}
struct SlidingWindow<'a> {
    data: Mutex<HashMap<&'a str, Vec<Option<f32>>>>,
    insertion_index: usize,
    length: usize,
    width: usize,
    labels: &'a [&'a str],
}

If I try this approach, rust says I need a named lifetime where you see LIFETIME_NEEDED, what can I put here? If I use 'static, then the values in the labels field don't live long enough:

error[E0597]: `string` does not live long enough
  --> src/lib.rs:39:26
   |
39 |                 let s = &string[..];
   |                          ^^^^^^ borrowed value does not live long enough
...
45 |                 map.insert(s, list);
   |                 ------------------- argument requires that `string` is borrowed for `'static`
46 |             }
   |             - `string` dropped here while still borrowed

error[E0597]: `columns` does not live long enough
  --> src/lib.rs:57:18
   |
52 | /     t.push(SlidingWindow {
53 | |         data: Mutex::new(map),
54 | |         insertion_index: 0,
55 | |         width,
56 | |         length,
57 | |         labels: &columns[..],
   | |                  ^^^^^^^ borrowed value does not live long enough
58 | |     });
   | |______- argument requires that `columns` is borrowed for `'static`
...
61 |   }
   |   - `columns` dropped here while still borrowed
#5

'static is the only lifetime that can go in a static. There is no solution where you put temporary references in a static (as you cannot do so).

It may help to highlight that lifetime constraints are checked at compile time in order to uphold Rust's safety guarantees. They aren't part of the run-time, and the compiler has to prove that they hold. They're also a local analysis, and not something that values carry around with them. Longer lifetimes can be coerced into shorter lifetimes, but they are not dynamic.

You need a way to pass temporaries to Elixir that doesn't involve static, or you need to pass cloned data to Elixir instead of temporaries, or you need to pass raw pointers and implement some sort of run-time system that ensures you don't use-after-free (etc.), or some other alternative.

2 Likes
#6

Thanks, this makes sense now. I was using the wrong data types. I changed my struct to remove the need for lifetime parameters and now everything builds.

struct SlidingWindow {
    data: Mutex<HashMap<String, Vec<Option<f32>>>>,
    insertion_index: usize,
    length: usize,
    width: usize,
    labels: Vec<String>,
}