Lifetime issues with `inventory` and boxed future

I'm trying to build an async dynamic registration system using the inventory crate.

The sync code is easy, and works fine; however, once async is becoming involved, and I need the registration to run in tokio, I'm having troubles with the lifetime of a reference passed into the registration function.

It's easier to illustrate with a code example:

I've got a collector, which collects (in this example), some string flags:

#[derive(Debug, Default)]
struct Collector {
    flags: RwLock<Vec<String>>,
}

impl Collector {
    pub async fn add(&self, flag: &str) {
        let mut lock = self.flags.write().await;
        lock.push(flag.to_owned());
    }
}

Using inventory, the user can submit functions, which can call Collector::add, to add their own flags:

type CollectFn<'a> = fn(&'a Collector)
    -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>;

struct CollectFunc<'a>(CollectFn<'a>);

inventory::collect!(CollectFunc<'static>); // notice the 'static lifetime

The user will write a function that adheres to the CollectFn signature and submits it:

struct FooFlag {}

impl<'a> FooFlag {
    pub(crate) fn ctor(collector: &'a Collector) -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>
    where
        Self: Sync + 'a,
    {
        Box::pin(async move {
            collector.add("Foo").await;
        })
    }
}

inventory::submit!(CollectFunc(FooFlag::ctor));

The signature of the ctor function is inspired by how the async-trait creates boxed futures.

However, due to inventory::collect! requiring me to specify a 'static lifetime, it means that I require a Collector with a static lifetime.

This code fails to compile:

#[tokio::main]
async fn main() {
    let collector = Collector::default();

    for func in inventory::iter::<CollectFunc<'_>>() {
        (func.0)(collector_ref).await;
    }

    println!("flags: {:?}", collector.flags);
}

Of course, for this particular case, this is incorrect. Collector needs to have a lifetime that's longer than the iteration loop. This is due to not storing the futures anywhere or not dropping collector, of course. Once we do this (and we can, with safe Rust), things start to break.

I can circumvent this with std::mem::transmute; but this is ... well ... unsafe and could spawn nasal daemons.

This code compiles, works, and even runs perfectly fine under miri:

#[tokio::main]
async fn main() {
    let collector = Collector::default();

    let collector_ref =
        unsafe { std::mem::transmute::<&Collector, &'static Collector>(&collector) };

    for func in inventory::iter::<CollectFunc<'_>>() {
        (func.0)(collector_ref).await;
    }

    println!("flags: {:?}", collector.flags);
}

Is there any way for me to avoid this unsafe transmute?

Thank you!

Full Code Listing (minimal reproducible example)
# Cargo.toml
[package]
name = "reproduce"
version = "0.1.0"
edition = "2021"

[dependencies]
inventory = "0.3.15"
tokio = { version = "1.41.0", features = ["full"] }
// main.rs
use std::future::Future;
use std::pin::Pin;

use tokio::sync::RwLock;

#[derive(Debug, Default)]
struct Collector {
    flags: RwLock<Vec<String>>,
}

impl Collector {
    pub async fn add(&self, flag: &str) {
        let mut lock = self.flags.write().await;
        lock.push(flag.to_owned());
    }
}
type CollectFn<'a> = fn(&'a Collector)
    -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>;
struct CollectFunc<'a>(CollectFn<'a>);

inventory::collect!(CollectFunc<'static>);

struct FooFlag {}

impl<'a> FooFlag {
    pub(crate) fn ctor(collector: &'a Collector) -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>
    where
        Self: Sync + 'a,
    {
        Box::pin(async move {
            collector.add("Foo").await;
        })
    }
}

inventory::submit!(CollectFunc(FooFlag::ctor));

#[tokio::main]
async fn main() {
    let collector = Collector::default();

    let collector_ref =
        unsafe { std::mem::transmute::<&Collector, &'static Collector>(&collector) };

    for func in inventory::iter::<CollectFunc<'_>>() {
        (func.0)(collector_ref).await;
    }

    println!("flags: {:?}", collector.flags);
}
1 Like

Does this work?

type CollectFn = fn(&Collector)
    -> Pin<Box<dyn Future<Output = ()> + Send>>;

struct CollectFunc(CollectFn);

inventory::collect!(CollectFunc);
struct FooFlag {}

impl FooFlag {
    pub(crate) fn ctor(collector: &Collector) -> Pin<Box<dyn Future<Output = ()> + Send>>
    where
        Self: Sync + 'static,
    {
        Box::pin(async move {
            collector.add("Foo").await;
        })
    }
}

inventory::submit!(CollectFunc(FooFlag::ctor));
#[tokio::main]
async fn main() {
    let collector = Collector::default();

    for func in inventory::iter::<CollectFunc>() {
        (func.0)(collector_ref).await;
    }

    println!("flags: {:?}", collector.flags);
}

This doesn't work because the ctor function will throw a compile-time error that the reference passed in needs to outlive 'static, which I guess is due to Pin.

impl<'a> FooFlag {
    pub(crate) fn ctor(collector: &Collector) -> Pin<Box<dyn Future<Output = ()> + Send>>
    where
        Self: Sync + 'static,
    {
        Box::pin(async move {
            collector.add("Foo").await;
        })
    }
}

results in

error: lifetime may not live long enough
  --> src/main.rs:32:9
   |
28 |       pub(crate) fn ctor(collector: &Collector) -> Pin<Box<dyn Future<Output = ()> + Send>>
   |                                     - let's call the lifetime of this reference `'1`
...
32 | /         Box::pin(async move {
33 | |             collector.add("Foo").await;
34 | |         })
   | |__________^ returning this value requires that `'1` must outlive `'static`
   |
help: to declare that the trait object captures data from argument `collector`, you can add an explicit `'_` lifetime bound
   |
28 |     pub(crate) fn ctor(collector: &Collector) -> Pin<Box<dyn Future<Output = ()> + Send + '_>>
   |                                                                                         ++++

error: could not compile `reproduce` (bin "reproduce") due to 1 previous error

Oh, right; I didn't think things all the way through. You should be able to get something to work with an explicit HRTB, something like this (though I'm less sure of the exact bounds/syntax necessary):

type CollectFn = for<'a> fn(&'a Collector)
    -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>;

struct CollectFunc(CollectFn);

inventory::collect!(CollectFunc);
struct FooFlag {}

impl<'a> FooFlag {
    pub(crate) fn ctor(collector: &'a Collector) -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>
    where
        Self: Sync + 'static,
    {
        Box::pin(async move {
            collector.add("Foo").await;
        })
    }
}

inventory::submit!(CollectFunc(FooFlag::ctor));
#[tokio::main]
async fn main() {
    let collector = Collector::default();

    for func in inventory::iter::<CollectFunc>() {
        (func.0)(collector_ref).await;
    }

    println!("flags: {:?}", collector.flags);
}

I had a similar idea; however, the type defined with a HRTB is a different type than the actual ctor function, resulting in the following error:

error[E0308]: mismatched types
  --> src/main.rs:35:32
   |
35 | inventory::submit!(CollectFunc(FooFlag::ctor));
   |                    ----------- ^^^^^^^^^^^^^ one type is more general than the other
   |                    |
   |                    arguments to this struct are incorrect
   |
   = note: expected fn pointer `for<'a> fn(&'a Collector) -> Pin<Box<(dyn Future<Output = ()> + Send + 'a)>>`
                 found fn item `fn(&Collector) -> Pin<Box<dyn Future<Output = ()> + Send>> {FooFlag::ctor}`

It looks like you need to move <'a> from the impl block to the function itself for some reason:

pub(crate) fn ctor<'a>(collector: &'a Collector) -> Pin<Box<dyn Future<Output = ()> + Send + 'a>>

-- or --

pub(crate) fn ctor(collector: &Collector) -> Pin<Box<dyn Future<Output = ()> + Send + '_>>
1 Like

Ohhh! The 'static on the Self bound was the key to it.

Wow. In hindsight, this makes sense, but I couldn't figure it out.

Thank you! :bow:

1 Like