Lifetime of callback arguments

#1

I have this function which tries to invoke a callback at some point. Unfortunately I’m totally unable to get the life times right here. I have tried to add explicit lifetime annotations, but from the error message I take that the compiler believes lifetime 'b will out live lifetime 'a. My Intention though is that 'b lives until the end of each match arm and 'a should live until the end of the function.

This all started because without the lifetimes, the compiler told me that test would go out of scope at the end of the match arm and there for would not live long enough for &test to be possible. This confuses me, because I though &HookType can only exist until callback returns. The compiler thinks otherwise though.

I would be glad if someone could explain to me what I’m doing wrong and how it’s done properly.

pub fn travel_ast<'a, 'b, T: PiggybackCapable + 'b, Func: FnMut(&'b HookType<'_, 'b, T>)>(ast: Ast::StatementList<'a>, mut callback: Func) {
    for statement in ast {
        match statement.item {
            Ast::Statement::Expression(expression) => {
                travel_expression(expression, &mut callback);
            },

            Ast::Statement::If(if_statement) => {
                let test = travel_expression(if_statement.test, &mut callback);

                callback(&HookType::ConsequentBody { test: &test, consequent: if_statement.consequent });

                if let Some(alternate) = if_statement.alternate {
                    callback(&HookType::AlternateBody { test: &test, alternate });
                }

                callback(&HookType::AfterIf { test: &test });
            }

            _ => (),
        }
    }
}


error[E0495]: cannot infer an appropriate lifetime for lifetime parameter `'ast` due to conflicting requirements
   --> src/traveler.rs:173:13
    |
173 |             Ast::Statement::If(if_statement) => {
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: first, the lifetime cannot outlive the lifetime 'a as defined on the function body at 166:19...
   --> src/traveler.rs:166:19
    |
166 | pub fn travel_ast<'a, 'b, T: PiggybackCapable + 'b, Func: FnMut(&'b HookType<'_, 'b, T>)>(ast: Ast::StatementList<'a>, mut callback: Func) {
    |                   ^^
    = note: ...so that the types are compatible:
            expected ratel::ast::Statement<'a>
               found ratel::ast::Statement<'_>
note: but, the lifetime must be valid for the lifetime 'b as defined on the function body at 166:23...
   --> src/traveler.rs:166:23
    |
166 | pub fn travel_ast<'a, 'b, T: PiggybackCapable + 'b, Func: FnMut(&'b HookType<'_, 'b, T>)>(ast: Ast::StatementList<'a>, mut callback: Func) {
    |                       ^^
    = note: ...so that the expression is assignable:
            expected &'b traveler::HookType<'_, 'b, T>
               found &'b traveler::HookType<'_, '_, T>
#2
Func: FnMut(&'b HookType<'_, 'b, T>)>

You don’t want that 'b in the borrow of HookType. Instead, you want an HRTB bound, which is:

Func: for<'c> FnMut(&'c HookType<'_, 'b, T>)>

which is typically just written as:

Func: FnMut(&HookType<'_, 'b, T>)>
1 Like
#3

oh wow well yes this undid everything I tried to fix my initial issue :smiley:

    Ast::Expression::Arrow(arrow_function) => {
        let ident = Backpack::new(expression);
        let arena = Arena::new();
        let mut statement_owner = Owner::new();
        let mut block_owner = Owner::new();
        let Ast::expression::ArrowExpression { params, body, .. } = arrow_function;

        let body = match body {
            Ast::expression::ArrowBody::Block(block) => block,
            Ast::expression::ArrowBody::Expression(expression) => {
                let return_statement = Ast::Statement::Return(Ast::statement::ReturnStatement {
                    value: Some(expression)
                });
                let return_statement_loc = Ast::Loc::new(expression.start, expression.end, return_statement);
                let return_statement_loc = statement_owner.own(return_statement_loc);
                let return_statement_node = Ast::Node::new(return_statement_loc);

                let block_statement = Ast::Block {
                    body: Ast::NodeList::from(&arena, return_statement_node)
                };

                let block_statement_loc = Ast::Loc::new(expression.start, expression.end, block_statement);
                let block_statement_loc = block_owner.own(block_statement_loc);

                Ast::Node::new(block_statement_loc)
            }
        };

        callback(&HookType::Function { ident: &ident, params, body, arrow: true });

        ident
    }, 

Because here I have the problem that the compiler tells me that statement_owner, &arena and block_owner all only life inside the outer match arm (so the end of my provided snippet) which I think should be fine, but rust doesn’t agree. I came up with the Owner struct so the variables lifetime could be extended to the outer block, but I don’t know why their lifetime is still too short. I’m having a hard time with these lifetimes.

Edit: this is why I started to mess around with the lifetime of callback.

This is the Owner struct:

pub struct Owner<T> {
    proteges: Vec<T>,
}

impl<T> Owner<T> {
    pub fn own(&mut self, value: T) -> &T {
        self.proteges.push(value);

        self.proteges.last().unwrap()
    }

    pub fn own_mut(&mut self, value: T) -> &mut T {
        self.proteges.push(value);

        self.proteges.last_mut().unwrap()
    }

    pub fn new() -> Self {
        Owner { proteges: vec!() }
    }
}
#4

Can you put together a minimal repro on the Rust playground?

1 Like
#5

I have tried to create a reproduction but I’m not able to do so without the Ast of the ECMAScript parser library I’m using. It appears that the lifetime issues stem from the AST structs but I can’t tell why. I’m using ratel if thats any help.
Like I said, the compiler says &arena doesn’t live long enough, but I can not see why the reference to arena is needed longer than the outer block.

Edit:
When I remove the line

Ast::Node::new(&block_statement_loc)

from the second inner match arm and let both match arms return (), the compiler is happy. So I’m pretty sure that Ast::Node is causing the trouble. Problem is, the definition of Node, looks like this and I can’t see why that would cause an issue.

#[derive(Clone, Copy)]
pub struct Node<'ast, T: 'ast> {
    inner: CopyCell<&'ast Loc<T>>
}

impl<'ast, T: 'ast> Node<'ast, T> {
    #[inline]
    pub fn new(ptr: &'ast Loc<T>) -> Self {
        Node {
            inner: CopyCell::new(ptr)
        }
    }

    #[inline]
    pub fn set(&self, ptr: &'ast Loc<T>) {
        self.inner.set(ptr)
    }

    #[inline]
    pub fn get_mut(&mut self) -> &mut &'ast Loc<T> {
        self.inner.get_mut()
    }
}

Edit:
After fumbling around a bit more, I noticed that the Node struct is always causing this. Every reference I pass to a new node doesn’t live long enough, even though I think they both should go out of scope at the end of the same block.

#6

What does HookType and its Function variant look like? It’s possible that the 'b in Func: FnMut(&HookType<'_, 'b, T>)> is also incorrect - it needs to be an HRTB bound as well. Keep in mind that 'b in the travel_ast() is caller specified - this means you won’t be able to borrow any local values for that lifetime (this is where HRTB typically helps).

1 Like
#8

Hook type looks like this:

pub enum HookType<'own, 're, T: PiggybackCapable + 'own> {
    Function {
        ident: &'re Backpack<'own, T>,
        params: Ast::PatternList<'own>,
        body: &'re Ast::Node<'own, Ast::BlockStatement<'own>>,
        arrow: bool,
    },
}

but has a few more variants which all look very similar. My current code looks like this though:

Ast::Expression::Arrow(arrow_function) => {
    let ident = Backpack::new(expression);
    let arena = Arena::new();
    let Ast::expression::ArrowExpression { params, body, .. } = arrow_function;

    let body = match body {
        Ast::expression::ArrowBody::Block(block) => block,
        Ast::expression::ArrowBody::Expression(expression) => {
            let return_statement = Ast::Statement::Return(Ast::statement::ReturnStatement {
                value: Some(expression)
            });
            let return_statement_loc = Ast::Loc::new(expression.start, expression.end, return_statement);
            let return_statement_loc = &*arena.alloc(return_statement_loc);
            let return_statement_node = Ast::Node::new(&return_statement_loc);

            let block_statement = Ast::Block {
                body: Ast::NodeList::from(&arena, return_statement_node)
            };

            let block_statement_loc = Ast::Loc::new(expression.start, expression.end, block_statement);
            let block_statement_loc = arena.alloc(block_statement_loc);

            Ast::Node::new(&block_statement_loc)
        },

    };

//  callback(&HookType::Function { ident: &ident, params, body: &body, arrow: true });

    ident
},

I think I have figured out what is going on. fn travel_ast doesn’t have any errors anymore, but travel_expression does still have erros in the snipped pasted above. From what I can see rust thinks that the newly created node exceeds the lifetime of the function.

The signature of travel_expression is the following:

fn travel_expression<'local_ast, 'b, T: PiggybackCapable + 'local_ast>(expression: Ast::ExpressionNode<'local_ast>, callback: &mut impl FnMut(&HookType<T>)) -> Backpack<'local_ast, T> {

rust tells me the type of the second block_statement_loc is:

&'local_ast ratel::ast::Loc<ratel::ast::Block<'local_ast, ratel::ast::Statement<'local_ast>>> 

and the type of the new node is

ratel::ast::Node<'local_ast, ratel::ast::Block<'local_ast, ratel::ast::Statement<'local_ast>>>

So I think since 'local_ast is supposed to also be the lifetime of the return value, rust thinks the node will outlive the function body as well, which is not the case (or shouldn’t be). What can I do to prevent this?

#9

Is there a github repo that demonstrates the issue? It’s a bit hard to help seeing just bits and pieces of the picture.

The way the new node is constructed starts with:

Ast::expression::ArrowBody::Expression(expression) => {
            let return_statement = Ast::Statement::Return(Ast::statement::ReturnStatement {
                value: Some(expression)
            });

return_statement presumely carries lifetimes coming from expression, which looks like it comes all the way from the input to the function. This is likely where 'local_ast stems from. However, this needn’t be a problem on its own though.

I’m also unclear on what you’re trying to do in travel_expression - why do you have a local Arena there that you allocate from? What are you trying to accomplish there?

I also just noticed that Node contains a CopyCell, which in turn makes the T it holds invariant - this makes dealing with lifetimes a bit more complex because you need to be a lot more precise about them. This may not be a contributor here, but does add to the list of things to take into account.

Finally, if you don’t have a github repo with a reproducer, please paste the involved functions in their entirety and also include the rustc errors.

#10

First of all, thanks for bearing with me :smile: and trying to help.

I pushed everything to this repo now: https://github.com/TitanNano/rusty/blob/master/src/traveler.rs

The issue is that ECMAScript arrow expressions can have two types of bodies and I’m trying to convert the Ast::expression::ArrowBody::Expression into a BlockNode so I can pass them as the same HookType. The entire point of the traveler.rs file is to make the traversal of the ECMAScript AST a bit easier for other parts of my application where I’m often only interested in some parts of the Ast.

#11

Thanks - this repo is helpful :slight_smile:.

So, it’s a non-trivial amount of code at play, but what I think happens here is what we alluded to earlier: Node<'ast, T> is invariant in T due to containing a CopyCell:

#[derive(Clone, Copy)]
pub struct Node<'ast, T: 'ast> {
    inner: CopyCell<&'ast Loc<T>>
}

#[derive(PartialEq, Eq)]
pub struct CopyCell<T> {
    /// Internal value
    value: T,

    /// We trick the compiler to think that `CopyCell` contains a raw pointer,
    /// this way we make sure the `Sync` marker is not implemented and `CopyCell`
    /// cannot be shared across threads!
    _no_sync: PhantomData<*mut T>
}

One of the errors is loc not living long enough here:

Ast::Expression::TaggedTemplate(template) => {
            let ident = Backpack::new(expression);
            let function = travel_expression(template.tag, callback);
            let template_expression = Ast::Expression::Template(**template.quasi);
            let loc = Ast::Loc::new(template.quasi.start, template.quasi.end, template_expression);
            let arguments = vec!(travel_expression(Ast::ExpressionNode::new(&loc), callback));

            callback(&HookType::FunctionCall { ident: &ident, function, arguments });

            ident
        },
error[E0597]: `loc` does not live long enough
   --> src\traveler.rs:344:78
    |
344 |             let arguments = vec!(travel_expression(Ast::ExpressionNode::new(&loc), callback));
    |                                                                              ^^^ borrowed value does not live long enough
...
349 |         },
    |         - borrowed value only lives until here
    |
note: borrowed value must be valid for the lifetime 'local_ast as defined on the function body at 190:22...
   --> src\traveler.rs:190:22
    |
190 | fn travel_expression<'local_ast, 'b, T: PiggybackCapable>(expression: Ast::ExpressionNode<'local_ast>, callback: &mut impl FnMut(&HookType<T>)) -> Backpack<'local_ast, T> {
    |                      ^^^^^^^^^^

ExpressionNode::new(&loc) is creating a Node<'local_region, Expression<'local_ast>>, where local_region is a name I’m picking for the small scope during which loc lives (that match arm). But due to the invariance of Node, the compiler really wants to see 'local_region: 'local_ast, which of course cannot happen. This is why the compiler is complaining that loc needs to live for 'local_ast.

That’s my guess as to what’s occurring. I’m not sure how to easily fix this. It’s unclear whether CopyCell really needs to be invariant, or whether it simply picked the “wrong” way to make itself !Sync (e.g. using *const T would’ve accomplished the !Sync part but kept the covariance around). If I have time, I’ll think a bit more about this but this doesn’t seem anything close to being trivial (at least in so far as I can tell) :frowning:.

2 Likes
#12

Oh well, I see. So my best bet would be to get in touch with the library creator? Let me know if you get any ideas. :grimacing: