Subtleties of borrow checking

[Arne]

I was encountering this issue where I tried to return a value where it's members have the same lifetime as some other input. When I hide the creation of the slice behind a member function the borrow checking fail. But when I replicate the functionality at the callsite it works. Why is that? And how can I have the functionality that I wanted?

Playground Link

Edit: pub fn as_slice(&self) -> &'a str
seems to work and now I am even more confused

On another not: I am still very confused about the diferent use of "+ 'a" (with traits) as well as "where SomeType: 'a" (with generics). Similar confusion exists with impl, dyn and with + generics (especially in the context of Fn traits where functions return functions? And where nesting impl does not seem to work.) If that could be cleared up a bit it would be greatly appreciated.

[RustyYato]

You can change as_slice to

pub fn as_slice(&self) -> &'a str {
    &self.tokens[self.position..]
}

to make it work playground

The reason for this is that &self.token[self.position..] desugars to &(*self.token.index(self.position..)). index comes from Index

pub trait Index<Idx>  {
    type Output: ?Sized;
    
    fn index(&self, index: Idx) -> &Self::Output;
}

for str you get something like below from the Index trait.

fn index(&self, index: Range<usize>) -> &str { ... }

If we track the lifetimes, we see that the input string (self) and the output string have the same lifetime.

Now, back to the problem.

fn as_slice(&self) -> &'a str {
    &self.tokens[self.position..]
}

desugars to below, as I showed before

fn as_slice(&self) -> &'a str {
    &(*self.tokens.index(self.position..))
}

Now, reborrowing a shared reference (the combo &* is called reborrowing) will keep the same lifetime as the input lifetime. This is because shared refereces can be copied around. So, you get the lifetime 'a for the output.

The issue before when you had,

fn as_slice(&'a self) -> &'a str {
    &self.tokens[self.position..]
}

was that you were declaring the lifetime of the output to be the same as the lifetime of the input. This is too restrictive in this case, so it didn't work with the closure you had.

---

One way to think of you type TokenStream, is that it is a view into a string stored somewhere else. So when you call as_slice you are producing a new view into the same string, but that view doesn't need to be tied to the TokenStream (since the TokenStream doesn't own the original string), so you can tie it to the lifetime of that original string (which is 'a in TokenStream<'a>).

---

It is bad style to reuse 'a everywhere, it makes code harder to read and harder to write, so if you see yourself shadowing lifetime names, please rename some lifetimes to get rid of the shadowing.

[DevinR528]

Sometimes this sort of thing helps me think about the issue

impl<'long> TokenStream<'long>
{
    // the borrow of self does not last the entire time but
    // the return value can
    pub fn as_slice<'short>(&'short self) -> &'long str
    {
        &self.tokens[self.position..]
    }
}

playground