Hello, I'm a new Rust user (with some C++ experience) who has been working through the scanning chapter in crafting interpreters and I've been struggling with understanding/managing the lifetimes with this code I have. Conceptually, I'm trying to have my Scanner take in a source code string (as a Vec for simplicity) and scan it into a list of tokens. Each token contains a lexeme, a slice pointing within the source vec (since the token doesn't need to own the source).
As a result, the Token has a lifetime, which I named 'source, to match that each token needs to live as long as the source code. In one of my methods in my scanner, scan_to_token, though, I'm trying to create a token with a lifetime of 'source but getting an error. A simplified version of the code is below (playground link):
#[derive(Debug)]
pub enum TokenType {
LeftParen,
RightParen,
Eof,
// rest ommitted for simplicity
}
#[derive(Debug)]
pub struct Token<'source> {
pub token_type: TokenType,
pub lexeme: &'source [u8],
pub line: usize,
}
pub struct Scanner<'source> {
source: Vec<u8>,
tokens: Vec<Token<'source>>,
start: usize,
current: usize,
line: usize,
}
impl<'source> Scanner<'source> {
pub fn new(source: &'source str) -> Self {
Self {
source: source.as_bytes().to_owned(),
tokens: Vec::new(),
start: 0,
current: 0,
line: 1,
}
}
pub fn scan_tokens(&mut self) {
while !self.is_at_end() {
self.start = self.current;
let next_token = self.scan_to_token();
self.tokens.push(next_token);
}
self.tokens.push(Token {
token_type: TokenType::Eof,
lexeme: b"",
line: self.line,
})
}
fn scan_to_token(&mut self) -> Token<'source> {
let next_character = self.advance();
match next_character {
b'(' => self.create_token_object(TokenType::LeftParen),
b')' => self.create_token_object(TokenType::RightParen),
// rest ommitted for simplicity
_ => panic!("Not handled for now"),
}
}
fn is_at_end(&self) -> bool {
self.current >= self.source.len()
}
fn advance(&mut self) -> u8 {
let result = *self
.source
.get(self.current)
.expect("out of range scanner index (this should never happen)");
self.current += 1;
result
}
fn create_token_object(&self, token_type: TokenType) -> Token {
let text = &self.source[self.start..self.current];
Token {
token_type,
lexeme: text,
line: self.line,
}
}
}
The error is as follows:
Compiling playground v0.0.1 (/playground)
error: lifetime may not live long enough
--> src/lib.rs:55:21
|
24 | impl<'source> Scanner<'source> {
| ------- lifetime `'source` defined here
...
50 | fn scan_to_token(&mut self) -> Token<'source> {
| - let's call the lifetime of this reference `'1`
...
55 | b')' => self.create_token_object(TokenType::RightParen),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ method was supposed to return data with lifetime `'source` but it is returning data with lifetime `'1`
What can I do to resolve this? Or better yet, model this in a more idiomatic way? Marking &mut self in the erroring method as &'source mut self creates issues with the loop in the scan_tokens method, so I assume that isn't the right way to go about it.
Chapter 4 sent me down a rabbit hole trying to fix the positions of some of the