Is it possible to parse the file line by line without doing an allocation per line

let f = File::open("input.txt")?;
let mut buff = BufReader::new(&f);
(&mut buff)
    .lines()
    .map(|l| l.unwrap())
    .map(|s: String| …)

I don’t understand why lines() returns an io::Result<String> and not an io::Result<&'a str> with 'a being the lifetime of the BufRead. Is it possible parse a text line by line without having to re-allocate a string for each line?

BufRead::lines() returns String, because it implements the Iterator interface. Iterator — by definition — promises you can always use all of the items at the same time, and they remain unchanged and valid even after you destroy the iterator (so e.g. collect() can make a Vec of them). To have all lines available as &str without ever invalidating any previous one, it would need the whole file read into memory first. And you can do that indeed:

let whole_file = std::fs::read_to_string(…)?; 
for line in whole_file.lines() { // gives &str borrowed from whole_file
}

For truly streaming solution, use string.clear() + BufRead::read_line() in a while loop.

4 Likes

Perfect, it’s exactly what I wanted. I thought that it was what BufReader was doing. But it look like it’s just parsing chunks one at a time.

Yep, the implementation for lines() is actually quite straightforward. It calls read_line() on an empty String and trims the trailing \r\n.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.