Idiomatic way of reading a text file line by line in reverse


#1

What would be the most idiomatic way to read a text file line by line in reverse. I tried to call rev() on BufRead#lines() however that does not work. Any ideas?


#2

What problem are you trying to solve? i.e., Why do you need to read the lines of a file in reverse?

I expect you will need to roll this on your own if you really need it. It is quite an uncommon use case. You’ll have to, for example, use seek calls to position the file pointer in the correct place. Using BufReader won’t work because it reads the file in normal order.


#3

It’s a rare problem. File system doesn’t allow you to read a file from the end.

You need to do the following(or something like this).

use std::io::{BufRead, BufReader};
use std::fs::File;

fn main() {
    
    let file = BufReader::new(File::open("read_file.rs").unwrap());
    let mut lines: Vec<_> = file.lines().map(|line| { line.unwrap() }).collect();
    lines.reverse();
    
    for line in lines.iter() {
        println!("{}", line);
    }
}

#4

Perhaps reading to a string is more spece-efficient.

use std::fs::File;
use std::io::Read;

fn main() {
    let mut file = File::open("foo.txt").expect("opening file");
    let mut text = String::new();
    file.read_to_string(&mut text).expect("reading file");

    for line in text.lines().rev() {
        println!("{}", line);
    }
}

#5

Thanks for the replies and examples!

I’m looking to read log files in reverse and be able to stop when I reach a specific time.


#6

The given examples are not really usable for a big file, because it will scan the entire file to the end.

You will have to seek to some position close to the end of the file, read a chunk, do with it whatever you need to do, and repeat.

No guarantees here, I’m just scribbling this down for reference:

let mut f = File::open("foo.txt").unwrap();
let mut buffer = String::new();
// Determine file size.
let file_size = f.metadata().unwrap().len();
 // read last 10240 bytes. Adapt this for your requirements.
let chunk_size = 10240;
let start_pos = if file_size < chunk_size { 0 } else { file_size - chunk_size };
// Seek to end of file - chunk_size.
f.seek(std::io::SeekFrom::Start(start_pos)).unwrap();
// Read to buffer.
f.take(chunk_size).read_to_string(&mut buffer).unwrap();

let mut lines = buffer.lines();

// The first line is probably a partial line now, cut off in the middle.
// Safe it for now...
let first_line = lines.next().unwrap();

// Do whatever with the actual lines.
for line in lines { println!("{}", line.unwrap()); }

// Repeat process by seeking back by chunk_size again.
// Note that you now have to merge the last line of the new buffer contents with the previous first_line...
.....
...

If you wrap this up in a nice struct it’s not so bad. :wink:


#7

Check out bounded_tail here https://github.com/uutils/coreutils/blob/master/src/tail/tail.rs which is doing somewhat similar things under the hood.


#8

Thanks for the replies all! They are helping me develop a small Iterator that reads a file line-by-line in reverse using an internal buffer.


#9

Nice. Might be useful to package that up into its own crate.


#10

I put together a crate for this and published it:

https://crates.io/crates/rev_lines


#11

Sent you a pull request to generalize it to work on other Seek+Read types other than just files. Thanks for making it. Should be useful for the occasional need.

Oh, and I think your Travis tests are failing due to the documentation code. I had the same problem running cargo test locally, and deleting your code comments fixed it.


#12

Thanks for the PR! Fixed up the docs and released another version.