Removing specific lines from a large file

let's say we have a large file about 4Gb plus in size, we wish to remove some specific lines from that file
for e.g

this is line 1
this is line2
another line
yet another line
unexhaustive line
unexplained line

can we do this just by using BufStream<File> type in rust, using certiain API of async tokio, and not by utilizing any buffer operations on this to open a new temp file and rewrite the contents of last file on the newly opened file ?

It is possible to open a file for both reading and writing using OpenOptions:

let mut file = tokio::fs::OpenOptions::new()
    .read(true)
    .write(true)
    .create(false)
    .open("./my_big_data_file.txt")
    .await
    .unwrap();

let mut buffered_file = tokio::io::BufStream::new(file);

The buffered file then supports seeking, reading, writing, etc. However, you have to be careful not to overwrite existing data before you're done reading it. Also, if your program crashes, you could leave your data in an inconsistent state. For this reason, editing files in-place is generally not recommended unless you have a really compelling reason.

2 Likes