Write to normal or gzip file transparently

[enormandeau]

Using flate2 and some examples I found online I made a function, called reader, to read both uncompressed and compressed files transparently using the presence of .gz at the end of the name file to decide whether the file is compressed or not.

I am now trying to have a second function, named writer, to write files to either normal or gzip-compressed files if there is .gz at the end of the filename. I have tried a few things but I am out of depths it seems. Namely, the .gz output file is a normal, uncompressed file.

Would anyone have an idea of how to solve this seemingly simple problem?

Also, I am not sure the Box<dyn BufRead> part is really needed. Can this be simplified?

Here is my code so far. The writer function doesn't work.

use flate2::read::GzDecoder;
use std::error::Error;
use std::ffi::OsStr;
use std::fs::File;
use std::io::{self, BufRead, BufReader, Write, BufWriter};
use std::path::Path;

/// Read normal or compressed files seamlessly
/// Uses the presence of a `gz` extension to choose between the two
pub fn reader(filename: &str) -> Box<dyn BufRead> {
    let path = Path::new(filename);
    let file = match File::open(&path) {
        Err(why) => panic!("couldn't open {}: {}", path.display(), why.description()),
        Ok(file) => file,
    };

    if path.extension() == Some(OsStr::new("gz")) {
        Box::new(BufReader::with_capacity(128 * 1024, GzDecoder::new(file)))
    } else {
        Box::new(BufReader::with_capacity(128 * 1024, file))
    }
}

/// Write normal or compressed files seamlessly
/// Uses the presence of a `gz` extension to choose between the two
// Attempting to have a file writer too
pub fn writer(filename: &str) -> Box<dyn Write> {
    let path = Path::new(filename);
    let file = match File::create(&path) {
        Err(why) => panic!("couldn't open {}: {}", path.display(), why.description()),
        Ok(file) => file,
    };

    if path.extension() == Some(OsStr::new("gz")) {
        // Error is here: Created file isn't gzip-compressed
        Box::new(BufWriter::with_capacity(128 * 1024, GzDecoder::new(file)))
    } else {
        Box::new(BufWriter::with_capacity(128 * 1024, file))
    }
}

/// Doing tests
fn main() -> io::Result<()> {
    // Test with uncompressed file
    let filename = "file.txt";
    println!("Testing reader with uncompressed file: '{}'", filename);
    let reader_file = reader(filename);
    for line in reader_file.lines() {
        println!("{}", line?);
    }
    println!();

    // Test with compressed file
    let filename = "file.txt.gz";
    println!("Testing reader with compressed file: '{}'", filename);
    let reader_file_gz = reader(filename);
    for line in reader_file_gz.lines() {
        println!("{}", line?);
    }
    println!();

    // Test writing to uncompressed file
    let filename = "file.output.txt";
    println!("Testing writer with compressed file: '{}'", filename);
    let mut writer_file = writer(filename);
    writer_file.write(b"asdf\n")?;
    writer_file.write(b"asdf\n")?;
    writer_file.write(b"asdf\n")?;

    // Test writing to compressed file
    let filename = "file.output.txt.gz";
    println!("Testing writer with compressed file: '{}'", filename);
    let mut writer_file = writer(filename);
    writer_file.write(b"asdf\n")?;
    writer_file.write(b"asdf\n")?;
    writer_file.write(b"asdf\n")?;

    Ok(())
}

[kornel]

Box<dyn Write> is correct in this case, because you dynamically change what code is going to be executed, and you need to return an owned value.

The code doesn't work, because you're using GzDecoder that decompresses. You need GzEncoder to make a compressed file.

[enormandeau]

Thanks! I got it to work like this:

use flate2::read;
use flate2::write;
use flate2::Compression;
use std::error::Error;
use std::ffi::OsStr;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Write};
use std::path::Path;

/// Read normal or compressed files seamlessly
/// Uses the presence of a `.gz` extension to decide
pub fn reader(filename: &str) -> Box<dyn BufRead> {
    let path = Path::new(filename);
    let file = match File::open(&path) {
        Err(why) => panic!("couldn't open {}: {}", path.display(), why.description()),
        Ok(file) => file,
    };

    if path.extension() == Some(OsStr::new("gz")) {
        Box::new(BufReader::with_capacity(
            128 * 1024,
            read::GzDecoder::new(file),
        ))
    } else {
        Box::new(BufReader::with_capacity(128 * 1024, file))
    }
}

/// Write normal or compressed files seamlessly
/// Uses the presence of a `.gz` extension to decide
// Attempting to have a file writer too
pub fn writer(filename: &str) -> Box<dyn Write> {
    let path = Path::new(filename);
    let file = match File::create(&path) {
        Err(why) => panic!("couldn't open {}: {}", path.display(), why.description()),
        Ok(file) => file,
    };

    if path.extension() == Some(OsStr::new("gz")) {
        // Error is here: Created file isn't gzip-compressed
        Box::new(BufWriter::with_capacity(
            128 * 1024,
            write::GzEncoder::new(file, Compression::default()),
        ))
    } else {
        Box::new(BufWriter::with_capacity(128 * 1024, file))
    }
}

/// Doing tests
fn main() -> io::Result<()> {
    // Test with uncompressed file
    let filename = "file.txt";
    println!("Testing reader with uncompressed file: '{}'", filename);
    let reader_file = reader(filename);
    for line in reader_file.lines() {
        println!("{}", line?);
    }
    println!();

    // Test with compressed file
    let filename = "file.txt.gz";
    println!("Testing reader with compressed file: '{}'", filename);
    let reader_file_gz = reader(filename);
    for line in reader_file_gz.lines() {
        println!("{}", line?);
    }
    println!();

    // Test writing to uncompressed file
    let filename = "file.output.txt";
    println!("Testing writer with compressed file: '{}'", filename);
    let mut writer_file = writer(filename);
    for _i in 1..=100 {
        writer_file.write_all(b"This is the end. Count your chickens.\n")?;
    }

    // Test writing to compressed file
    let filename = "file.output.txt.gz";
    println!("Testing writer with compressed file: '{}'", filename);
    let mut writer_file = writer(filename);
    for _i in 1..=1000 {
        writer_file.write_all(b"This is the end. Count your chickens.\n")?;
    }

    Ok(())
}