Flate2 decompress is returning an empty slice

[Macho-Onion]

I switched over from the enflate/deflate crates to the flate2 crate. All went well; until, I noticed that my decoding function was failing the integration test with a strange error:

thread chunk_pipline_works panicked at called Result::unwrap() on an `Err` value: Io(Kind(UnexpectedEof))

I investigated, And found that the decoded data was an empty slice: []

Here's the minimal code, I'm not sure what's wrong:

#[derive(Clone, Serialize, Deserialize, PartialEq, Eq, Debug, Default)]
struct Chunk {
    id: u16,
    hp: u8,
    stress: u8,
    lighting: i8,
}


#[inline(always)]
pub fn encode_chunk(data: &Chunk) -> Vec<u8> {
     GzEncoder::new(bincode::serialize(data).unwrap(), Compression::default()).finish().unwrap()
}



#[inline(always)]
pub fn decode_chunk(bytes: Vec<u8>) -> Chunk {
    // This is where the panic happens
    bincode::deserialize(&GzDecoder::new(bytes.as_slice()).into_inner()).unwrap()
}
pub fn write_chunk(name: String, data_encoded: Vec<u8>) {
    let mut chunk_file = File::create(Path::new(&name)).unwrap();
    chunk_file.write_all(&data_encoded).unwrap();
}

pub fn read_chunk_data(selected_chunk: &String) -> Vec<u8> {
    let mut chunk_from_file = Vec::with_capacity(CHUNK_SIZE_SQUARED);
    let mut read_chunk = std::fs::File::open(selected_chunk).unwrap();
    read_chunk.read_to_end(&mut chunk_from_file).unwrap();

    chunk_from_file
}

fn main() {
    let encoded_chunk = encode_chunk(&Chunk::new());
    
    let name: u128 = random();
    
    let name_formated = format!("saves/{}", name);
                std::fs::create_dir(Path::new(&name_formated)).unwrap();

    let chunk_path = format!("saves/{}/chunk{}_{}.dat", name, 0, 0);

    write_chunk(chunk_path.clone(), encoded_chunk);
    
    let chunk_from_file = read_chunk_data(&format!(
        "saves/{}/chunk{}_{}.dat",
        name.clone(),
        0,
        0,
    ));
    
    let chunk = decode_chunk(chunk_from_file);
}

[RedDocMD]

At which line is the empty buffer produced?

[Macho-Onion]

GzDecoder::new(bytes.as_slice()).into_inner()

[H2CO3]

You are abusing the library. The new() and into_inner() methods take and return the underlying reader/writer, not the already processed (compressed/decomoressed) data, as this is evident from the docs. You have to use the types' Read and Write impls to actually perform (de)compression.

[Macho-Onion]

I followed your advice and I updated my code:

#[inline(always)]
pub fn decode_chunk(bytes: Vec<u8>) -> Chunk {

	let mut buf: Vec<u8> = Vec::new();
	
	let mut g = ZlibDecoder::new(bytes.as_slice());
    	
    g.read_to_end(&mut buf).unwrap();
    	
    bincode::deserialize(&buf).unwrap()
    
}

But now I get this error:

thread 'chunk_pipline_works' panicked at 'called Result::unwrap() on an Err value: Custom { kind: InvalidInput, error: "invalid gzip header" }'

I wonder If I encoded it wrong somehow.

This is gonna be a long day.

[H2CO3]

You are encoding with zlib and trying to unconpress it with gzip. There's no reason why that should work.

[Macho-Onion]

Fixed that, Now I am getting a corrupt deflate stream. that last one was on me XD

[H2CO3]

If you are correctly compressing with Gzip and also decompressing with Gzip, then it should definitely work. A minimal repro does indeed work, so you are doing something else wrong, too.

[Macho-Onion]

Looks like the culprit might be the bytes.as_slice() in my code. Problem is, It has to be a slice, since Vectors don't support Read.

pub fn decode_chunk(bytes: Vec<u8>) -> Chunk {

    let mut buf: Vec<u8> = Vec::new();

    let mut decoder = GzDecoder::new(bytes.as_slice());
    decoder.read_to_end(&mut buf).unwrap();
    bincode::deserialize(&buf).unwrap()

}

[H2CO3]

as_slice() likely has nothing to do with this. It's literally just a conversion from an owned vector to a borrowed slice. Did you verify that bytes contains the correct, complete, compressed data?

[Macho-Onion]

Looks like it (I had to rather trim it to fit with the character limit:

Summary

[0, 16, 0, 0, 0, 0, 0, 0, 3, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 4, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1, 31, 139, 8, 0, 0, 0, 0, 0, 0, 255, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0]

I did a test to make sure that the encoded chunk was the same as the encoded chunk read from the file. They are.

[semicoleon]

Are you sure you're importing the Encoder and Decoder from the right module? flate2 has encoders and decoders which function on Write types and ones that function on Read types. Mixing those up may do wacky things.

[Heliozoa]

A runnable example that reproduces your issue would be great. The one in the OP has stuff like an echk variable and Chunk type that aren't defined anywhere and the imports are missing.

I see you fixed the decoder function, but what about the encoder?

[Macho-Onion]

There, The echk was one I forgot to remove. And I added a Chunk struct.

[Macho-Onion]

Changing the import to

use flate2::{Compression, write::{GzEncoder, GzDecoder}};

Does nothing.

---- chunk_pipline_works stdout ----
thread 'chunk_pipline_works' panicked at 'called `Result::unwrap()` 
on an `Err` value: 
Custom { kind: InvalidInput, error: "invalid gzip header" }', 
src/lib.rs:236:31
note: run with `RUST_BACKTRACE=1` 
environment variable to display a backtrace

Current code:

#[inline(always)]
pub fn encode_chunk(data: &Chunk) -> Vec<u8> {
    GzEncoder::new(bincode::serialize(data).unwrap(), Compression::default()).finish().unwrap()
}

pub fn decode_chunk(bytes: Vec<u8>) -> Chunk {
    let mut decoder = GzDecoder::new(Vec::new());
    decoder.write_all(&bytes).unwrap();
    bincode::deserialize(&decoder.finish().unwrap()).unwrap()
}

[tornewuff]

The encoder/decoder types in flate2 are wrappers around Read, Write, or BufRead types. None of them "do" anything useful just by creating them.

A flate2::write::GzEncoder is an adapter that takes some other writer (e.g. an open file), and gives you a new writer that will compress any data you write to it, and then write the compressed data to the original writer. So, your code here is saying "give me a writer that will write compressed data to the end of the Vec<u8> returned by bincode::serialize", and then you don't write any data to that writer. This will just append a gzip header to the end of the serialized data with no compressed data, and then return the original Vec<u8>, now containing the original uncompressed data followed by a gzip header.

You probably want something like this (not actually tested):

pub fn encode_chunk(data: &Chunk) -> Vec<u8> {
    // create an encoder that will write to a vector that starts off empty
    let mut encoder = flate2::write::GzEncoder::new(Vec::<u8>::new(), Compression::default());
    // write the serialized data to the encoder
    encoder.write_all(bincode::serialize(data).unwrap()).unwrap();
    // finish the compression operation and return the vector that it was writing to
    encoder.finish().unwrap()
}

[semicoleon]

@Macho-Onion

GzEncoder::new(bincode::serialize(data).unwrap(), Compression::default()).finish().unwrap()

This line isn't actually performing any encoding, you're just constructing the encoder, then destroying it.

I think it's simpler to just have bincode serialize directly into the encoder rather than trying to do it in two distinct steps.

use flate2::{read, write, Compression};
use serde::{Deserialize, Serialize};

#[derive(Clone, Serialize, Deserialize, PartialEq, Eq, Debug, Default)]
pub struct Chunk {
    id: u16,
    hp: u8,
    stress: u8,
    lighting: i8,
}

#[inline(always)]
pub fn encode_chunk(data: &Chunk) -> Vec<u8> {
    // Compress the data into a Vec via `std::io::Write`
    let mut encoder = write::GzEncoder::new(Vec::new(), Compression::default());

    // Have bincode write to the encoder
    bincode::serialize_into(&mut encoder, data).unwrap();

    encoder.finish().unwrap()
}

#[inline(always)]
pub fn decode_chunk(bytes: Vec<u8>) -> Chunk {
    // Decompress the data via `std::io::Read`
    let mut decoder = read::GzDecoder::new(bytes.as_slice());

    // Have bincode deserialize directly from the decoder
    bincode::deserialize_from(&mut decoder).unwrap()
}

fn main() {
    let test_chunk = Chunk {
        id: 1,
        hp: 1,
        stress: 1,
        lighting: 1,
    };

    let encoded_chunk = encode_chunk(&test_chunk);

    let chunk = decode_chunk(encoded_chunk);

    assert_eq!(test_chunk, chunk);
}

[H2CO3]

It would also be better to update your code to what you are actually running. Even your latest example still shows mismatching compression and decompression, and there's no fn main()/#[test] fn. Since details like this can make the difference between working and non-working code, it's very hard for us to guess how to help you if you don't include this sort of information exactly.

[Macho-Onion]

Sorry about all of this, It's solved now, next time, I will be sure to provide a much better post.