Why tokio read_to_end only fill the buffer capacity

I am trying to test a zstd encode and decode function.

With Cargo.toml like this

[package]
name = "zstd-test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokio = { version = "1.16.1", features = ["full"] }
async-compression = { version = "0.3.12", features = ["all"] }

Code

use tokio::{fs, io::{BufReader, AsyncReadExt, AsyncWriteExt}};
use async_compression::{
    tokio::write::{ZstdEncoder},
    tokio::bufread::{ZstdDecoder},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut f = fs::File::open("/etc/group").await?;
    let mut buf = vec![];
    f.read_to_end(&mut buf).await?;
    let f = fs::File::create("group.zst").await?;
    let mut f = BufReader::with_capacity(4096, f);  
    let mut encoder = ZstdEncoder::new(&mut f);
    tokio::io::AsyncWriteExt::write_all(&mut encoder, &buf).await.unwrap();
    encoder.flush().await.unwrap();

    let f = fs::File::open("group.zst").await?;
    let mut f = BufReader::new(f);
    let mut decoder = ZstdDecoder::new(&mut f);
    let mut buf = Vec::with_capacity(100); // why?

    let r = decoder.read_to_end(&mut buf).await;
    f.flush().await?;
    println!("{:?}", r);
    println!("{:?}", buf.len());
    
    println!("{}", String::from_utf8_lossy(&buf));
    Ok(())
}

The code above create a buffer with 100 capacity, and then call read_to_end,But turn out it only read 100 bytes.

How can I actually read to end rather than only fill the buffer

Hm, that's weird. It normally works perfectly fine. The amount changes as you change the initial capacity? What if you set it to zero?

It will get 32 bytes just like I use Vec::new() to create a buffer.

I found what's wrong L16 should be encoder.shutdown(); rather than encoder.flush()

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.