How to compress huge file chunk by chunk?

Given that I need to compress a huge file for HTTP download, instead of reading the whole file and compressing it in memory, I need to compress each chunk of the file and send chunk by chunk via HTTP response.

But the flate2 crate does not export API to access intermedia compression results, it just collects the output of each chunk and only finish() returns the final result. But how about OOM?

In fact, the zlib c library provides the interface to access the intermedia result.

I know I could use Compress to compress buffer, but for gzip case, I need to write header and footer by hand, so I don't want to reinvent the wheel.

Why don't you just replace the File with GzEncoder<File>?

I don't get your point.
What I need is compressing chunk by chunk.
For example, I have a 1 GB file, and a 10 MB buffer, I need to read the file to the buffer each time and compress it and write it to HTTP response. According to HTTP spec, I need to compress the whole file, and send it chunk by chunk.
However, the GzEncoder has no interface to read each compression result, and it only provides finish() to finish the compress. The next time calling write(), it would be a new compression stream, which is unexpected.

It is not. Just treat it like a File, and it does all the things you've mentioned on the fly.

1 Like

You mean this class?

That's the wrapper over the Read side so to use it you need to wrap the http body reader. Both do the same thing. It's up to your preference to wrap the reader or the writer.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.