How to compress data in memory?

In golang I can do this simple

func main() {
    buff := new(bytes.Buffer)
    zw := zip.NewWriter(buff)
    addFile("/path/to/file1.txt", zw)
    addFile("/path/to/file2.txt", zw)
    
    ioutil.WriteFile("some.zip", buff.Bytes(), 0755)
}

func addFile(fname string, z *zip.Writer) {
	if FileExists(fname) {
		w, _ := z.Create(fname)
		data, _ := ioutil.ReadFile(fname)
		w.Write(data)
	}
}

How to in Rust? Thanks for answer.

The zip crate contains ZipWriter, which wraps an io::Write + io::Seek. The latter bounds are satisfied by io::Cursor, which you can use to wrap a slice-like object (e.g. an [u8; N] array or a Vec<u8>).

2 Likes

Like this?

    let buff = std::io::Cursor::new(Vec::new());
    let mut zw = zip::ZipWriter::new(buff);
    zw.start_file("some.txt", FileOptions::default())?;
    zw.write_all(b"hello")?;
    zw.finish()?;

I can't understand how to write bytes from Cursor to filesystem :frowning:

Like this (untested):

    let mut data = Vec::new();
    let buff = std::io::Cursor::new(&mut data);
    let mut zw = zip::ZipWriter::new(buff);
    zw.start_file("some.txt", FileOptions::default())?;
    zw.write_all(b"hello")?;
    zw.finish()?;

...and then do whatever you want with data.

But

31 |     let mut buff = std::io::Cursor::new(&mut data);
   |                                        --------- mutable borrow occurs here

Please provide the full error message. There's a good chance it will explain exactly what is wrong and give you suggestions for fixing it.

1 Like

Full error

error[E0502]: cannot borrow `data` as immutable because it is also borrowed as mutable
  --> src/main.rs:67:17
   |
31 |     let buff = std::io::Cursor::new(&mut data);
   |                                     --------- mutable borrow occurs here
...
67 |     f.write_all(data.as_slice()).unwrap();
   |                 ^^^^^^^^^^^^^^^ immutable borrow occurs here
68 | }
   | - mutable borrow might be used here, when `zw` is dropped and runs the `Drop` code for type `ZipWriter`

For more information about this error, try `rustc --explain E0502`.

Well you are trying to write (the still empty) data into itself. That is probably not what you wanted.

Why you think data is empty?

use std::fs::File;
use std::io::prelude::*;
use std::io::Cursor;
use zip::write::FileOptions;
use zip::ZipWriter;

fn main() {
    let mut data = Vec::new();
    let buff = Cursor::new(&mut data);
    let mut zw = ZipWriter::new(buff);
    zw.start_file("some.txt", FileOptions::default()).unwrap();
    zw.write_all(b"hello").unwrap();
    zw.finish().unwrap();
    let mut file = File::create("foo.zip").unwrap();
    file.write_all(data.as_slice()).unwrap();
}

Sorry, I have no idea. I tried to reproduce the error on my machine, but after fixing the borrowing error (by wrapping the zipping part into a block scope), it correctly created the file which I was able to unpack.

Please, show what you fixed.

    zw.finish().unwrap();
    drop(zw); // here
    let mut file = File::create("foo.zip").unwrap();

Thanks!

This is because ZipWriter implements Drop and the variable zw will be dropped at the end of the enclosing scope by default, leading to a longer lifetime of zw thus &mut data.

So you can wrap (all) the code about zw in a block or equivalently call drop function on it to stop the lifetime of &mut data.

Refer to 2094-nll - The Rust RFC Book if you're interested in it.

use std::fs::File;
use std::io::prelude::*;
use std::io::Cursor;
use zip::write::FileOptions;
use zip::ZipWriter;

fn main() {
    let mut data = Vec::new();

    {
        let buff = Cursor::new(&mut data);
        let mut zw = ZipWriter::new(buff);
        zw.start_file("some.txt", FileOptions::default()).unwrap();
        zw.write_all(b"hello").unwrap();
        zw.finish().unwrap();
    }

    let mut file = File::create("foo.zip").unwrap();
    file.write_all(data.as_slice()).unwrap();
}
1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.