IPC and DataFrames

I've been coding for about 2-3 weeks in Rust. I'm absolutely loving all the great features of the language.

My script aims to accomplish IPC with DataFrames via a named pipe. We serialize a DataFrame into Arrow's IPC format and write the contents to an in memory buffer. We then use this buffer to pass our serialized DataFrame into our named pipe. This will be later consumed by some other process.

Is it possible to create a 0 overhead iterator for the DataFrame so that you copy the DataFrame values as the type they need to be when writing them into the pipe?

Is it possible to move the underlying memory of the DataFrame into the buffer since it is not needed after writing it to the pipe? Don't want to duplicate memory. This would also save an allocation & deallocation.

These questions are just theories I've had I am completely open to any other ideas as well.
Any suggestions that could make this little script better would also be great.


    use std::io::prelude::*;
    use std::io::Cursor;
    use polars::prelude::*;

        // mkpipe creates a named pipe and returns its path
        fn mkpipe(name: String) -> String {
            let prefix = "/tmp/datamon/";
            let path = format!("{}{}", prefix, name);
                .expect(&format!("Error: couldn't create the folder {}", prefix));
            let filename = std::ffi::CString::new(path.clone()).unwrap();
            unsafe { libc::mkfifo(filename.as_ptr(), 0o600); }
            return path

        fn main() {
            // df creation
            let a = UInt32Chunked::new_from_slice("a", &[1, 900, 3]).into_series();    
            let b = Float64Chunked::new_from_slice("b", &[10., 8., 6.]).into_series();
            let mut df = DataFrame::new(vec![a, b]).unwrap();
            // create in memory buffer
            let mut buf:Cursor<Vec<u8>> = Cursor::new(Vec::new());
            // serialize and write to buffer
            IpcWriter::new(&mut buf).finish(&mut df)
                .expect("ipc writer");
            // create named pipe
            let path:String = mkpipe("data".to_string());
            // open fifo
            let mut fifo = std::fs::OpenOptions::new().write(true).open(path)
                .expect("Error: couldn't open the named pipe");
                // pass buffer into IO object
                .expect("Error: couldn't write the named pipe");

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.