Using array_chunks_mut for liburing buffers

[ccarral]

Does array_chunks_mut add an indirection level to the resulting slices, rendering them inadequate for liburing syscalls?

I am trying to perform a PoC of liburing through rust via tokio/io-uring. In short, I am trying to queue multiple preadv2 calls into a file, and use an array as a backing buffer.

This works:

fn liburing_cp(in_path: &PathBuf, out_path: &PathBuf) -> Result<()> {
    const ENTRIES: usize = 1;
    const IOVCNT: usize = 8;
    const IOVCNT_BUFLEN: usize = 8;
    const BUFSIZE: usize = ENTRIES * IOVCNT * IOVCNT_BUFLEN;
    let mut buffer = [0u8; BUFSIZE];
    let mut io_uring = IoUring::new(ENTRIES as u32)?;
    let in_file = File::open(in_path)?;

    let slice = &mut buffer[0..IOVCNT * IOVCNT_BUFLEN];
    let mut read_chunks: Vec<IoSliceMut> = slice
        .chunks_exact_mut(IOVCNT_BUFLEN)
        .map(IoSliceMut::new)
        .collect();

    assert_eq!(IOVCNT, read_chunks.len());

    let readv_e = opcode::Readv::new(
        Fd(in_file.as_raw_fd()),
        read_chunks.as_mut_ptr().cast(),
        IOVCNT as _,
    )
    .build()
    .flags(squeue::Flags::IO_LINK);
    unsafe {
        io_uring.submission().push(&readv_e)?;
    }
    io_uring.submit_and_wait(1)?;
    let completion_results: Vec<i32> = io_uring.completion().map(|cqe| cqe.result()).collect();
    assert_eq!(completion_results.len(), 1);
    dbg!(completion_results);
    assert!(completion_results.iter().all(|read_res| *read_res > 0));
    Ok(())
}

However, when I try to put it inside a loop to queue multiple preadv2 calls, it fails.
Note that we are still queuing a single preadv2 call, even though it's in a loop.

fn liburing_cp(in_path: &PathBuf, out_path: &PathBuf) -> Result<()> {
    // NOTE: Changed the number of entries to 1
    const ENTRIES: usize = 1;
    const IOVCNT: usize = 8;
    const IOVCNT_BUFLEN: usize = 8;
    const BUFSIZE: usize = ENTRIES * IOVCNT * IOVCNT_BUFLEN;
    let mut buffer = [0u8; BUFSIZE];
    let mut io_uring = IoUring::new(ENTRIES as u32)?;
    let in_file = File::open(in_path)?;
    for slice in buffer.array_chunks_mut::<{ IOVCNT * IOVCNT_BUFLEN }>() {
        let mut read_chunks: Vec<IoSliceMut> = slice
            .chunks_exact_mut(IOVCNT_BUFLEN)
            .map(IoSliceMut::new)
            .collect();

        assert_eq!(IOVCNT, read_chunks.len());

        let readv_e = opcode::Readv::new(
            Fd(in_file.as_raw_fd()),
            read_chunks.as_mut_ptr().cast(),
            IOVCNT as _,
        )
        .build()
        .flags(squeue::Flags::IO_LINK)
        .user_data(0x01);
        unsafe {
            io_uring.submission().push(&readv_e)?;
        }
    }

    io_uring.submit_and_wait(ENTRIES)?;
    let completion_results: Vec<i32> = io_uring.completion().map(|cqe| cqe.result()).collect();

    dbg!(&completion_results);
    assert!(completion_results.iter().all(|read_res| *read_res > 0));
    Ok(())
}

The second snippet fails with errno either -14 (EFAULT) or -22 (EINVAL), which makes me think something funky is happening in array_chunks_mut, either i'm passing a pointer that is invalid when the loop ends or is dropped before the syscall completes.

Any ideas?

[quinedot]

Just spitballing after reading the code, but

@ccarral

    for slice in buffer.array_chunks_mut::<{ IOVCNT * IOVCNT_BUFLEN }>() {
        let mut read_chunks: Vec<IoSliceMut> = slice ... collect();
        let readv_e = opcode::Readv::new(
            ...
            read_chunks.as_mut_ptr().cast(),

read_chunks gets dropped at the end of the for loop.

[ccarral]

Yes, I think you are right. And even more so, i think the slices get dropped after the for loop. I tried "unrolling" the inner read_chunks and just iterate over a vec of IoSliceMut but the problem remains.

Any ideas?

[quinedot]

You need to keep things alive as long as the io_uring may need them. (This time I noticed you're dropping the readv_e too, but you pushed a reference to it, which presumably coerced to a raw pointer.)

One starting point (still untested) could be

    let mut slices: Vec<_> = buffer
        .array_chunks_mut::<{ IOVCNT * IOVCNT_BUFLEN }>()
        .map(|slice| {
            let mut read_chunks: Vec<IoSliceMut> = slice
                .chunks_exact_mut(IOVCNT_BUFLEN)
                .map(IoSliceMut::new)
                .collect();
            
            assert_eq!(IOVCNT, read_chunks.len());
            read_chunks
        })
        .collect();
    
    let readv_es: Vec<_> = slices
        .iter_mut()
        .map(|slice| {
            // Build Readv here
        })
        .collect();

    for readv_e in &readv_es {
        unsafe { io_uring.submission().push(readv_e)? };
    }

And with a bit more work you could use arrays instead of Vec.

(This is still all based on reading the example code; someone familiar with uring might have better advice.)

---

I do idly wonder if you'd be better off with some more Rustic wrappers around the pointer-taking things that took references instead, so you'd get borrow check errors.

[ccarral]

This actually worked. Thanks for answering!