How to read/write and get the bytes written without needing to wait

help
1

Hello, I'm trying to write data to block device (for example, /dev/sda). I am using the read and write_all from the std.

However I have noticed after my read/write loop:

loop {
                    let read_len = match input.read(&mut buf) {
                        Ok(0) => {
                            dbg!("Nothing written");
                            break;
                        }
                        Ok(new_read_len) => new_read_len,
                        Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {
                            // something went wrong
                            dbg!(e);
                            continue;
                        }
                        Err(e) => {
                            let msg = format!("Error reading file: {:?}", e);
                            println!("{}", &msg);
                            panic!();
                        }
                    };
                    wrtn1.fetch_add(read_len, Ordering::SeqCst);
                    output
                        .write_all(&buf[..read_len])
                        .expect("File writing failed");
                }

my device is still busy. So I have to currently call

nix::unistd::fsync(output_fd).expect("Call to fsync failed");

To ensure that the write is finished. But I want my writecall to only return when the write operation has finished in the device. I don't mind blocking the thread.

How can I get a blocking write to the device? For example, this program never needs to call fsync. I don't want the user to wait for an indefinite time for fsync to finish.

Source if necessary

2

If you want each write to be flushed to the device before continuing, call flush() after every write().

3

The imagewrite example also doesn't perform any fsync(2) call so it should behave identically with your Rust snippet without fsync, except it throws unwritten data away if it fails to write entire buffer within single system call, which is handled correctly via .write_all() in Rust.

4

Edit: Oops: On second look I see you already shared variable "pBar" and "written" to communicate progress. So disregard most of below.

I notice that this program uses the Qt GUI library and it's event loop. In the writeData() method I see this:

 for (i = 0; i <= realSize; i++)
    {
        ...
        written = ::write(ofd, buffer, read);
        if (written == -1)
        {
            ...
            break;
        }
        ...
        qApp->processEvents();

        if (progress.wasCanceled())
             break;
     }

So what is happening there is that it write chunks of data of size 1048576 bytes each, and makes a call to "qApp->processEvents();" after each write. In this way the Qt GUI interface does not freeze whilst the entire image is being written. The user gets to see the progress bar move and is able to cancel the operation because of that processEvents call. The entire write could take a long time but the program remains responsive throughout.

I notice in you Rust code you have a similar loop to do the write inside a thread of it's own. But there you indcate the operation is complete by setting a "finished" flag. Then your main line is polling that finished flag. So presumably everything freezes until the whole job is done.

I would look into changing that finish flag into a percentage amount of completion. The the UI thread could read that and display some kind of progress indicator to the user.

Or use mpsc channels to communicate progress from the writer thread to the UI thread: channel in std::sync::mpsc - Rust

Make calls to flush() after each write so that the user does not need to wait a long time at the end when you do a flush.

5

Actually I am polling on progress, which is bytes written and it's a load()ed value from written: Arc<AtomicUsize>. While it is lower than tl (total len) the polling is kept alive.

In action my writer thread freezes, granted but it also keeps updating the atomic usize and atomic bool. After the end of the loop it sets the finished flag. The Glib polling picks it up sometime later and cuts off the animation.

@Michael-F-Bryan @ZiCog, Even after flushing, I see that my device is busy (the flash drive LED keeps blinking).

6

How are you opening the device? The example code you linked specifies the O_SYNC flag to open. You’ll probably have to do the same.

7

Great catch! Turns out, the default OpenOptions doesn't enable the sync flag. Adding custom_flags(libc::O_SYNC) gave what I was looking for.

8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.