Towards a more perfect RustIO


#41

An example that occurs to me would be a page- or extent-oriented database file, doing some compaction and cleaning. You’d have a write window and a set of read windows collecting records from sparsely-populated pages, which can then be freed and recycled for more writes. Probably something with larger pages than your typical postgresql MVCC - perhaps a CoW VM disk image file.


#42

I see. In this case, we may want to allow for concurrent slices, which can be done by allowing one to create a read/write slice from an &self (and using as much internal mutability as necessary behind the scenes).

You would also need to wrap read slices in addition to write slices, because you must be able to discard the underlying memory mapping or buffer when it is not used anymore (which was done automatically in the previous API).

End result would look like this:

fn read_managed(&'a self) -> Result<ReadWrapper<'a>>
fn begin_write(&'a self) -> Result<WriteWrapper<'a>>;

One important question that must be resolved is whether the file slices should be borrowed (as I am currently proposing) or owned. The use case that @newpavlov presents would benefit from owned slices that can be e.g. passed from the IO thread to some processing thread, but it is also important to realize that such owned slices would come at a cost. IO sources, such as Files, would have be atomically reference-counted + synchronized so that their destruction is delayed until all slices in flight have been discarded.

Here is the perspective which I am coming from regarding the unsafety problems of memory-mapped files:

  • There is no such thing as a read-only file on a writable storage media, which is the vast majority of computer storage today. That could only be achieved by an OS with a strong commitment towards file immutability, and current OSes do not provide that.
  • On current-gen OSes, “read-only” filesystem flags are just a protection against basic usage errors and a documentation for system administrators. They are not a useful protection against mmap-induced data races:
    • The fact that a file was marked as read-only at the time where you opened it doesn’t mean that it will remain read-only during the entire time where you will be using it.
    • Even if the whole filesystem on which a file is located were mounted in read-only mode from your perspective, it can still be mounted in writable mode elsewhere (especially in VM or distributed filesystem scenarios).

Therefore, from my perspective, files should always be assumed to be shared mutable resources, and any attempt to provide a safe interface to memory-mapping must provide a synchronization protocol through which one can avoid data races between programs which concurrently mmap the same file.

The simplest reasonably efficient synchronization protocol that can be used here is to enforce reader-writer lock semantics between concurrent users of a given file region. I’m opened to other synchronization protocol suggestions if you have them.

One problem is that as a crufty heir of UNIX, Linux does not even provide the required building blocks for enforcing such synchronization system-wide. All we may (or may not) be able to achieve is to guarantee that consenting applications can opt-in to such a synchronization protocol. This is what I would like to achieve here. In this prospect, opening a file would be unsafe with a “make sure no one else will be touching that file as long as it is open” contract, but further transactions on that file could be considered safe if that core contract is upheld.

A reader-writer lock normally works by blocking until the target resource is free. However, in the case of file access, I think this is inappropriate, because it could result in an application hanging permanently on file open if another application is currently manipulating the same file. For an mmap use case, I think it would be more appropriate to assume that concurrent read+write or write+write memory mappings normally do not happen, and that if they happen it is the result of a bug. In this case, returning an error from the API when a racey memory mapping is requested would be the right thing to do.

I agree with you here. The hazards exist in any case, and must be handled, due to cross-process access to shared files. I just thought that the “one slice at a time” model would ease reasoning in the common case where a file is only opened by one application at a time, and is only opened once in that application.

If you think that the ergonomics (and implementation simplicity) benefits of only allowing one slice at a time are not worth it in the face of the increased flexibility that concurrent slices bring, then we can go for the more complex, but more flexible solution sketched above.


#43

I believe the right approach to a read trait which will allow us to create several zero-copy views into underlying buffer is borrow regions functionality. In read_managed and seek methods we have mutable part (counter) and immutable (buffer), so we need somehow to convey this information to borrow checker. All other workarounds will be less ergonomic or efficient.

As for hazards associated with memory mapped files I have a feeling that we should change perspective a bit. Instead of making creation of mapped files unsafe, it could be better to make acquisition of &[u8] unsafe instead (same for &mut [u8]). So you will be able to safely create mapped files, slice (instead of &[u8] you’ll get opaque MmapSlice<'a>), index (volatile read of one byte) and to read and write data via existing Read and Write traits, implementations of which will use volatile operations under the hood. You will be able to get zero-copy &[u8] from MmapSlice<'a> using unsafe method, but you’ll have to deal with potential problems. Hopefully it will allow us to localize some of the problems.


#44

Regarding use cases, you mention Network, Database, and File IO. File IO in particular is a deep topic that doesn’t end with the POSIX api and this is a place where Rust can do really well.

Generally there are three main use cases of File IO:

  1. Object - a file with write isolation and no dirty reads. It is only available when the application has finished writing it. (e.g. media files).
  2. Log - a file that is continually appending. Dirty reads are possible but only with record based framing (e.g. you can see the last record; but you cannot see a corrupt piece of a record being written). (e.g. WAL for a db).
  3. memmapped for persistent caches (with recoverability provided by logs). (e.g. actual db files).

Objects + Logs get you most of what you want. Being able to e.g. open up a typed channel to a file system sink (mspc::Receiver or crossbeam-io::channel::Receiver) would be a great.


#45

From an API perspective, I would like to see a more fluent IO API for Rust. The current one effectively forces an imperative programming style:

...
fn foo() -> Result<String> {
    let mut contents = String::new();
    File::open("foo.txt")?.read_to_string(&mut contents)?;
    Ok(contents)
}

To have the API create and return the buffer as a Result would be more ergonomic IMHO:

...
fn foo() -> Result<String> {
    File::open("foo.txt")?.read_to_string()
}

This kind of stuff adds up fast. The former (our current API) feels much clunkier for cases in which the buffer is not being re-used. For these cases, I’d like to see a variant of the IO routines which create and return a Result<BufferType>, rather than having them passed in.


Rust beginner notes & questions
#46

I kind of skimmed many parts of this threuad, but I did like @peter_bertok’s mention of iterators. Would it be a good idea/possible to implement IO/networking/streaming in the same fashion as iterators? You could do something like:

let file1 = File::open("foo.txt").unwrap();
let file2 = File::open("bar.doc").unwrap();
let combined = file1.into_stream()
    .add_file(file2)
    .mmap(mmap::Options::default())
    .collect<String>();

You could optionally add memory maps. You could add other methods for processing files and things with encodings or compressed formats. Each method would add a struct like the iterators do, and each struct would contain various configuration settings for that step (like the name of a file, or how large of memory maps to use, etc), then either process the data using a closure or collect the data into a vector of a certain type (integers, floats, whatever) or a String.
Each file could have an encoding specified or detected, and the data would not necessarily have to be represented in u8’s, it could be an associated type.

Just my quick thoughts from reading this and the other thread. It might be a horrible idea, I haven’t thought it out much but it would be nice to have an easy, flexible, and generic way to deal with data coming from some source (network, file, database, even a reference/slice to some data already in memory).


#47

Added this to the OP to highlight some recent new developments with respect to Tokio.


#48

This is already supported by the Read trait via fn chain(). This kind of thing is commonly required for compressed file formats that are provided in chunks or parts, such as the OOXML packaging format used by Microsoft Office.

I’ve been thinking about this a bit more, and the requirements are quite complex in the general case, much in the same way iterators and parser combinators have to support a wide range of scenarios.

I keep thinking of the worst-case scenarios, with the theory that working backwards from there to the simplest scenarios will then handle everything in between. Two good examples are:

  • Handling thousands of HTTP sockets using an efficient select()-style I/O, with:
    • TLS encryption ("CryptRead").
    • Chunked transfer encoding. This means that first you have to parse a header, and then chunks of data. Luckily, HTTP length-prefixes the chunks.
    • Each format has its own independent decoder, which itself takes a Read-derived source.
    • Some of these formats have 2-3 stage decoders, such as first a decompressor, and then a UTF-16 to UTF-8 converter, etc…
    • The final stage is possibly a parser such as the nom crate, JSON, XML, etc…
  • Handling a streaming format where the chunked layers are not length-prefixed. This can occur in some cases when nested formats are used and the inner format has a terminator marker instead of a length prefix.

In all cases, it’s important to be able to interact with some of the layer handlers:

  • Retrieve/verify authenticated encryption (AEAD) success or failure.
  • Retrieve TLS certificates.
  • Retrieve/verify compression checksums.
  • Retrieve the output of a parser or other consumer of the data.

So essentially a solution would have to support:

  • Format conversion that can change the length of the data (UTF-16 <-> UTF-8).
  • Format conversion that can’t change the length of the data, and is hence more efficient to perform in-place over the same buffer.
  • Chaining and Muxing multiple sources into multiple targets (HTTP/2).
  • Creating a subset similarly to slicing or skip(offs).take(len), e.g.: starting from some offset, create a child reader with a provided exact length to read.
  • Creating a subset that terminates itself (based on a marker in the stream).
  • Switching inner decoders mid-stream. E.g.: after decoding one chunk, the remaining data must then be passable to a different handler, which may have to be dynamically chosen (e.g.: based on a header).
  • Converting a stream to an iterator in various ways. E.g.: bytes(), lines(), packets(), or whatever…

Interestingly, there are virtually no I/O errors that are meaningful in these scenarios other than “unexpected end of stream”. For example, it generally makes no sense to have an “access is denied” for a stream that is already open. The only exception that I can think of is if one of the sources is something like a “RetryReader” that automatically re-opens the source file or socket if interrupted.

Conversely, it is fairly important to support non-I/O errors such as invalid cryptographic stream, corrupt compression, checksum failed, parsing errors, unexpected end-of-data, etc…


Stream API and types
Rust beginner notes & questions
Stream API and types
Rust beginner notes & questions
#49

@peter_bertok is it your intention to come up with a solution that duplicates the functionality of things like tokio and nom/peg/etc? Because it seems like some of that would normally be handled by other crates currently. I’m just wondering how you’d want to handle that. I thought this was more like a base crate for streaming and io stuff but what you’re describing is much more complex.

If your desired solution is too complex to easily implement with the resources available it might be prudent to start with a base project then expand to more complex features.


#50

My sense is that @peter_bertok is showing the terrain that a generalized approach must address. The next step would seem to be sketching a set of traits that would support such a wide span of processing architectures. Only at that point will it become apparent what issues are already addressed by existing Rust crates, thus needing only minor adaptation, and which issues require substantial new work.

Personally, I applaud the constructive direction that this thread is taking, rather than the antagonistic one of its predecessor.


#51

The reason I allowed multiple files to be added was to address something I read earlier about retrieving multiple files at a time instead of one per syscall, I don’t know if the Read trait recognizes that but I’m very skeptical it would.