I am looking for a concurrency construct that allows multiple writers to append to a file, while also enabling readers to read from it concurrently, with minimal contention and synchronization overhead. So, it is something like a (maybe lockless) queue but is actually a file. The only thing that a writer can do is append. And I don't want to take a Read-Write lock on it as I want to allow readers to read. What concurrency solution in rust can be used to support these requirements? I feel like lockless constructs, maybe, are required here.
These requirements are from the Google File System paper. The extracts from the paper that motivate my question are:
Third, most files are mutated by appending new data rather than overwriting existing data. Random writes within a file are practically non-existent. Once written, the files are only read, and often only sequentially. A variety of data share these characteristics. Some may constitute large repositories that data analysis programs scan through. Some may be data streams continuously generated by running applications. Some may be archival data. Some may be intermediate results produced on one machine and processed on another, whether simultaneously or later in time. Given this access pattern on huge files, appending becomes the focus of performance optimization and atomicity guarantees, while caching data blocks in the client loses its appeal.
The system must efficiently implement well-defined semantics for multiple clients that concurrently append to the same file. Our files are often used as producer-consumer
queues or for many-way merging. Hundreds of producers, running one per machine, will concurrently append to a file. Atomicity with minimal synchronization overhead is essential. The file may be read later, or a consumer may be reading through the
I feel that this question is not rust specific, but I don't know any other place which in my opinion is more suitable than here, to ask the question (Please do point me to a better place for the question if you have any in mind).