Can you recommend a high-level approach to storing and retrieving sequences of Rust structures in parquet files?
There seems to be a number of relevant crates, including at least
arrowarrow2arrow2_convertparquetparquet2serde_arrowpolars
I'm overwhelmed by the possibilities in an area with which I'm not familiar, and whatever approach I try, there is always some low-level fiddling involved that I'd rather not have to deal with.
To a first-order approximation my initial requirements are to be able to
- Iterate over a sequence of Rust structs, whose total size exceeds available RAM, and write it out to parquet.
- Iterate over the data written out in part 1, with modest use of RAM.
Is there a simple, high-level way of doing this?