High-level approach to IO of iterable-of-Rust-struct <-> parquet

Can you recommend a high-level approach to storing and retrieving sequences of Rust structures in parquet files?

There seems to be a number of relevant crates, including at least

  • arrow
  • arrow2
  • arrow2_convert
  • parquet
  • parquet2
  • serde_arrow
  • polars

I'm overwhelmed by the possibilities in an area with which I'm not familiar, and whatever approach I try, there is always some low-level fiddling involved that I'd rather not have to deal with.

To a first-order approximation my initial requirements are to be able to

  1. Iterate over a sequence of Rust structs, whose total size exceeds available RAM, and write it out to parquet.
  2. Iterate over the data written out in part 1, with modest use of RAM.

Is there a simple, high-level way of doing this?

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.