Hi,
I am looking for a Rust tool that makes data loading easy. Like dlt (https://dlthub.com) is doing for Python.
Something that allows for automatic pagination, retries, schema inference, incremental loading (only new or changed data) and such.
Something that makes the EL from ELT or ETL easy to write in Rust.
Cheers,
Peter
I'm not aware of any library that is exactly like dlt, but if you know your sources and destinations, I'm sure you can glue together the right libraries. I'm wondering why you want to do this in Rust though. Feels a bit like the wrong language for such a tool as you can't really use the strictness of Rust's type system to your advantage if you want to infer schemas and work with dynamic data. I don't find working with arrow data sets very pleasant in Rust, for example. Maybe split the extraction and loading parts from the transformation part and do the former in Python and the latter in Rust? Once you got nicely structured data in your destinations that you can easily map to Rust types, I feel like that's where Rust can really shine.
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.