Rust crate closest to Apache Spark?

zeroexcuses · September 17, 2020, 1:00am

What Rust crate (or group of crates) gets us closest to Apache Spark ?

The reason I am not giving up Spark directly is that I am unwilling to give up RAII and I really like Rust.

However, I need a way where I am programming an AWS cluster rather than programming a single machine.

What Rust crate (of group of crates) gets us closest to this ?

fosskers · September 17, 2020, 3:01pm

Consider perhaps Amadeus with Constellation.

jorgecarleitao · September 17, 2020, 4:21pm

There is also Ballista. It is built on top of the Arrow format, and thus leverages SIMD operations, memory alignment, and interoperability with other Arrow implementations.

More details here

XVilka · November 13, 2020, 5:38am

See also Rust equivalent of Apache Spark?

system · February 11, 2021, 5:38am

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.