Reed-solomon-erasure : Reed-Solomon Erasure Coding

announcements
1

Hello everyone,

I'd like to announce reed-solomon-erasure 1.1.0.

Erasure coding is a form of forward error correction that allows you to reconstruct missing data pieces. Common use cases include backup storage, data transfer, or wherever data redundancy is desired.

I'd recommend reading BackBlaze's blog post on the topic if you're curious how it works.
If you're interested in how this library is used in practice, a full example is provided in the repo's README

Notes

The implementation closely follows BackBlaze's Java implementation, and the test suite mirrors Klaus Post's Go implementation.

Stability

This library was originally developed as an internal crate for a backup utility, thus stability and robustness have always been the goal.

Metrics wise, the current test suite contains 55 tests, and has code coverage of >99%.

Links

9 Likes
2

reed-solomon-erasure went through a major overhaul and is released as 2.0.0.

The new version is 4-5 times faster than the old versions, but still slower than Klaus's Go implementation due to not utilising SIMD operations.

SIMD operations will be supported for speedup in later releases.

3 Likes
3

Progress

I've added Nicolas Trangez's C SIMD files with minor modifications mainly to make everything to use unaligned memory access, the work is currently at the this branch : https://github.com/darrenldl/reed-solomon-erasure/tree/rewrite-using-klauspost-go , and will be released as 2.1.0-beta after some more tuning.

The benchmark on my laptop currently shows a 4-5x speedup compared to version 2.0.0,
clocking at around 4000-5000MB/s. Klaus's is still faster, roughly 2x the throughput (7000-8000MB/s), but this is still a pretty good speedup I'd think.

Request for help

I am looking for people to help benchmark and try on mainly x86_64 and ARM64 platforms.

If you would like to help, please clone this benchmark repo : https://github.com/darrenldl/rse-benchmark
(It contains the beta library already, so you only need to run cargo run --release in the folder)
And send the output, and also your CPU specs, preferably the output of lscpu or
cat /proc/cpuinfo, to me via messaging me here, via email, or any way you'd prefer and usable to me.

Thanks!

1 Like