Lz4rip: safe Rust LZ4 on par with C impl

announcements
1

lz4rip (repo) is a Rust LZ4 block and frame codec, derived from lz4_flex. Faster compression through aggressive skip acceleration and generational hash tables, with first-class dictionary support.

v0.2.0 just released. Perf improved on aarch64. Overall beating C LZ4 with safe Rust!

Perf: x86_64

summary_x86_64

Perf: aarch64

summary_aarch64

2

Hey, what's the third-party dep story? Are you dragging in a ton of external deps, or no? What's the testing story? Super cool. I've wanted to explore it for a long time but just haven't been able to find the time.

3

You can just look at Cargo.toml:

[dependencies]
twox-hash = { version = "2.0.0", default-features = false, features = ["xxhash32"], optional = true }

4

Haha. Obviously, but I didn’t have the time. I wanted to give him a chance to talk about it, too.

5

One optional dependency: twox-hash (xxHash32 only), for checksums in the frame format. Block codec has zero deps.

~100 tests (unit + integration + doctests), 10 fuzz targets under cargo-fuzz (round-trips, corrupt input, cross-validation against C lz4). Beyond that I do a lot of end-to-end perf testing in OMQ.rs, measuring compress+send+decompress latency over PUSH/PULL TCP with and without dictionary compression.

For inputs under 64 KB (the common case for message transports), the hash table is 4K×u16 = 8 KB, small enough to fit in L1d and leave room for application data alongside it. Trades some ratio for speed and makes the compressor practical to embed in a hot loop.

The sweep chart below shows the tradeoffs across input sizes. lz4rip is tuned to favor speed on small messages at the cost of some ratio on large inputs, and all the hot paths (compress, decompress, frame, sink) are #[forbid(unsafe_code)].

sweep_x86_64

And finally, the compression chart from OMQ.rs itself (it auto-trains a dict on the first 100 small messages and ships it to peers once):
omq_tokio_compression

To me, it's mind-blowing. I've been dreaming of ZMQ with a compression transport for at least 10 years.

6

This is really cool. I'm watching and testing today! Great job.

7

Thanks! Let me know how it goes.

Regarding OMQ.rs:
I've just released a new version. LZ4 dict auto-training is now opt-in. Simply because the increased CPU usage might be unexpected.