Crate of the Week

How is pipeliner different than rayon?

I only tried rayon for a short time before giving up on it. So what I'm about to write might not be accurate.

The use-case was parallelizing network requests(client side), to avoid accumulating latencies. Those requests are not CPU-bound. So, requesting multiple parallel threads on a dual core device makes sense!

  • Rayon

  • Does not work with iterators directly.

  • It's not clear, at least from a cursory look at the docs, how to configure the number of threads used with par_iter().

  • Errors can contain hard to decipher unsatisfied trait bounds.

  • [minor] Wildcard/glob use required.

  • Pipeliner

  • Works with iterators directly.

  • You can pass .with_threads(num) anywhere in your iterator chain. You can even pass it multiple times with different numbers.

  • Errors are easy to decipher (your typical borrow checker errors. Unsatisfied 'static requirements).

  • [minor] No wildcard/glob use required.

In short, pipeliner may not have a fancy work-stealing algorithm implemented behind the scenes. But it provides more flexibility and easier integration with iterators.

3 Likes

Cargo-fuzz. Command line wrapper for using libFuzzer.

5 Likes

PumpkinDB. PumpkinDB is a compact event sourcing database, featuring fast on-disk storage, flexible approach to event structure and encoding, sophisticated event indexing and querying.

1 Like

μtest. Unit testing for microcontrollers and other no_std system.

5 Likes

I'd like to nominate @BurntSushi's FST crate .
This provides a very compact in-memory representation of maps, in the same ballpark as gzip, but queryable!
The compact representation makes analyses possible that would otherwise explode the RAM of even the biggest compute nodes.

In Andrew's typical style, the crate, the theory behind it, and the design trade-offs are extremely well documented via blogpost. After reading it, I felt like I had learned something new about theoretical computer science, and had a good feeling for when to use the crate.

My department is currently using the crate to analyse genome regulation, by identifying the counts of all possible 16-mers in the human genome, so 4^16 = 4.294.967.296 keys. Our algorithm was impossibly slow before we encountered FST, producing hundreds of gigabytes of temp-files that had to be processed line-by-line.
Thanks to FST we can do unbiased analysis into what makes humans tick: how our genomes are regulated.

I'll add that we're using fst via it's python bindings, by Johannes Baiter. Johannes was supportive and quickly responded to our questions. (He works at the digitisation centre of Bavarian State Library, so I'm very curious what use-case he found for fst there.)
I include this tidbit to demonstrate that there is already a (minute) user-community around fst, and that it is (apparently) broadly applicable.

All in all: fst is a crate that, through its great implementation, excellent documentation and great API, makes high-performance data structures available to a new audience, enabling real research and bringing humanity forward.

:heart: Andrew Gallant / @BurntSushi

(p.s. also a little :heart: for @ehiggs, who introduced me to it)

19 Likes

duct - "A cross-platform library for running child processes and building pipelines. duct wants to make shelling out in Rust as easy and flexible as it is in Bash. It takes care of gotchas and inconsistencies in the way different platforms shell out."

8 Likes

Serenity - "Serenity is a Rust library for the Discord API. It enables you to create awesome bots for the Discord chat service in an easy and organized way."

3 Likes

pretty_assertions - "Overwrite assert_eq! with a drop-in replacement, adding a colorful diff."

13 Likes

Congratulations to pretty_assertions for making it into TWiR 175!

As a happy user of fst without any further involvement, I hope I'm not violating etiquette by re-nominating fst.
Please tell me if I'm overstepping!

I nominated fst a few hours after TWiR 174 came out, because 174 didn't have any crate "for lack of nominations". I can understand my suboptimal timing may have caused it to accidentally not be considered for 175.
I'd say 12 collected likes (at time of writing) is enough show of support to give it a rematch for 176.

8 Likes

I've used fst and it's great.

1 Like

Hi guys,
I would like to nominate Official Aerospike Client for Rust as the Crate Of The Week.

3 Likes

https://github.com/brson/rust-skeptic

Saw this the other day, tests for your README.

3 Likes

Hello! I saw this in my RSS reader today:

Sadly, for lack of nominations we have no Crate of this Week.

Therefore, I'd like to nominate trace-error for a crate of the week. In a nutshell, it exports try!() macro replacements that thread error tracebacks behind the scenes. In author's words:

more lightweight and less macro-based alternative to error_chain

2 Likes

Hello. I've been working on a binary crate to generically decode protobuf messages:

I'd like to submit it for crate of the week consideration.

-Sevag

2 Likes

stdweb - A standard library for the client-side Web.

5 Likes

indicatif: "A rust library for indicating progress in command line applications to users."

10 Likes

Hi,

I would like to nominate PX8. It is an Open Source Fantasy Console !

You can create tiny game in full Rust, or use Python/Lua. You have a built-in editor where you can modify your sprites.

Thank you

3 Likes

I nominate remove_dir_all by @XAMPPRocky.

You know that remove_dir_all function in std? Yeah, it basically doesn't work on windows, because files with open handles can't be deleted, and there are plenty of situations where this happens unexpectedly.

The remove_dir_all crate tries hard to really delete directories on windows. It is used now by the tempdir crate, and similar code is open-coded within rustup and rust-installer.

12 Likes

Hi!
I would like to nominate oath2 - library for 2 factor authentication. It has all 3 OATH protocols implemented: HOTP, TOTP and OCRA.
Edit: author of the original lib oath has merged all changes from ocra2. Now ocra2 is abandoned.

4 Likes