Stdx updated for Rust 1.19

1

stdx is updated for Rust 1.19, which is scheduled to be released
Thursday.

stdx is a project to identify and validate a set of foundational
crates representing the best of the Rust ecosystem. It is currently
best viewed as a learning resource.

This is the most significant release of stdx in a while. It bumps all
crates to the latest versions, and adds num_cpus, threadpool,
walkdir, itertools, and fnv.

This release also finally gets the examples under test, so you can be
sure that the examples all at least compile, if not do something
useful.

Some notes about the future of stdx:

One of the goals of stdx is to do more extensive validation around this
core set of crates. To that end there are two projects in progress:
@alisha17 is working on extending stdx's own test suite to also run
the test suite of every crate in stdx; and @davidpeklak is
working on a stdx-check tool that will inspect any given crate and
determine whether it is "stdx compatible", basically meaning that if
it uses any particular stdx crate, it builds and tests against it, and
without pulling in duplicate revisions. With this I expect to set up
stdx's CI to automatically test stdx against some number of crates and
post a dashboard indicating ecosystem compatibility.

As far as stdx crate selection, I'm starting to feel like stdx is
approaching some level of completeness, with some notable exceptions.
Some of the next additions start to get into areas I'm less
knowledgable about so am interested in feedback.

As background, right now I consider the scope of stdx to be quite
limited. Today the README says:

The criteria for inclusion in stdx is conservative, and fuzzy. It's
mostly crates that are pretty super important, considering criteria
like portability, quality, conformance to conventions, documentation,
interoperability with other crates, reliability of maintainers,
de-facto adoption, historical context and precedent.

stdx is focused on core features, crates that are quintessentially
Rust and relied on by many Rust programs. It is intentionally
limited for the sake of simplicity and ease of comprehension.

And I'm still pretty happy with that limited scope for now.

When I look at the libz blitz crate working list, which is
what I mostly refer people to these days when discussing this subject,
a few areas stand out:

  • compression - stdx includes flate2, but nothing else. I have the
    impression that lzw, lz4, and xz2 are all likely candidates
    for a standard library, but don't have much confidence in the area
    myself, nor knowledge of these specific crates. Precedent from other
    standard libraries would help.

  • encoding - base64 vs data-encoding and hex - I already have an
    issue open to add encoding_rs, which I feel confident about, but I
    have less experience with these others. It looks to me like both
    base64 and data-encoding accomplish similar things, with
    data-encoding having a broader scope. Not sure what to do here. That
    'base64' name is real sweet and easy to find when what you want is
    the common base-64 encoding, but it seems questionable to include
    both. hex is a tiny crate but it does what it says. I guess I should just
    include it.

  • xml - Seems like something a standard library should include still.
    Is the xml crate the right one? Is it ready?

  • OS integration - nix, winapi, core-foundation. These
    serve an important function, but I don't know much about them. I
    know the history of core-foundation is as a pretty 'rough-and-ready'
    solution, and imagine it is not all that complete or polished. I
    have concerns that winapi has not been updated since mid-2016, and I
    don't understand the interrelationships of all the many related
    windows crates. It might be that these serve such an important function
    that they should just be included regardless of their state of
    development.

  • native builds - gcc, pkg-config, cmake. I guess my main
    concern here is just that these are build-time crates, and I need to
    think more about how to present them.

  • glob vs. globset - I just don't know which to pick. glob came
    out of std, but globset is by @burntsushi so is presumably used in
    a bunch of battle-tested places. I guess the latter is probably the
    right pick going forward but don't know.

  • mime, fs2, cfg-if. Just haven't decided about these yet. MIME
    is a common thing to have in a stdlib, fs2 has the fairly crucial
    cross-platform flocks, cfg-if is a really nice convenience that I've
    never used.

  • crypto. Not sure what do do here yet. native-tls has a strong role
    in the ecosystem today, but hyper is not directly represented in
    stdx and the two are often paired together. I don't feel confident
    picking between ring, sodium and the various pure-rust crypto
    crates.

  • html5ever, select, image - I'm inclined to think these are out
    of scope for stdx in its current incarnation, even though they are
    all good crates, because they serve somewhat specialized domains.

I've also had some doubts about replacing json with serde_json in
stdx (json is currently listed as an "alternative" crate). serde_json
does basically provide a superset of json, but it also pulls in serde,
so when you want json but not serde, json is a reasonable choice. I
guess with stdx including serde it's appropriate to pick the serdefied
json, but I feel like there is a good case for the json crate still.

Finally, I'm starting to get an itch to publish a working stdx
super-crate. The major blocker here is that macro reexport is not
stable, and is not going to be any time soon, so we need a
different solution, and it's probably going to have to be real ugly.

0.119.0-rc

Contributors: Alisha, Anna Liao, Brian Anderson, David Peklak, dowon
Cha, Isaac Grosof, plafü, Tshepang Lekhonkhobe

3 Likes
2

This morning I thought of two more things that might belong in stdx:

  • remove_dir_all. The std definition of this function has problems on windows.
  • A yet-unmade home_dir crate. Again, the std definition of this function is incorrect on windows.

So stdx is a place to paper-over incorrect code in std, at least until such time as std is fixed.

3

Can you clarify the problems with these two functions in the standard library? Surely we should have at least the documentation fixed in std to explain the issues?

4

Re json vs serde_json: After having worked with both crates I definitely in favor of serde_json, not because json is a bad crate but because you want to use serde with struct-deserializing in the majority of cases. It offers better guarantees at compile time and it even allows more concise code in many cases. In all other cases you can still fall back to serde's json enum which can do everything the json crate can. Pulling in serde as a dependency isn't an issue for me; Many other crates pull in dependencies that are not as thoroughly maintained and tested as serde.

1 Like
5

The link for core-foundation is broken, as it should be hyphenated, but the crate docs aren't on docs.rs anyways because it will only build on Mac.

6

winapi has been in development, I just haven't published a new version because I decided to completely reorganize everything which takes a while for a crate that has over 100,000 lines! Those hundreds of -sys crates are all going away as part of that reorganization so you'll only have to include winapi in stdx once I publish 0.3 finally. My hope is to get a release candidate ready this August and once I've tested that enough major crates are able to build against it, I will publish it.

5 Likes
7

@droundy `home_dir` incorrectly considers `HOME` environment variable on Windows · Issue #43321 · rust-lang/rust · GitHub and fs::remove_dir_all rarely succeeds for large directories on windows · Issue #29497 · rust-lang/rust · GitHub. home_dir incorrectly considers the HOME environment variable and remove_dir_all often fails because of transient windows file locking.

Thanks for the update @retep998.

8

I made a replacement for home_dir.