Popularity of crates.io categories over time

I ranked categories by the number of new crates published in each category per year, and plotted categories' rank on a chart, by year, since 2015 (click through for a readable SVG):

pop-thumbnail

(y axis is rank, x axis is time)

I'm using the most recent categorization of each crate and backdating it, so that newly-created categories with old crates appear to have a long history. Categorization uses lib.rs data, so data includes crates automatically assigned to categories (~100K crates).

Some observations:

  • cryptocurrency category gets by far most new crates published. This is mainly due to a pattern of cryptocurrency projects creating little brands and splitting them over 50 crates each, like foo-sdk, foo-crypto, foo-rpc-client, foo-rpc-server, foo-spec, foo-pow plus forks of every dependency they use like foo-sentry, foo-sentry-types, and so on.
  • there's lots of proc macros,
  • rust-patterns, data-structures, and algorithms were relatively more popular early on,
  • game dev seems to be growing,
  • but overall the chart turned out to be bigger and messier than I anticipated, so that will be a one-off visualization.
17 Likes

This one was to be expected.

I'm still looking forward to the day that the "proc macros need to live in their own crate" restriction is lifted :slight_smile:

All the same, thanks!

1 Like

There is no category for malware crates?

I don't know what I'm supposed to learn from this.

I worry about all those cryptocurrency projects, that area still sounds suspect to me. But I don't know what they do with it all.

I was curious if this would show whether Rust's focus is shifting over time, e.g. async or embedded taking over, but it looks like Rust is growing all over the place.

Indeed, Rust seems to be gaining traction in everything from embedded to web development.

I've always thought it was a bit off when I heard people introducing Rust as a "systems programming language" (what ever that means now a days) and going on to say it was suited to places where one would use C but not so much where one would use JS or Pythin.

As far as I can tell Rust despite Rust having features that make it a C replacement it is also a high level language that can cope with everything else nicely.

Perhaps I learn that my intuition about that was not wrong.

2 Likes

Why? Is it some variation of expectation of the second coming?

It's pretty obvious why that wouldn't happen in our lifetime: when you are cros-compiling procmacro have to be compiler for host platform, not target platform and the simplest way to achieve that is to move it to separate crate.

With Rust becoming more popular in embedded and mobile chances if it dropping cros-compilation are exactly zero.

The most realistic approach would be to disable support for proc macros entirely, but then it's not clear what one is supposed to replace them with.

The only niche where Rust doesn't work is “I want something working quick and don't care if there are lots of bugs” and with legislators finally noticing that sloppy programming is costing us trillions of dollars that niche would be shrinking pretty rapidly in the near future.

P.S. If you want to be amused then I recommend you to read ASF blog. The amusing parts is not with most likely outcome of these laws that are now in development. What's amusing is that various open source guys assumed these outcomes are untended consequence of wording of these acts when in reality these precise outcomes (move of a control over important open source projects from volunteers to corporations, etc) were parts of an explicit goal of what these laws were supposed to accomplish! That revelation puts all that activity that was happening for more than a year in a pretty funny position.

It's because of the impact on the architecture of libraries that incorporate a proc macro as part of their API: suddenly you don't have 1 crate, but at least a sandwich of library/proc macro/library. This can be done with a workspace, but is annoying to say the least.
It makes the proc macro a second class citizen, and that needs to be remedied.

While what you say is factually true, it is rather limited in imagination. In particular, it doesn't account for eg efforts to compile proc macros to WASM. With that, not only are proc macros isolated, they can be run pretty much anywhere.

The real issue is with whatever originally made it a necessity to require proc macros to live in their own crate. That still needs to be solved.

That's about as realistic as traits being dropped from the language. It's a stable, supported language feature, so that means it's not going anywhere.

1 Like

How would sqlx macros work in that world? They very much need access to host SQL server.

They are just DSO's that are added to the compiler to expand macors. Initially they were just plugins and had access to the compiler's AST, later they got simplified AST to ensure that they would have stable interface to the compiler internals.

But there was never any limitations as for what can they do when loaded. Which is both their incredible strength (wow: you macro may pull data from your Google Doc document during compilation time… we should immediately adopt that for velocity and flexibility!) and weakness.

Even if you would make it possible to compile and run portable WASM plugins it would take years before old-style anything-goes crates would stop being used (if that would ever happen).

That's about as realistic as traits being dropped from the language. It's a stable, supported language feature, so that means it's not going anywhere.

Yet you are proposing precisely that, ironically enough. Rust-on-WASM is not regular Rust, there are lots of limitations.

Proc_macro-on-WASM wouldn't be a proc_macro, anymore. You may try to introduce these and then [try to] slowly move people to use them, but adoption would be as swift as adoption of modules in C++ (as in: would take two decades of talks and then another two decades before they would be adopted… if ever).

Cool graph, but I don't quite understand it. What is the scale for the y axis? Since you say it is "number of new crates published in each category per year" having an actually numbered scale would be useful.

But I would have expected earlier years to be less overall then, is it scaled so that each year is 100%?

Maybe a better way to visualise this would be with a stacked graph of some sort (so that the whole graph is additive). This would also more clearly show how the total number of new crates varies for each year.

1 Like

The scale is relative ranking.

Unfortunately stacked graphs don't scale beyond a few categories, because noise adds up.

1 Like

Doesn't stop people (including cratesio for crate downloads) from using them, unfortunately. But if you're graphing proportions and everything always adds up to a flat sum, they're pretty decent at that even if individual category's slices of the pie chart move around.

We're at least partially a victim of overlapping naming here. Proc macros will always necessarily[1] be a separate crate (translation unit). What they absolutely don't need to be is a separate package (publication unit).

And it's worth pointing out that const evaluation and build.rs happens on the host and not the target. Splitting evaluation between what happens at build time in the host context and what happens at runtime in the target context even within a single crate/package isn't all that out there.

"Everything constexpr" may likely be decades away for Rust, but it's possible we'll live to see it.

With wasi+threads, we get surprisingly close to a fully capable POSIX environment. It's (deliberately) not simple to just "turn off" the isolation and you can't just assume link availability of host system library packages, but the functionality is available to implement what's left currently stubbed in std.

Though given the massive overhaul wasi is going through to develop and move to the component model, Rust probably won't move all that meaningfully forward until wasi-preview2 and the wasm component model become available.


  1. well... it would be possible for the compiler to extract and compile the "proc macro part" of a target separately a la const or cfg(test). ↩︎

3 Likes

The historical data for that is bad because (1) live malware doesn't self declare; (2) malware is deleted as soon as it's found so it would vanish off the graph even if you manually classified it like lib.rs does

2 Likes

Yes.

I was going to comment that too harsh a “cryptocurrency = malware” comment, as this answer kind-of implies, is probably overly simplistic, and practically I’m still doing that a bit with this comment, anyways, but actually thanks for that link! I’ve never read that “Magic Beans” category description @kornel seems to have put in place there :⁠-⁠D

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.