Where is the libraries team?

I guess it would make sense to put this question into context.

I came across Rust while looking at TechEmpower benchmarks, a year or so ago - I believe. Back then it was crushing the charts, being far beyond almost anything else. It picked my attention, and I decided to inquire a bit further about what it was and how it worked. How the hell did it get there, with such a leap beyond everything else?

I've begun to study the book and the answer soon became quite apparent. Rust managed to combine the highest level FP concepts with C-level performance, "zero-cost abstractions", all the while enforcing the type safety and making sure that the guy or gal that was writing his or her software wasn't writing anything that could blow up in his/her face without any prior notice.

This wasn't simply "cool". This made sense, on a deeper level. It meant that people could start focusing on doing the hard work beforehand to avoid the debugging nightmares later. This meant that the programs written could have C/C++ - like performance, all the while having the necessary level of abstractions above to hide all the unnecessary (for most folks) complexity underneath. Fast, performant, efficient, safe, productive, reliable - with, arguably, just one caveat: compilation speed. And a steep learning curve - but pretty much anything is hard at the beginning, so it's not much of an argument against a language.

I kept coming back to it again and again - and I still do. I love the fact that I'm able to squeeze my CPU to the limit without wasting unnecessary resources, something which interpreted languages force you to do - and yet I'm able to enjoy the creation of my masterpiece without the anxiety that comes with languages like C++, due to their very nature. And yet, I continue to forget about it just as often.

I have to use other languages, not only because there's no real alternative to some languages - like HTML/CSS/JS on client side, but because despite the Rust's ability to abstract things away, no one seems to care. Saving up precious CPU cycles, which boil down to additional cost for companies, running their back-end processes, seems to be of a big deal only to a select few employees once a while, who might choose to replace some of their Go codebase with Rust. Django-, Laravel- like frameworks? A dedicated part of the community, who could be working on all the tools, used on a daily basis in production environments? - non-existent. Everyone does their own thing. Other guys might come in and use Rust as part of their own project, but no one inside the language community itself seems to give a crap. Which feels just a little bit off.

I've taken a look at the Governance Board - unsurprisingly, the least of teams include all kind of wonderful people, which are doing a beautiful job, but there's one team that is missing - and the fact that is missing can be sensed a mile away, not just by the insiders, but by most of the people who take Rust into any kind of serious consideration whenever they're facing a decision regarding the right tools to use for their next project. Libraries Team.

A group of people, who would work in close proximity to both the current community of Rust, as well as all the possible adopters that would come flushing in if only they knew in advance which kind of tools they could rely on when starting their next project in Rust. A team, that would oversee and prioritize the development of various libraries, depending on what we currently lack - to make sure the whole community thrives and prospers, both in terms of the tools available for any given important task, as well as for establishing Rust as a new standard among programming languages - on the job market and beyond. It has all the potential to replace not only C and C++, but Node / Deno, PHP and RoR. This is not going to happen, however, until a team of dedicated professionals makes this a priority.

"Awesome Rust" is not going to do it. A whole web site dedicated to as to "Why You Shouldn't Use Rust in Production" is not going to help. There are, obviously, different fields, each with its own challenges - Game Developers are not going to be interested in the same things that Back-End Developers care about, and those are not going to be concerned much about the development of AI-related tools and libraries. But unless we come together and make a conscious effort not only to develop the best tool possible, as a standalone language, but to use this tool to create other tools, in the forms of packages, libraries and frameworks, which could then be used by ever greater and greater amounts of people, we might simply get stuck, for a long period of time, in the niche of embedded, too-low-level-to-be-used-in-anything-practical programming, with a whole array of attempts to make viable tools, with none of them ever becoming anything other than hobby projects.

Yes, I'm biased towards web development myself. I strongly believe there is a good reason why Actix Web made it to the top of the charts in that benchmark, and although its author did a terrific job, the language itself contributed a ton to it. Forcing people into contributing to projects they don't care about is obviously a dumb idea - but I rarely read about any concrete, focused plans or even priorities from the community as a whole. Every Ruster (pardon me, I like the term better than "Rustacean") enjoys the language, but rarely seems to notice the struggles of people nearby, nevermind anyone else, who could be thrilled to adopt an efficient, productive, safe tool for almost any job, should there be any viable alternative for what they're using right now.

Establishing the upmost priorities for different categories of libraries - WebDev, GameDev, AI, CLI, GUI - and regularly reminding people about them, in "This Week in Rust", promoting the most promising libraries, actively asking for contributions, and actively contributing to the existent projects, instead of trying to rewrite each and every time a "yet another thing" - could help tremendously.

Forming a "Libraries team", dedicated to all of the above, could be a great start.

Thoughts, opinions, comments?

P.S. The "Future Team" could be a bit more appropriate of a name - there's already a "Library team", with a slightly more sparse and diverse range of responsibilities, than would be necessary to create a somewhat more directed and unified structure within the community.

2 Likes

I think...

The English language, or whatever human language you like to think of, somehow defines a lexicon, a syntax and a semantics. It has a grammar and rules about how compose words into sentences, paragraphs and so on. It likely has dictionaries to define/describe what words mean. And so on.

To some extent such human languages have "standard libraries". Chaucer, Shakespeare, and so on in English.

But really, the important things of English, or whatever language, is that empowers those that know it to communicate with others, to create and share their own works. Which far outnumber the standard works of the language.

It is impossible for any English, or whatever language, guardians to shepherd, approve, authorize, all of the creations of the language speakers.

And so it is with programming languages. Especially Rust, which as far as I can tell has made a conscious decision to keep it's standard library as small as possible. Whilst empowering Rust users to create what they perceive is missing.

Yes that sounds chaotic, and so it is. But over time masterpieces of useful works will arise and become "classics" of the language.

How will this pan out? Will it result in the miserable offerings of the C++ standard library? Will it result in the gigantic Java library world?

I don't care, there is more than enough of what I can imagine I need in the crates.io ecosystem already. With more coming everyday. How fabulous is that!

5 Likes

You can't just take a bunch of people and tell them "write big mature libraries in Rust, now!"

These big important projects grow out of a need. If you need RoR-like project, you will have to start it and drive it. If you want others to write it for you, you will have to hire them and pay them, or find people interested and motivated enough to do it for free.

14 Likes

I do think you have an important point regarding language adoption here: ecosystem maturity.
Surely, that is an adoption barrier for any new language and different languages. For languages created by large corporations (Go, Swift, whatever#) they can just put loads of money behind it and solve the problem more quickly.
For languages on a more open community, the "path to glory" may involve quite more bumps as getting involvement from people without a pay is definitely a challenge.
Perhaps, when Rust becomes an independent foundation and if they get some funding for dedicated development this can be suggested as a project the foundation should support and fund.

One thing I did not get from your post though is why the existing libraries team do not fit the picture?
I do think Rust should move a few more things into standard library (like rand, log or perhaps serde) because only by seeing something on a standard library one can know there will be avoidance of changing APIs
But in terms of open-source contribution it seems to be doing quite ok.

1 Like

This is not about ordering people to write big and mature libraries - it's about making it a priority, alongside the rest of the goals which the community has. We've been focusing on making the async part of the language solid and beautiful - and arguably, we're already there. The same thing can happen with libraries, frameworks, additional tools and utilities.

All of these projects do grow out of need, I'm with you on that. If I had 10+ years of experience in the field, managing and directing projects of this magnitude, along with proficiency in Rust itself - I would start it and drive it myself. I don't want others to write what I want for me - I just want to see the community a bit more engaged with itself, and not just how cool the language is. It is awesome. It's the most loved lang on the Stack Overflow for a reason - but why are we focusing so much on the language and not that much on the ecosystem around it?

Scratch the "Libraries" or the "Future" team term as well. Make it "Eco Squad". A group of people that would monitor the needs of people in different fields and draw the attention of others towards the most perspective and ambitious projects, instead of just hoping for them to flourish on their own.

Is it that unimportant?

There have been some attempts at this in the past, but they haven't had a big impact and it's been hard to keep them going. Perhaps it would be useful to give an example of such a team from another programming community, so we can see how they go about their work, and learn from their experience.

Currently, in the Rust community, this sort of work is mostly being done by domain-specific Working Groups. For example, from the Game Development WG charter, the first listed goal is:

Gather a list of all the main gaps that separate us from the ideal state described in the introduction. Could be tooling gaps, a language feature gap, a library feature (eg. custom allocators).

Some of them, like the Embedded Working Group, have already been involved in the creation and maintenance of a large number of new libraries!

9 Likes

Exactly my point, thank you.

I didn't mean that the existing team doesn't fit the picture at all - but from being a (albeit, quite a small and insignificant, compared to some) part of this community for a while, I couldn't notice a somewhat strange mix of a strong dissatisfaction with the current maturity of Rust's ecosystem and quite a noticeable, somewhat irresponsible, throw-your-hands in the air attitude from most of the people (including myself), when it comes to actually do anything about it. "Hey guys, our libraries suck. Let's hope it'll change soon." The idea behind my post was to put a bit of more attention towards it first - the effort, the energy, the community and the projects are going to follow afterwards.

There are simple, trivial things that can be done to encourage people to get involved in projects that matter - and not just their hobby creations. Creating badges, in this community, for developers of "Actix Web" or "Rocket". Mapping some "testing days", when the whole community could dedicate themselves to testing and reporting the bugs for - frameworks, libraries, tools, utilities - which the whole community recognizes as important. This is what I have in mind.

The standard library is a beast of its own. Having a team that works both on SL and the rest of the ecosystem is not enough if we want to develop the latter properly.

Precisely what I mean, thank you.

We can create polls to identity what the ideal state is - then break it down into smaller parts, identifying what exactly we're lacking, on the Back-End side, Game-Dev side, AI side, and so on.

Once that is done, we can take a look at the existing libraries and see which of them are the most promising of all - and encouraging people to contribute to them, via code for the willing, via tests for the rest. In the domains, where we have no libraries at all - it would help to encourage people to create completely new ones, and continue to encourage them as their creation starts to flourish.

We can even start by creating a tree structure of the whole ecosystem of ours:

  • Rust
    • Game-Dev
      • Shaders
      • 3D
      • ...
    • Web-Dev
      • Back-End
        • Server
        • Framework
        • Authentication
        • ...
      • Front-End
        • ...

Breaking it down into smallest possible fragments, and then monitoring the progress of the best - and the worst - tools, libraries, frameworks, projects. Encouraging adoption, competitiveness as necessary, involving community to help test different projects out, clearing pointing out the areas where there's a hole that needs to be taken care of.

It doesn't look to me like people don't care about all the environment that our language creates. If that is the case, we've got a much bigger issue on our hand - as it would mean that Rust has established itself quite solidly as a hobbyist language of sorts, but nothing more serious. It doesn't seem to be the case. People want to contribute - but most of them have no clue where to start. And unless we create a structure to understand our weak points ourselves first, we won't be able to work on them.

That seems like a very pessimistic view.

In my year of Rust I have not noticed "a strong dissatisfaction with the current maturity of Rust's ecosystem" expressed here. Seems people understand that Rust is quite new and all the luxuries they are used to from other language ecosystems have not arrived yet.

I also don't see an " irresponsible, throw-your-hands in the air attitude from most of the people (including myself), when it comes to actually do anything about it. "Hey guys, our libraries suck. Let's hope it'll change soon.". In fact I'm amazed what efforts people are putting in.

Personally I am again amazed at what already exists. Web server crates, check. Web socket server crates, check. NATS client crates, check, Database access, check. And a lot more besides. All the way down to support for micro-controllers. Amazing!

I'm more optimistic. I think smart programmers will be attracted by a good thing that makes their lives easier and builds much more robust and much better performing software than they are used to. Being smart, if there is something missing they need, they will build it.

Which will apply to organizations/corporations that can see what is good for them as much a individual hobbyist programmers.

2 Likes

As far as I can tell this is not true.

Apart from the fact that our new little company has bet the farm on Rust everyday I read of big boys getting serious about it as well. For example:

https://blog.cloudflare.com/tag/rust/

And many more. These are not trivial things.

2 Likes

There's no pessimism here, and I'm not making fun of the efforts the people are putting into it, I'm simply paying attention to the stats:

There's not going to be better training and documentation until there is a decent ecosystem in place.

Companies are not going to adopt Rust until they'll be able to rely on existing bits and pieces, and not being forced to reinvent the wheel. The second column can also be attributed to lack of sufficienty abstracted material, which new users can rely on, without having to dive as deep as Rust allows you to dive. The third column is self-explanatory. And if the fourth column isn't related to the fact that our eco-system is not properly developed yet, I don't what it can be related to.

Once again, I'm not bashing the existing efforts and I'm not saying what we're currently doing is wrong - what I'm saying is we can speed things up tremendously if we start paying attention not just to how beautiful Rust is, but how to create the right kind of environment around it to encourage adoption, penetrate the job market, and make Rust a proper standard, and not a yet another option.

I've never said it was true:

It doesn't look to me like people don't care about all the environment that our language creates. If that is the case, we've got a much bigger issue on our hand - as it would mean that Rust has established itself quite solidly as a hobbyist language of sorts, but nothing more serious. It doesn't seem to be the case.

I'm simply pointing out the fact that our efforts could bear a bit (a lot?) more fruit if we focus them in just a slightly more different direction - not just enjoying the language, but actively contributing to its ecosystem.

Your company made a wonderful choice - but how many other companies in your position choose to do the same thing you guys did? You know the answer better than me.

serde, serde_json, anyhow, thiserror, log, glob, regex, csv, libc, backtrace, postgres, openssl, native-tls, cxx, proc-macro2, quote, syn, hashbrown, parking_lot, bitflags, getopts, env_logger, lazy_static and many more are all (I believe) maintained by members of the Rust library team.

While many of us enjoy the language, it's pretty clear that we are also contributing to the ecosystem. All of the crates above are pretty core to a lot of common programming tasks. Many of them are things you'll find in the standard libraries of other languages.

10 Likes

I don't know any answer better than you. Perhaps I interpret what news I read about Rust more optimistically.

I would not doubt the stats you present in that bar chart. As much as we can rely on any such polls.

All I am saying is that it early days still for the Rust ecosystem. That the pull of Rust to attract more people to use and contribute to it is unstoppable.

Thank you - personally and to all of the members. Without you Rust wouldn't be where it is today.

However:

and there are a lot of crates which need to be much higher than core in order to accelerate the adoption rate. Yes, more of them will appear as times goes on. Maybe. What I'm suggesting is merely to make a conscious effort to model other languages and their ecosystems, in order to speed up Rust's growth.

Rust itself is very core, but it appears to want to remain very core - this is what concerns me and this was the reason behind my post. It wasn't the desire to tell you that you're doing a terrible job, as I believe the exact opposite.

And I'm not saying that I know any better. It's not about optimism/pessimism - not for me, at least - it's about what can be done to improve the situation, ideally - dramatically, and as soon as possible.

I agree - we're in the early days. I'd simply love to get those early days to mature and wonderful days a bit quicker than it would take otherwise. Perhaps it's too audicious of a wish, perhaps not.

A couple of guys here seemed to have expressed the same concerns as me, though.

So it just might be that it's not just my obsessively-compulsive idea.

Hang on. That's not what you said. What you said is "not just enjoying the language, but actively contributing to its ecosystem," which is clearly a pretty unfair characterization of the status quo. And that's what I was responding to.

Literally nobody is going to disagree with building more stuff. You either need to wait for people to do it naturally, pay people to do it or start organizing it yourself. Complaining about it here while implying that existing contributors are just diddling around doing hobbyist stuff isn't going to accomplish anything good.

9 Likes

Rust's learning curve is widely known. Even if one doesn't find it intimidating, it is still a significant time investment that one may not wish to make right now. Same reason I never ended up switching to vim or emacs or spacemacs or whatever; what I have now works well enough and allows me to write the code I want today. I don't personally see the option as having anything to do with the ecosystem.

1 Like

Perhaps your reason was not so clear.

As far as I understand it a conscious of the Rust devs to keep Rust's standard, "core", library as small as possible.

I personally think this is a wise decision.

As far as I'm concerned, with cargo and the crates system the line between "core" libraries and any other is blurred. Makes no difference, I use the crate, it works, I'm happy. This is very different from the experience in say C++ or Java.

The crates system adds much needed flexibility to all this. It would be so easy to rush in and standardize a bunch of things. Only to find later that they are not the best solution and other options have been created. Too late, once it is welded into a "core" part of the language it is there forver.

Agreed, my bad - should have phrased that better.

By "actively contributing to its ecosystem" I was implying that it might be a good idea to focus a bit of our efforts, as a community, to develop somewhat more higher-level and abstracted-away tools, packages, libraries and projects - Rust is a king of low-level programming, but its environment remains fairly crude for most other things, therefore - my argument was - we could improve the stance of the language significantly if we were to shift our focus - just a little bit, perhaps, for starters - from lower to higher level.

I wasn't discrediting your work by any means, once again - but I do believe we have a lot of contributors focusing on lower-level things, and not enough of them contributing to higher-level projects. Doesn't mean that those who code away abstract stuff are in any way, shape or form better, than those who focus their efforts on more bare-bones material, but if we want to accelerate our growth of adoption, penetration and everything that I've outlined previously, I strongly believe that it would make quite a lot of sense to start attracting attention of more people towards the fact that we might really all benefit from higher-level projects, supported by the whole community, and not just by single individuals.