Rust beginner notes & questions

peter_bertok · July 14, 2018, 7:01am

Conversely, my argument is that a rich set of blessed libraries is the standard library. Externalising bits and pieces into a package manager just makes it hard to see where the officially anointed libraries end and the heathen filth begins. 8)

I’m having exactly this experience with C# right now. For example: NuGet, just like cargo.io, fails to properly include the full set of “dependency” flags actually required in complex (real!) scenarios. Nice, easy versioning of “3.0”, “3.5”, “4.0”, etc… has gone out the window, and what I’m left with is a mess of incompatibilities that I shouldn’t have to deal with.

In my particular case, something like this* happened:

I’m writing a C# PowerShell module, which is basically just a DLL library. This part is easy.
This library has a NuGet dependency that in turn has a transitive dependency on a .NET Core module that is also a NuGet package. This is generally not optional now, because the dotnet core standard library is modular. So importing a dependency like System.Data.SqlClient will import System.Data behind the scenes. They’re independently versioned, just like cargo packages.
The linkage is dynamic and done at runtime, so neither NuGet nor the dotnet compiler have any idea what this dependency is exactly, because the pwsh runtime pulls in some specific version of the standard library modules.To work around the issues this causes, there is a wonderfully opaque set of compatibility shims all over the place.
If I stay on version 4.0 of the package I’m using, it doesn’t work, because of a bug in that version of the library.
If I “upgrade” to 4.5 everything breaks on Linux because some legacy compatibility shim was removed. This shim is still required by pwsh on Linux, but not Windows. I have no control over this, it’s a component of dotnet core pulled in by the pwsh executable, not me, but it’s incompatible with my DLL.

None of this is documented, reflected in NuGet.org, managed by nuget.exe, or in any reasonable way discoverable. This is 100% code written by one organisation, Microsoft. Half of it was written by one team (pwsh) and half it by just one other team (dotnet core). They’re likely working in the same building and yet there are already entire categories of scenarios where everything just blows up in my face unpredictably and there is nothing I can meaningfully do on my end to fix it. I literally tried every combination of csproj settings through brute force to see if it could be made to work. It can’t.

I did a lot of research, and I worked out that basically the pwsh team had likely never sat down to write a “third-party” cmdlet module in C# that uses a NuGet dependency. They’ve written a bunch of C# modules, but it’s all part of pwsh, and hence versioned in sync with it, so they’ve just never been exposed to the mess that they’ve created for everyone else, the 99.9% of the developers who will actually have to use this stuff.

I just have a feeling (perhaps unjustified), that a lot of Rust development is similar. It works in a controlled environment, but I doubt it can handle the combinatorial explosion that is already making NPM a nightmare for developers. The more popular cargo gets, the worse it will get.

Let me paint you two scenarios in analogy with my pwsh experience:

Scenario 1, the Servo team (or similar):

A medium-to-large team with a long-term project. Timelines measured in years.
A lot of overlap with the Rust core language team and the developers of the key crates.io packages that make up the “semi-standard” library of blessed modules. These guys likely often meet in person, work in the same building, or correspond on the internet on a regular basis. There is trust built up over years.
Dependencies change slowly, and they have months to patch up any small inconsistencies.
They directly control most of their dependencies, including transitive dependencies. The packages were written for Servo, or by a Servo team member, or someone in close collaboration. Something like 75-90% of the code is under their “control”. They know exactly what they’re importing and where it comes from.
The code is open source. It’s not “sold”, and even if it is, there’s a disclaimer that says that they’re not liable for anything. There’s no warranty.

For scenarios like above, I am in no way disputing that the Rust development environment “just works”. It would be a big step up from C++, provide a lot of flexibility, and generally enhance productivity. This is great. Game developers would be in a similar boat, and I’m sure there’s many more examples.

Scenario 2, an enterprise tool:

Lone developer or small team, some of whom… are not great developers. You can’t control everything, and the people you work with make mistakes or are just a bit sloppy. Maybe good, but overworked.
The goal is to plumb together a bunch of libraries. Feed XML containing JSON into a database. Make it authenticate with LDAP and SAML. Talk to a legacy Java app. Import from a mainframe. Etc…
Your dependencies are published by corporations, not “Bob” down the corridor. You have zero control over these packages. Pull requests are silently ignored, assuming it’s even open-source to begin with.
You’re stuck on an old compiler because of the above.
Timelines measured in weeks. If something breaks, you are screwed. Deadlines make wooshing noises and emergency meetings are scheduled to recur daily by project managers who don’t care about “package incompatibilities” and other meaningless technical talk.
You can’t spend significant time researching the pedigree of every dependency, there’s hundreds of transitive modules being pulled in, looking through them all would eat up your entire dev time budget before you wrote a single line of code.
Even if you miraculously do check everything, maintenance is done by a different team on a different continent. They’ll blindly pull in the latest “updates”. You have no control over this either, management six levels above you signed an outsourcing contract for BAU support.
This is going to be processing sensitive data worth millions. If it’s insecure or crashes, your managers will avoid all responsibility and blame you. You’re lucky if you’re only fired. You’ll likely avoid jail, but lawyers will probably get involved.

Now, in this scenario, Rust and Cargo are… not great. If this happened in scenario #2, that poor solo enterprise developer would not be a happy person:

A dev in scenario #1 is much less likely to be affected, and could just laugh something like this off. A dev in scenario #2 would just avoid Rust if he’s got any brains. I certainly would not use it as it stands, because there’s virtually zero protection from the kind of vulnerabilities that were not just predictable, but predicted, and have occurred for real. Why would I risk it? For what? Twice the runtime performance? Pfft… I could just request a server 2x as big and not risk my job and my career.

So imagine if I was a bad actor like that guy who put malware into the eslint NPM package. Just pretend that I’ve been contributing to cargo.io under a pseudonym (real names and code signing not required, remember!). There’s a popular package that I’ve uploaded years ago with a cutesy name that lots of people use.

I’m going to inject malware into it next time I fix a critical bug. This malware will steal github.org and cargo.io credentials, which I will use to further distribute malware into as many more packages as I can get my hands on.

Now what? What are you going to do about it?

The clock is ticking. Seriously. In a week or so, one of your dependencies is turning evil. You don’t know which one. You need to update it along with a bunch of others. Tick… tock… tick… tock…

*) I have no idea what’s going on exactly, the entire thing is a ludicrously complex black box as far as I’m concerned. Other people have reported the same issue, and it’s still open. Nobody from the dotnet core team has any clue what to do.

juleskers · July 14, 2018, 11:44am

I’m really starting to like this thread, the tone has improved, I’ve gotten my breathing under control again, and there are lots of deep, insightful discussions

(I’m on mobile, so I’ll keep it brief-ish)

Utf-8 Vs utf-16: all of the systems that have chosen utf-16 has regretted this. It seemed like a good idea at the time, because it used to be a fixed-length encoding. Now that it isn’t anymore, utf-8 is better in all respects.
I admire the guts windows and java showed in adopting unicode early, but history has not been kind to their “first-mover” status.
I am glad that rust chose the future-proof path with utf-8, even though this means shimming and OsStr… and thanks to the good, generic From/Into shim traits infrastructure, that shimming will remain as simple as possible, within the limits of the existing, divergent platforms/environments

Trustworthy packaging is definitely a concern, but the infrastructure surrounding cargo is very future-proof.
For example, the entire crates.io index is a git-repository, meaning that it can be cryptographically verified. Signing this is actively being discussed (link to follow).

2-factor login for crate authors is possible if you chose to authenticate via GitHub, and there is an ongoing internals discussion on making 2FA required for publishing. (Inspired by the NPM eslint attack)

Crate signing would definitely be cool, and is only waiting for someone to put in the hours to implement it. Thanks is to the index being a git repo, the signing public key can even be distributed securely.
The crates team is strongly aware of their responsibility, and has been discussing alternatives like the cryptographically secure “TUF” since the get-go.
Their security-mindedness inspires a lot more confidence in me personally than the ad-hoc-ish track record of NPM (and with NPM community board member Ashley Dubs being a prominent part of the Rust team, you can bet that the NPM lessons are not lost on Rust)

Crate discoverability is indeed not yet optimal, and this is known in the community and in the team. See for example the current experiment with crates.rs (Todo:link)

Searching and ranking is being investigated, it since it’s mostly a social problem, not a technical one, we are talking “harder-than-NP-hard hard”…

stevensonmt · July 14, 2018, 3:37pm

With regard to the crate discoverability complaint, crates.rs (announced on this forum here) makes a good attempt at improving this in a highly performant way.

Adding a maintenance quality metric to the ranking algorithm would probably address the only complaint about the crate ecosystem that it does not already address. Something like checking that it compiles with latest stable compiler first. If it compiles then it is given a maintenance score of 1.0. If it does not compile with latest stable, score should be reduced according to date of last version update – 0.9 if the last version was a month before the last compiler update, for instance, but 0.1 if the last version was two years ago. Use this score as a multiplier for the current “popularity” index.

peter_bertok · July 15, 2018, 3:36am

I know I can sound… harsh, but it’s “tough love”. I like the Rust project and I want to see it succeed, because C++ is long overdue to have real competition.

I think what a lot of open source projects are missing is someone loudly saying “No!”. Too many people patting each other on the back for a “job well done” when the job isn’t done leads to very happy people pleased to produce something that ultimately is a failure when faced with reality.

For an example of how open source can work, look at Linux. In my humble opinion, the single most important thing that makes Linux successful is Linus Torvalds. He’s brutally harsh in his feedback, infamously so. People constantly criticise his language, when he is simply stating facts. Often these are uncomfortable facts people don’t like to hear, but must hear to have any chance at success. Some bars are set very high, such as “entry into the kernel”, but people are sloppy and lazy. Linus points out that this is unacceptable. People don’t like it, but that’s not the point, because the computer doesn’t care about you feel about the quality of your code.

The rust-lang forum helpfully suggested some related topics, which shows that I’m certainly not the only one with such issues:

To give an example, on Saturday I tried to read and write from a TcpStream using a buffered reader and buffered writer. Using the examples in the docs this was impossible because once the writer was constructed the reader could no longer use the stream.

This kind of “difficult to assemble layers of stream processing” commentary ended up being the bulk of this topic…

And, a topic created by yourself:

I’m trying to learn Tokio and finding it too complicated. The problems for me are:

The sheer number of types involved.

Choice of names for the types is such that they don’t indicate the intent clearly.

Also at a coarser level, many things seem non-intuitive. For example, why is a ServerProto, which is meant to represent a protocol, creating a transport? There are many such design choices which are non-intuitive.

Also a very similar point.

I see a bunch of people are politely saying the same thing over and over.

Has it worked?

Rust is, what, 8 years old now, and the 1.0 release was 3 years ago! Yet it is still obscenely difficult to do really basic things with the language, like read and write a socket. Please excuse my language when I say: “What-the-f***!?”

HadrienG · July 15, 2018, 4:48am

The main problem with brilliant jerks such as Linux Torvalds is that they are not composable.

Put one of them in a room with a polite person, and they will quickly exhaust that person until she leaves.

Put two of them in a room, and hell breaks loose as they start yelling at each other over the most trivial matters.

Ultimately, having a brilliant jerk around greatly limits your ability to assemble significant amounts of manpower, which is the name of the game in software.

Complaining, whether politely or impolitely, is ultimately a very ineffective way of moving things forward in a software community. If you really want something done, the most effective way is almost always to go and do it yourself.

Time is too precious a resource to expend sizeable amounts of it in Internet arguments.

steveklabnik · July 15, 2018, 4:21pm

The Rust project specifically rejects the idea that you must be mean in order to give feedback. It’s not only a moral thing; it ends up distracting from the actual feedback, making it harder to sort out what’s vitriol and what can actually be done to improve things.

See this very thread. The extra arguing has made many, very offtopic comments, rather than actually figuring out what could be done to improve things. It took a while to get back into the topic.

Criticism: good. Being a jerk while giving it? Not good.

juleskers · July 15, 2018, 7:38pm

As Hadrien and Steve have already pointed out, we are trying emphatically to avoid having “loud” dictators in the Rust World.

Obnoxiousness alienates, and not even a truly brilliant, won’t be seen for another thousand-years genius, can outcompete a dozen people of “only normally smart” level, but who are working together at a normal, pleasant blood pressure level.

Tokio composability has been the singular focus of the hugely-breaking 0.2 rewrite, with gigantic usability improvements.
Another equally big leap forward is to be expected after impl Trait.
So obviously"people politely repeating requests" has done exactly what was required, with a pleasant experience for the implementors, who will stick around because of it, and still enjoy improving their work, rather than taking their ball and going home because their first API design (in a revolutionary, never-seen-before ownership-based environment, where no established wisdom exists) was “criticised” with slurs. (Edit: deleted uncalled for criticism of other communities by name, thank you for calling me on it @burntsushi)

There is a reason I am a rust-enthusiast, and the excellent examples of good behaviour (unfortunately not my own…), are a large part of it.

BurntSushi · July 15, 2018, 8:29pm

@juleskers I’m strongly in favor encouraging others to work together respectfully, but let’s do it on our own terms and avoid unstructured critique of communities/projects/individuals that work differently than us. Not only is it uncouth, but it’s just going to inspire more off topic debate instead of focusing on what’s important: in Rust spaces, we’re kind and respectful to each other.

I’m hopeful that further discussion in this thread can be productive. Enough folks have reminded @peter_bertok to be kinder with their words. Let’s try to avoid further meta discussion.

peter_bertok · July 15, 2018, 11:45pm

I’m just an outsider looking in, with no skin in the game. Feel free to ignore me. 8)

Pease allow me a chance to clarify the (very off-topic) point that I was trying to make, I don’t want to be seen as advocating for more rudeness, because obviously that’s not constructive…

My entirely unscientific observation over decades is that successful languages seem to be usually designed by surprisingly few key people, providing “a clear design direction”, at least initially. E.g.:

C - Dennis Ritchie
C++ - Bjarne Stroustrup
Python - Guido van Rossum
Perl - Larry Wall
Java - James Gosling
Ruby - Yukihiro Matsumoto
JavaScript - Brendan Eich

I agree with you that Linus-style leadership is probably not the best example to use, because he can definitely rub people the wrong way. So instead, consider the list above, because it’s more apt to Rust anyway. Are any of those people rude? Brash? I don’t think so, or at least not infamously so, like Linus. Still, I would argue that their clear leadership was required for the success of their languages.

Notably, most of those languages have changed over time, being taken over by cooperating committees or communities with no clear leadership. In particular, C and C++ have stagnated and wandered around aimlessly for over a decade because of the incompatible requirements of the various vendors all having their say.

It’s entirely possible that Rust can develop to be a successful, widely used language that isn’t relegated to some specialised niche. I’d love it, if this happened! I’m also watching the community-driven process with absolute fascination. Maybe it can work! It’s certainly educational to watch it all unfold in real time. To my knowledge, no mainstream language has ever started out community-driven and succeeded. Maybe Rust can pave the way…

A side-note on tokio improving: coincidentally, this was just posted today:

In particular, this bit is impressive:

For a long time, many users have asked for us to remove the Error type from Future, because they did had a non-IO use case in which the Error type was inappropriate. We’ve come around to agreeing with them; having the Error type would require that async functions always return a Result, and would not enable non-IO use cases for asynchronicity (such as lazily & concurrently evaluated pure computation).

So in this case at least the community process seems to have worked! The “pollution” of the Future trait with std::io::Error was one of my main issues with its design, and it seems that similar feedback by others with a similar opinion has had positive impact. 8)

However, note this warning from a post about the announcement: https://news.ycombinator.com/item?id=17536912

Languages like Python, Perl and JavaScript have been taken over by their open, anyone-can-contribute module systems and usually this has resulted in an absolute mess. I’m just a bit concerned that Rust is headed down the same path, and that this is hard to see during the honeymoon phase when it hasn’t yet scaled up to the point that it has 8 different incompatible async I/O libraries like Python…

vitalyd · July 16, 2018, 12:24am

Rust didn’t start out community-driven either - it was a single Mozilla researcher’s “pet” project. It’s important to keep in mind that when the languages you listed started out, it was hard to impossible to have a community; internet wasnt as ubiquitous as today, let alone enablers like Github. If they were then perhaps those languages would’ve evolved differently as well, possibly for the better.

I also don’t think Rust is as community driven as you make it sound. There is a core team, that while open to outside opinions and contributions, ultimately decides on changes and steers the ship.

gurry · July 16, 2018, 1:37am

Yes, it has. Very, very resoundingly, it has. You just need to compare Tokio when I made the above complaint and what it is now.

(And this is my last comment on the off-topic topic)

juleskers · July 16, 2018, 4:59am

Thank you for calling me on my bad form, I’ve removed the uncalled-for community-namecalling from the post. In retrospect, it even detracted from the point I was trying to make.

Since this is the second time I’ve broken courtesy on this thread, I obviously can’t handle the heat here. I’m excusing myself to go look for the mute button.

DrizztVD1 · July 16, 2018, 8:31am

Your input is clearly valued by the Rust community. What is needed, however, is to turn your views into actionable points that can be utilised to improve the direction of Rust. Prior to the 2018 edition release, this discussion is rather pertinent.

Please consider writing up pre-RFCs or whatever on a handful of the most important additions and modifications (actionable detailed items that can be executed) that would be needed to ensure that the dystopian future of incompatibility of Rust packages you described does not happen. That will serve to turn your knowledge into practical currency to further Rust constructively.
I disagree that a single individual/core team is necessary to drive a successful language, instead, a unified vision and mission-statement is required. In addition, the willpower needs to exist to realise that not everyone can be satisfied with one single language and that that, sometimes, calls for a certain feature/change should be rejected purely on the grounds that there are already viable alternatives that are better suited to some people’s needs - especially if this feature/change is not aligned with the chosen mission statement of Rust lang.

Focus on a common vision is imperative to ensure that a common focus is maintained in the core community since doing a few things extremely well is repeatedly shown to be better than doing many things averagely. The Rust roadmaps have shown to be a good solution to unifying the community, and could perhaps eventually morph into an overarching “Rust will always do x,y,z well, and anything that subtracts from that is not ok”.

Your points on side-channel attacks on Cargo is highly relevant to the perceived Rust vision of security and can be said to be a viable core focus of Rust. Why not develop the counter-measures further and lead the improvements?

gilescope · July 16, 2018, 11:26am

Just a reminder that this more general thread has sporned a more targeted thread for exploring some concrete improvements on the IO front:

I'd suggest crating separate topics for other specific improvements so people can give more targeted discussion.

KillTheMule · July 16, 2018, 11:28am

Totally OT, just can’t resist:

I see what you did there

gbutler69 · July 16, 2018, 11:42am

Sporned? Is that where you use a “Spork” to dig out of the morass?

gilescope · July 16, 2018, 2:00pm

Sometimes my dyslexia gets the better of me!

Ixrec · July 16, 2018, 10:37pm

Sorry to add to the meta chat, but I think this point is extremely important, and all the previous reponses to it have been rather abstract, philosophical or second-hand, so I feel the need to add a more concrete “lived experience” response.

This sort of attitude immediately drives me away. I would never consider contributing to Linux because I simply don’t want to deal with the kind of people who think “simply stating facts” is impossible to do without “sounding harsh”. That’s not constructive. That’s not fun. That doesn’t inspire anyone to do their best work. I’ve already wasted far too much of my free time on internet communities (programming and otherwise) that devolved into a toxic mess because they tolerated this sort of attitude.

In this particular thread, I basically stopped looking at your posts after the first half dozen or so, because I couldn’t even figure out what you were trying to say and it was raising my blood pressure. So instead, I skimmed everyone else’s responses to see if they’d managed to extract any genuinely interesting ideas from beneath the vitriol, but even that failed to impart anything useful to me because so much text was wasted on responding to the vitriol itself.

In contrast, Towards a more perfect RustIO has been constructive all the way through, so now I finally understand what you were trying to tell us, and I’m still happily reading everyone’s posts over there. So congrats to the amazing community here for turning that around so completely, and let’s try to focus on that thread now.

gilrain · July 17, 2018, 11:54am

Just another voice who is here because I simply won’t be anywhere jerks (to be kind) are tolerated or even lionized. A number of other open source projects have variously admired or envied Rust’s phenomenal growth. It is because Rust welcomes and makes comfortable more than just the hostile-by-default.

Ekleog · July 17, 2018, 12:32pm

OK so as this meta-thread looks like it’s not going to die, a few
comments on it that I’ve been refraining from putting out so far.

There are two kinds of jerks. The one who attacks individuals, and the
one who attacks the language (in the present case).

I haven’t read the whole thread, only ~10 days of it. But I’ve only seen
@peter_bertok being a jerk against the language. On the other hand, I
have seen attacks (at least one that couldn’t be miscategorized for
anything else and has been flagged, but others that felt like it to me
too) against @peter_bertok personally.

In my opinion, harsh attacks against the language are OK. Harsh attacks
against individuals are not.

And the reasons why I think harsh attacks against the language are OK:

If someone at the commands of the language attacks it, it won’t be
harsh, it’ll be loving
Hence, only people that have relatively little influence in the
community can be harsh against the language
Which means that they can just be ignored (in opposition to eg. Linus
who couldn’t be ignored because he is at the helm of the kernel)
And so these persons’ feedback will at best be read by someone and
acted upon, and at worst be ignored
Oh, and if we feel personally offended when our favorite language is
attacked, maybe that means it’s time to stop writing Rust for a bit of
time and get some air