Conversely, my argument is that a rich set of blessed libraries is the standard library. Externalising bits and pieces into a package manager just makes it hard to see where the officially anointed libraries end and the heathen filth begins. 8)
I’m having exactly this experience with C# right now. For example: NuGet, just like cargo.io, fails to properly include the full set of “dependency” flags actually required in complex (real!) scenarios. Nice, easy versioning of “3.0”, “3.5”, “4.0”, etc… has gone out the window, and what I’m left with is a mess of incompatibilities that I shouldn’t have to deal with.
In my particular case, something like this* happened:
- I’m writing a C# PowerShell module, which is basically just a DLL library. This part is easy.
- This library has a NuGet dependency that in turn has a transitive dependency on a .NET Core module that is also a NuGet package. This is generally not optional now, because the dotnet core standard library is modular. So importing a dependency like
System.Databehind the scenes. They’re independently versioned, just like cargo packages.
- The linkage is dynamic and done at runtime, so neither NuGet nor the dotnet compiler have any idea what this dependency is exactly, because the pwsh runtime pulls in some specific version of the standard library modules.To work around the issues this causes, there is a wonderfully opaque set of compatibility shims all over the place.
- If I stay on version 4.0 of the package I’m using, it doesn’t work, because of a bug in that version of the library.
- If I “upgrade” to 4.5 everything breaks on Linux because some legacy compatibility shim was removed. This shim is still required by pwsh on Linux, but not Windows. I have no control over this, it’s a component of dotnet core pulled in by the pwsh executable, not me, but it’s incompatible with my DLL.
None of this is documented, reflected in NuGet.org, managed by nuget.exe, or in any reasonable way discoverable. This is 100% code written by one organisation, Microsoft. Half of it was written by one team (pwsh) and half it by just one other team (dotnet core). They’re likely working in the same building and yet there are already entire categories of scenarios where everything just blows up in my face unpredictably and there is nothing I can meaningfully do on my end to fix it. I literally tried every combination of csproj settings through brute force to see if it could be made to work. It can’t.
I did a lot of research, and I worked out that basically the pwsh team had likely never sat down to write a “third-party” cmdlet module in C# that uses a NuGet dependency. They’ve written a bunch of C# modules, but it’s all part of pwsh, and hence versioned in sync with it, so they’ve just never been exposed to the mess that they’ve created for everyone else, the 99.9% of the developers who will actually have to use this stuff.
I just have a feeling (perhaps unjustified), that a lot of Rust development is similar. It works in a controlled environment, but I doubt it can handle the combinatorial explosion that is already making NPM a nightmare for developers. The more popular cargo gets, the worse it will get.
Let me paint you two scenarios in analogy with my pwsh experience:
Scenario 1, the Servo team (or similar):
- A medium-to-large team with a long-term project. Timelines measured in years.
- A lot of overlap with the Rust core language team and the developers of the key crates.io packages that make up the “semi-standard” library of blessed modules. These guys likely often meet in person, work in the same building, or correspond on the internet on a regular basis. There is trust built up over years.
- Dependencies change slowly, and they have months to patch up any small inconsistencies.
- They directly control most of their dependencies, including transitive dependencies. The packages were written for Servo, or by a Servo team member, or someone in close collaboration. Something like 75-90% of the code is under their “control”. They know exactly what they’re importing and where it comes from.
- The code is open source. It’s not “sold”, and even if it is, there’s a disclaimer that says that they’re not liable for anything. There’s no warranty.
For scenarios like above, I am in no way disputing that the Rust development environment “just works”. It would be a big step up from C++, provide a lot of flexibility, and generally enhance productivity. This is great. Game developers would be in a similar boat, and I’m sure there’s many more examples.
Scenario 2, an enterprise tool:
- Lone developer or small team, some of whom… are not great developers. You can’t control everything, and the people you work with make mistakes or are just a bit sloppy. Maybe good, but overworked.
- The goal is to plumb together a bunch of libraries. Feed XML containing JSON into a database. Make it authenticate with LDAP and SAML. Talk to a legacy Java app. Import from a mainframe. Etc…
- Your dependencies are published by corporations, not “Bob” down the corridor. You have zero control over these packages. Pull requests are silently ignored, assuming it’s even open-source to begin with.
- You’re stuck on an old compiler because of the above.
- Timelines measured in weeks. If something breaks, you are screwed. Deadlines make wooshing noises and emergency meetings are scheduled to recur daily by project managers who don’t care about “package incompatibilities” and other meaningless technical talk.
- You can’t spend significant time researching the pedigree of every dependency, there’s hundreds of transitive modules being pulled in, looking through them all would eat up your entire dev time budget before you wrote a single line of code.
- Even if you miraculously do check everything, maintenance is done by a different team on a different continent. They’ll blindly pull in the latest “updates”. You have no control over this either, management six levels above you signed an outsourcing contract for BAU support.
- This is going to be processing sensitive data worth millions. If it’s insecure or crashes, your managers will avoid all responsibility and blame you. You’re lucky if you’re only fired. You’ll likely avoid jail, but lawyers will probably get involved.
Now, in this scenario, Rust and Cargo are… not great. If this happened in scenario #2, that poor solo enterprise developer would not be a happy person:
A dev in scenario #1 is much less likely to be affected, and could just laugh something like this off. A dev in scenario #2 would just avoid Rust if he’s got any brains. I certainly would not use it as it stands, because there’s virtually zero protection from the kind of vulnerabilities that were not just predictable, but predicted, and have occurred for real. Why would I risk it? For what? Twice the runtime performance? Pfft… I could just request a server 2x as big and not risk my job and my career.
So imagine if I was a bad actor like that guy who put malware into the eslint NPM package. Just pretend that I’ve been contributing to cargo.io under a pseudonym (real names and code signing not required, remember!). There’s a popular package that I’ve uploaded years ago with a cutesy name that lots of people use.
I’m going to inject malware into it next time I fix a critical bug. This malware will steal github.org and cargo.io credentials, which I will use to further distribute malware into as many more packages as I can get my hands on.
Now what? What are you going to do about it?
The clock is ticking. Seriously. In a week or so, one of your dependencies is turning evil. You don’t know which one. You need to update it along with a bunch of others. Tick… tock… tick… tock…
*) I have no idea what’s going on exactly, the entire thing is a ludicrously complex black box as far as I’m concerned. Other people have reported the same issue, and it’s still open. Nobody from the dotnet core team has any clue what to do.