Cryptofin as a strong foothold for Rust

1

Moved from internals forum here. Some initial comments were posted there, but lets continue here.

cryptofin is a term I use for Crypto-Finance - a growing industry using software and cryptography to build systems transferring value. It notably includes, but is not limited to: Bitcoin, other crypto-currencies, blockchains etc.

Before we begin: Please let's keep our personal opinions about Crypto Finance, Bitcoin and alikes to minimum.

I'd like to start with observation that programming languages greatly benefit from establishing themselves in certain niches. It ensures traction, real life usage, corporate support, mind share etc.

Notable example: Kotlin's popularity exploded after it became official language of Android. Similarly other recent yet successful languages established themselves like this: Swift has iOS, Go - distributed systems, Kotlin - Android.

Rust is very versatile and is specifically trying to establish itself in webassembly, embedded and core systems programming. While obviously worthwhile and with some success, embedded and general system programming are a more difficult areas, because they already do have estabilished dominant programming languages.

Fintech seem so interesting is that it is a fairly new niche, without a strong, long estabilished or otherwise dominant PL. Additionally Rust is really a perfect fit for cryptofin, which has unique and demanding set of requirements:

  • security, safety and reliability - as cryptfin operates on valuable digital assets in adversary environments, there is no room for mistakes;
  • performance - as performance characteristics are critically important for economic and functionality of the system;

The fit is good, that without much attention Rust has already become de facto language of choice for cryptofin. A lot of popular cryptocurrencies and cryptofin systems is using and is fond of Rust, despite Rust community never explicitly trying to address that niche. Even Bitcoin, which was initially implemented in 2009 in C++, has a growing community of Rust developers.

While cryptofin is not yet a huge industry, it is really quite well funded, and as a principle - is typically based on Open Source: any cryptofin Rust project tends to provide Rust community with a lot of Open Source work.

Cryptofin is also on the cutting edge of many efforts: security, trust minimization, cryptography, etc. Dominating cryptofin means getting a mindshare in many areas important to high-caliber, mission-critical system software.

I think it's worthwhile to raise the awareness of this quiet success, and possible future benefits . While at it, I'd like to list from the top of my head problems with Rust for cryptofin (and specifically rust-bitcoin, which I'm somewhat participating in). Addressing or helping address these, could lead to even stronger dominance of Rust in this area.

Trust (bootstrapping). Cryptocurrencies are a lucrative target for cyber-crime, malicious software etc. Because of this cryptofin developers are very distrustful, and are trying to eliminate any technical risks. Specifically, right now most Rust users are trusting rust compiler binaries, compiled and distributed by Rust team. That is putting a lot of trust in Rust team - their infrastructure, employees etc., especially in the light of possible state-level actors trying to subvert or shut-down such systems. One could just compile the compiler itself... but you can only do that with a Rust compiler, again ... provided by Rust Team. It's a circular dependency, and unlike languages like C/C++ Rust has only one feature-complete implementation ATM. The problem was somewhat solved by independent bootstrapping using mrust, but it is still a a pain point. I don't know how much can be done about it - maybe Rust developers would be OK with making sure newer versions of the compiler, compile with somewhat older versions, to ease the pain of independently bootstrapping the compiler, or mrust could get more help to keep up with the evolving language.

Language and ecosystem stability guarantees. Partially because of the bootstrapping problem, and partially because of general conservative approach, lack of any LTS releases, backporting fixes, minimum required rust compiler version management, etc. is a problem. Cryptofin developers can not and do not really want to roll with the "just use the latest stable version" mantra. I observe that this creates a lot of tensions (example). One one hand I understand that language and ecosystem are very alive and quickly evolving, on the other hand I understand that people working on rust-bitcoin are used to their code compiling with no problems on 10-year old c compilers. Rust is simply not yet there (not even 10 year old in the first place), but it seems to me that it would be really beneficial for perception of Rust's stability and maturity to start rolling out some stability guarantee: eg. starting with 6-month-long LTS cycles, that could get extended with time, as the language stabilizes, and encouraging some backward-compatibility minimums in the ecosystem.

6 Likes
2

From my perspective as someone who works on the compiler I am not OK with that. Not being able to use new features or more importantly, being forced to keep in old garbage, already causes pain in developing rustc and would not want to make that experience substantially worse.

I would just like to note that if you want to find out what the latest stable features are in any given release you can rather easily do so by reading https://blog.rust-lang.org/ (for highlights) and https://github.com/rust-lang/rust/blob/master/RELEASES.md (for details).

6 Likes
3

Speaking as a rank outsider and newbie to Rust a few things bother me about these ideas:

  1. Clearly the Rust language, the compiler and tools, the libraries and pretty much everything rusty are still very new and under development. Trying stabilize things with standards and LTS releases etc seems somewhat premature.

  2. I don't buy the idea that "cryptofin", whatever that is, is a special case that needs special attention from the Rust folks. I can understand that it has performance requirements and safety/correctness requirements. But my take on things is that so does everything else, web browsers, cloud services, operating systems, embedded systems, especially the new world of IoT widgets. Having the Rust folks pay special attention to some particular niche group of users does not sound like a healthy idea.

  3. "Cryptofin" has to be the ugliest expression to come out of the technology world since the "SFINAE" of C++

  4. I have my own hopes and dreams for the future of Rust, which I won't go into here, I don't want anyone diverting their attention.

6 Likes
4

I'm surprised. It is a bit like saying that Rust from a year ago was such a bad language that it's unbearable to work with. I was using Rust a year, and two ago I was happy. Sure, I enjoy the new features, and I'm looking forward to even newer ones, but I'm quite content with what we've got already.

Just to be clear: I'm not saying that we would have to have Rust compiler build with Rust compiler from 2015, and Rust developers are stuck with Rust 2015. All I'm thinking about is that each Rust compiler in version X, would be able to compile by certain previous version: eg. last LTS release (if there is such a thing), or X-3 or something. So Rust compiler developers would get access to shiny new features - just a little bit slower.

The benefit of such approach is that people that require trustless bootstrapping, would not have to do it via every release X, X+1, X+2, X+3 (which from what I known also often require some custom patches to work), and instead could cut the chain to X, X+4, X+8 etc.

Is that reasonable? And if not - do you think Rust would in near/medium term reach a level of maturity that Rust developers would be OK with using language from couple of months, a year or so ago?

I don't think LTS releases are changing anything about language standardization. They are an additional infrastructure and maintenance burden, in the name making life of slow moving/conservative users easier.

They don't require special attention. Their needs are voiced by users in many different areas, that would like to use Rust for serious stuff, where they can't just download latest rust release from a 3rd party and be YOLO about it.

Backward compatibility and stability are a strong requirement in many areas. And Rust community knows that - otherwise what would be point of having "Editions". One could just tell people to download newest Rust compiler and not bother with supporting two versions of the language.

So my question is - could some initial efforts be made at this stage, to make Rust to be easier to live with for people that want to migrate some serious use-cases from C/C++ world, where compilers can be easily cross-compiled for trustlessness and something less radical than "use only latest version" is a realistic approach. Or the general answer for such users is "sorry, we're not read yet, stick with C++ for now".

5

Forming a "working group" like the Embedded Devices WG or the Command-Line Interfaces WG might be a useful way for cryptocurrency developers to coordinate in making proposals or communicating their use cases to the Rust teams.

7 Likes
6

It is reasonable to assume that large code bases are the primary drivers for, and beneficiaries of, many of the changes in rustc, cargo, rustfmt, clippy and other Rust tooling. Given the innate feedback between rustc development and rustc features, @Centril's position about using the new features immediately rather than hobbling rustc development to the last LTS release is completely understandable.

I read the t-compiler threads on zulip every day. Many of these new developments replace prior approaches, so it is not feasible to keep both within an instance of rustc (e.g., lexical scopes for lifetimes -> NLL -> Polonius). That's also the case at a less-visible level within rustc, such as the ongoing effort to implement dataflow-based const qualification. It pretty much has to be "Out with the old, in with the new."

7

The nature of bootstrapping rustc is that we often have to remove or change unstable things from it and the rustc compiler and especially the standard library relies on tonnes of unstable things. This also includes unstable aspects of cargo that we want to take advantage of.

For example, I recently had to remove the [X..] syntax (which was used by rustc) and replaced it with another unstable syntax X @ ... Thankfully I was able to throw the old syntax out of libsyntax and use a third alternative that relied on neither. A few weeks later we changed it to X @ .. instead.

We often also change intrinsics and whatnot.

1 Like
8

Did I misunderstand something? I got the impression from some Rust presentation on Youtube recently that the intention (or even promise) is that future Rust compilers will be able to compile old code, backwards compatible to the release 1 standard at least.

If so, what is the issue, aren't we good to go to create millions of lines of code now that will be buildable into the future?

9

Weren't these backward compatible?

It seems to me that there are two obstacles here.

One is just being impatient to use new shiny features, which is is no different than authors of popular and important crates having to delay their gratification, because their crates are simply too important to break all downstream users.

The seconds is intrinsic, but intuitively it seems to me that the compiler code and the code that compiler provides/uses as intrinsics to the code it compiles can be separated. Does not seem like an insurmountable problem.

Both are probably at least inconvenient, but even if not now, then at some points Rust developers should consider to make the language... well... more suitable for some serious use-cases.

10

Each successive improvement supports code that fails to compile without the improvement. Backwards compatibility means that older versions of rustc can be compiled by newer versions. Restricting rustc to not use recently-stabilized features is like asking C compiler coders to limit themselves to the features of C99.

11

Isn't that exactly what is happening? I mean - do gcc devs implement gcc with C++18 features in C++18? I doubt it, but maybe I'm wrong.

12

Unlike other crates, in this case, the "downstream users" of rustc is just the people who compile newer versions of rustc.

We intentionally use "new and shiny" unstable features in developing rustc and libcore/alloc/std to dogfood these features in a large codebase. This is important because some features will not see any usage before they are stable so we need to test them internally in rustc. Clippy does the same form of dogfooding with their lints.

I don't know what you want to say by "serious" here -- surely most uses today of rustc are "serious".

3 Likes
13

The problems of using an unsupported compiler version are almost entirely centered around incompatible dependencies, and this will be solved when Cargo learns to resolve dependencies to only compiler-compatible versions:

https://github.com/rust-lang/rfcs/pull/2495

If you're happy to use an outdated compiler, you should be happy to use outdated dependencies too (either way, you can't force the entire ecosystem to wait for you).

LTS won't help here. For example, Rust 1.31 (2018) would have been a good LTS candidate, and it still doesn't work for you! The complaints you've linked to are about base64 using 8-months-old "LTS" 1.31 which is still too new.

6-month LTS won't solve the Debian problem, which has a 2-year cycle. If they don't align, that's a 2.5-year lag you're asking for. Even if Rust itself really committed to supporting 2-2.5-year old versions, you still couldn't convince everyone on crates.io to follow this.

7 Likes
14

My apologies, "serious" is the wrong word. All I mean is - in some applications:

  • one can not trust binaries provided by a third party, or built by binaries provided by a third party for regulatory, legal, or just tin-foil hat reasons,
  • can not tolerate 6-month upgrade cycles, involving builds breaking and lack of backports.

I think it's fine to say we're not there yet / we don't have resources. But I hope to get acknowledgement that these obstacles would be great to overcome eventually.

1 Like
15

Note that there's already an open Cryptography WG application:

As someone who straddles the "crypto means cryptography" and "crypto means cryptocurrency" spaces, I think it would be perfectly fine for this WG to be inclusive of cryptocurrency developers, and that these developers represent a significant portion of Rust cryptography users in general.

--

My 2c on one of the issues broached here, as a stakeholder in the cryptocurrency space:

I see no unaddressed needs from the Core Team. I think mrustc answers the general question of how to bootstrap rustc without rustc, solving the "trusting trust" problem from my perspective.

To me the only unaddressed issue is bit-for-bit reproducible builds of rustc, which are tracked here:

There has been some great recent progress on this front:

So all I have to say is: keep up the good work!

2 Likes
16

I work directly in the cyber-coin space too.

I agree with @bascule that "trusting trust" appears adequately addressed already. I'll add that MIR and miri might provide another method for improving trust in the compiler.

I also agree that reproducible builds represent our major compiler level concern. In fact, we'd happily fund work on reproducible builds, so please contact our collaborations team if interested: GitHub - w3f/General-Grants-Program: Web3 Foundation General Grants Program

We'd also love NRVO for doing smaller air gaped signer devices, but that's a secondary concern.

17

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.