What does Rust need today for server workloads?

I've been writing a high-performance database in Rust for research purposes, and that has highlighted some problems in this space:

  • Async channels are a pain: when sending on an async channel, the channel is consumed, and is only returned on wait. This makes it fairly annoying to deal with threads that issue work internally, and that should block until that operation finishes. The sender can no longer be a part of a struct dedicated to that thread, because the send would move out of the struct.
  • Channels are slow: async channels are a fair bit slower than the sync channels in std (at least last time I measured). Unfortunately, since mpsc doesn't support select (well, beyond an unstable and old nightly-only feature), async channels are the only feasible alternative here.
  • Signal handling is essentially non-existent: this has been mentioned a couple of times already, but is worth repeating. Without being able to neatly clean up the program, or do things like reload configuration in response to particular signals, writing a server that fits into a regular devops deployment is fairly hard.
  • Non-lexical lifetimes: while this isn't really server-specific, again and again when writing request handlers I come across cases where I need to introduce extra scopes to satisfy the compiler. This adds some unfortunate friction to an otherwise fairly pleasant experience.
  • Non-trivial overheads: tokio, and the various layers on top of it (e.g., tokio-proto) introduce fairly substantial overheads (especially in terms of latency, but also throughput) compared to doing blocking I/O. I don't know exactly where this comes from, it seems to be a little in every layer. Nailing this down should be a priority if the goal is to have high-performance servers written in Rust. Think of something like memcached where the speed at which you can do network I/O is nearly 100% of the workload!
12 Likes

Here is my wish list for server-side development:

  • Complete TLS story wrapped in tokio-tls, including SNI, ALPN, access to client certificate chain validation during the handshake.
  • More ergonomic work with futures. I imagine bringing together impl Trait (as it's on its way to stabilization) and future combinators is very important to enable building async services. And yes, async/await would help a lot on top of this.
  • more focus on practical use cases with libraries. tokio and hyper do have decent APIs already but it is puzzling at times to understand how to bring things together to make it do what you have in mind.
  • Show me how not to be scared of panic slipping into one request handling and crashing another 1K requests being handled at time.
  • HTTP/2 is a must.
  • Putting together a thorough sample app that does quite a few common things apps on server normally do would surely help in understanding pains. I'm talking about security, auth, logging, handling requests, talking to DB. That might be a few months (or even years away) as the ecosystem grows but would sure be an encouraging place to start for many people interested in running rust on server.
4 Likes

Is tokio-signal what you're thinking of here? @blt and @jonhoo would that work for your use cases?

2 Likes

Hadn't seen that -- that's a pretty good start! I'm weary of making all of these things tokio based though. Having a version available where you can register a global signal handler directly would be nice, as that is how many developers are also used to thinking about these things. Furthermore, it allows handling signals without pulling in all of tokio.

2 Likes

Automatic generation would be nice. I use that feature of alembic extensively in production - I found that out of the box, it is pretty bad, but if you take the time to enable some of the extra features (like detecting field type changes) and implement some extra hooks for custom data types it works great.

I even added a layer to automatically detect required data migrations within JSON columns when the corresponding python classes change, and generate corresponding stubs within the migration file.

I do wish it generated pure SQL migrations though.

1 Like

Puh, where to start.

As a note before hand: A lot of this will sound very critical. I love Rust and all the work that is going on in the ecosystem, so take it as an attempt of constructive criticism that is focusing on the negatives rather then "everything is bad, stay away".

  • Tokio

The stack is very complex, and even simple code is hard to write and even harder to read.
The compiler errors are an undecipherable mess, and often you have absolutely no idea what the types are after your third chained closure when you re-read some code. impl trait helps somewhat, but only marginally.

I can figure it out, I know coworkers who would run. IMO it's just not feasible to gain any kind of large adoption without async/await / generators in the language, with a clean syntax and compiler support for good error messages.

If you can handle all of the above, you are still only in an event loop on one core.
Getting a performant multi core application is still up to you. There's futures-cpupool, but that still leaves a lot of the design and implementation of how to spread out the work load up to you. How do I distribute the workload across cores? do I just use a cpu pool with one core less then the system total? Do I go for some home grown incomplete actor system since there isn't a larger well adopted library for that? If so, what messaging patterns do I use? There is only a mpsc channel type in std, which is limited and doesn't have that great performance, so now I need to go searching for a channels crate. How do I monitor and possibly restart my worker threads/actors?

Fitting all the pieces together, designing and implementing something relatively trivial can takes hours in Rust where I can just pick up Go and have something with easy concurrency and good performance in 15 minutes. You'd have to be either very dedicated to the language or really dependant on high performance to go with Rust over Go, Node or the Beam (Erlang, Elixir) for the async story right now. It's just too complicated.

Also, Spreading tokio over so many different crates makes the learning curve even steeper.
You need to understand:

  • futures (streams)
  • tokio-core
  • (maybe tokio-proto)
  • (maybe tokio-service)
  • pretty soon, you'll dive into mio
  • futures-cpupool

Maybe collapse everything back to futurres and tokio and build from there?

Also, as someone else mentioned: a lot of the libraries and clients for Tokio a very fresh, and the disk io story is not clear. You need to first figure out what can block and offload that to cpupool or worker threads.

  • HTTP

There is a lot of work going on, but the ecosystem is immature.

Sync hyper client/reqwest basically cannot be used for production right now because you can't set DNS timeouts in std and a timeout will freeze your whole thread indefinitely, which I've run into repeatedly. That can get tackled with async hyper, but the only option for production http clients right now is curl, which doesn't have a convenient API wrapper crate which offers something like reqwest. Meh.

Hyper async has the problem that you need to understand the whole Futures/Tokio stack and is not near stability.

Getting to a stable battle-tested http2 stack is even further from the horizon.

On the framework side:
Iron is a bit cumbersome to work with and there isn't a lot of development going on.
Rocket is great and much easier to use than Iron, but still maturing and needs more flexibility in some areas. Also, it's only on nightly and will stay there for a long time, which is quite a no-no for production.

  • DB

There are pretty good clients for postgres and mysql, but they are both driven (almost) exclusively by one developer each. Not too so sure if I'd want to rely on those for production. Also they don't support a light weight orm approach that at least allows deserializing into structs with a derive.

Diesel is a really great approach, but it suffers from several problems:

  • like futures: confusing / messy compilation errors.
  • very, very little documentation
  • only 1-2 active developers
  • strong inherent limitations. I tried to use it for work but it supports at most 52 columns, and all the dbs I wanted to use it with have at least one 60+ column table. I tried to implement a new feature bumping to max 256 columns, but that took over an hour to compile, so also a no go
  • fallback solutions for more traditional untyped sql

There's also Rustorm but it's not too active (and no mysql support, for example).

  • Libs, libs, libs

Things I want, stable tested and actively maintained. Preferably supporting futures/tokio:

  • redis

  • memcache

  • kafka

  • etcd

  • protobuf

  • Conclusio

There is some really awesome work going on in both the language and the ecosystem. Serde, mio, futures, tokio, diesel, slog, hyper,...

I have high hopes.

A lot of pieces are moving into place, and you can get a lot of things done already right now. But a lot of things also need time to mature and a lot of dedicated contributors to move things along.

19 Likes

Let's start with a disclaimer: I am a PHP developer. :slight_smile:

I tried to write a (simple) RESTful API in Rust a while ago, using hyper+tokio, and my pain points were:

No framework except for mio-based hyper works with stable Rust

Rust is already hard to sell, being a new language, with its very small talent pool to hire. If I said "we need to use nightly" they would just laugh at me.

So it's not an option.

It's extremely hard to have both async and multithreading.

I tried and almost ended up rewriting hyper from scratch.

We have both IO-bound and CPU-bound endpoints in our API, so a simple async service doesn't work very well, but we also have very IO-bound (disk, MongoDB, S3) endpoints where both webserver-level and local async operations would be wonderful.

Dependency injection was impossible to achieve

This is probably my fault. Probably I still don't understand "the way of rust" yet, so I was thinking like in PHP.

In all our endpoint we need to inject several objects into our controllers/actions. We have dozens of actions with different dependencies. Most of the examples for rust frameworks are with just 2-3 endpoints, defined as closures in the router. That would be an impossible mess to manage for us. We need separate classes (or structs) for the actions, or at least for the resources (in REST terms).

Some of the dependencies are services (a Mongo collection, the S3 client, etc.). Some are "one shot" (an array of strings allowed for a query, an s3 bucket based on the connected user, or the day of the month, etc.). I didn't find a documented way to do it.

Logging is hard

Compare Monolog in PHP:

$log = new Logger('name');
$log->pushHandler(new StreamHandler('path/to/your.log', Logger::WARNING));


$log->warning('Foo');

with slog:

let log_path = "your_log_file_path.log";
   let file = OpenOptions::new()
      .create(true)
      .write(true)
      .truncate(true)
      .open(log_path)
      .unwrap();

    let decorator = slog_term::PlainDecorator::new(file);
    let drain = slog_term::FullFormat::new(decorator).build().fuse();
    let drain = slog_async::Async::new(drain).build().fuse();

    let _log = slog::Logger::root(drain, o!());

Besides that, decorating the logger is quite complex. We have this class:

class ContextLogger implements LoggerInterface {
  public function __construct(string $context, LoggerInterface $logger, array $extra = []) {
    $this->context = $context;
    $this->logger = $logger;
    $this->extra = $extra;
  }

  public function log(string $level, string $message, array $data = []) {
    $data['context'][] = $this->context;
    $data += $this->extra;
    return $this->logger->log($level, $message, $data);
  }
}

which allows to do this:

// $logger is injected
$childLogger = new ContextLogger('Process child', $logger);
$this->spawnChildren(new ChildProcess($childLogger), 10);

$logger->info('Children started');

This will log all parent messages "plainly", and all children's message with an additional contextual array of... contexts. :slight_smile:

It's hard to work with web data

Iron does a decent job to ease this, but the documentation is quite lacking. Inline docs can be useful to lookup what you already know, but without a good tutorial and detailed explanations on how to use the components, it's nearly useless.

There is no example on how to use cookies, for example, or on how to write middlewares, or a semi-automatic way to decode data coming from the client based on the Content-type (automatic JSON, multipart or uploaded files decoding), or an easy way to add attributes to a request in a middleware.

Take a look at the PSR-7 to see what I mean, or to the Slim framework's Request object and its getParsedBody() method.

The web is stringly-types, so there should be a way to "accept anything and check later". I know this means dynamically-typed structures, but now serde is 1.x and having something like serde-json's Value "integrated" in the framework would make a difference, IMHO.

My wishes

I didn't manage to go much further, to be honest, but another nice thing I would have liked was a way to have multiple protocols managed by the same binary. For example HTTP on a port and gRPC on another. Maybe it's possible, especially with tokio, but again, the documentation is quite lacking.

The idea I got is that the quest for purity and performance is probably getting in the way. Sometimes it's good to give up 20% or even 50% of the performance, or just allocate a struct instead of passing references around to give the developer an easy interface to work with.

We are now working with servers giving us up to 200 req/sec per thread. We don't need to go to 100,000 req/sec. We would be extremely happy with 5,000 or even 2,000, but if it takes 3 weeks to develop the same thing we do in 2 days in PHP, it's never gonna be used. We'll just pay for bigger servers. It's cheaper.

To sum up, I would like to thank everybody working on these projects. I know it's not easy, because I tried. :smiley:
Rust is growing extremely fast, and I have no doubt it will be great, but it's still too early for web-based sites and APIs, IMHO.

7 Likes

@alexcrichton I'm in kind of the same boat as @jonhoo. I hesitate to pull in tokio. For my largest daemon cernan it's not clear to me that I could without a fundamental rework of its internals. Even for the small daemons, I hesitate to attempt a run at tokio given that โ€“ as I understand it โ€“ the API is still in flux. My experience with trying to learn tokio โ€“ and mio, much before that โ€“ mirrors the experience of others in this thread.

All that said, I would for sure be open to example uses of tokio-signal in non-tokio applications.

What does "accommodating" look like, for a platform like Heroku? I'd be at least a bit interested in helping make that happen, if I can, but I'm not sure what the missing pieces are.

-o

2 Likes

My reply is going to be very different from the others, I think; but these are the issue that I've heard from others in the project when we were considering moving (parts of) tor (the C daemon powering the Tor network) to Rust. I hope the thoughts here are still useful in the context of the question.

Starting a new project in Rust is easy, as is targetting a platform that is on the Tier 1 platform list. If you don't run that platform, maybe it won't work for you, no big whoop - the platform you care about is covered. But if your currently supported platforms range from OpenBSD, Android, Solaris, arcane BSD and windows crosscompile via mingw, all backed by a big autotools-using build system, Rust doesn't look so pretty.

A lot of this is architectural. Maybe it wasn't the smartest idea to have the Tor binary be a client and a server at the same time. But it certainly helped get it to where it is now, with many thousands of servers on a variety of platforms contributing to a single network, relied on by millions every day. That said, most of our relays operate on some version of Linux on amd64-compatible hardware, most of them using Debian. Our sysadmins run Debian, our release infrastructure relies on Debian, our windows built are cross-compiled from Debian (or Debian-derived) linux systems. Rust support on Debian is terrible. There are very few people working on making it better, but they have a hard time. Their bugs don't get much traction, people excited about helping out are quickly overwhelmed when they realize just how incompatible the approach of cargo is to that of traditional package management with responsible packagers who offer security updates, stability of dependencies for a deployed system for at least a couple of years, and a large set of supported platforms.

So, what are my suggestions? I don't have solutions, but a few pointers.

Clarify what Tier 2 means. Right now, anything not Tier 1 is treated as "we must not ever rely on this to work, if we moved to Rust users on these platforms would not be able to use Tor in the future" by those who are sceptical about a migration. Is that true? Is Rust actually that terrible on all of the Tier 2 platforms, or just some of them? Looking at Debian -- Package Search Results -- rustc it seems pretty bleak. The next Debian stable has rustc versions only for x86 and amd64. It will not ship cargo Debian Package Tracker because the version that was to be included ships a vulnerable libgit2 as a hardcoded dependency. Security updates from the Rust team aren't available, the answer "move to a newer version then" doesn't work.

Our users (and myself, for that matter) will use that system for the next two years or so. "just install rustup via this curl | sh thing" won't fly. It is not auditable, it gives too many people an opportunity to install completely unauditable code, one of the big reasons why using a distribution such as Debian is such a nice thing. As a developer, I can work around all these requirements, and I happily do. But forcing all my users to do that is something I cannot do until support is more widely available.

Help distributions get this right. Ensure rustc and cargo are embeddable into larger linux distributions for a variety of platforms. Make sure they know how to backport security fixes to older releases of rustc and cargo, and make them available yourself. Ensure that everyone realizes that bumping the required compiler version is always a breaking change and you cannot claim to adhere to semver if you don't bump the major version for such an update.

Think about how to use rustc and cargo as a building block of a larger build system like autotools. Ensure out of tree builds can be made to work for Cargo, all the tools have easily usable switches to force entirely offline operation, including rustup. Make sure errors that occur because a resource isn't available include information about what the resource is and why it's needed, and what exactly it is.

For us, the libraries are basically fine. They will get better with time, that's great and very welcome, but until then, we can write our own where truly needed or rely on the C code that we already have. What we cannot do is the infrastructure work. We cannot get Rust into all the distributions our users rely on. We cannot declare which platforms are actually supported to which extend.

20 Likes

So I'm don't have much experience with Heroku, but when I looked at 'supported' languages and frameworks, there are things like node, ruby, java, scala, go, and maybe some more. I guess I was thinking for rust to be in that list. However, i was informed of buildpacks, and it looks like there are ones people have already made for rust. Maybe those would work for me!

My guess is that it is for the reasons discussed here: Split hyper into more crates ยท Issue #894 ยท hyperium/hyper ยท GitHub.

In addition to the many things mentioned already, something I very often want and am frustrated by not having is an equivalent of the crypto/tls and crypto/x509 packages from Go. I do a lot of work in "cloud native" infrastructure (e.g. Kubernetes) and would prefer to write tools for it in Rust, even though that ecosystem is absolutely dominated by Go. The biggest pain point is having to shell out to openssl or cfssl to work with X.509 certificates and other tasks related to CA management.

Perhaps taking that a step further, it might be fair to say that Go's standard library experience is one of the reasons I think it will be hard to convince people in the cloud infrastructure world to try Rust for writing tools. This is more or less the batteries-included-vs-not debate that's been hashed out a million times, but regardless of what's in Rust's stdlib or not, I think everything that's in Go's stdlib must exist as well known, high quality libraries in Rust before Rust will be seriously considered by a lot of people in my part of the industry.

14 Likes

For us async/await is the biggest item currently. We have a lot of futures-using code (partly for performance, partly to manage concurrency as the initial post suggested) and a lot of it is really difficult to write and understand. Having to use loop_fn is horrible. It's also very difficult to debug this code, although since we're writing a debugger the onus is on us to fix that!

2 Likes

Yes! The absence of async db clients is a barrier for server develop.

thanks for the feedback.
In our organization, we would also be interested in FAAS frameworks build on top of Kubernetes for example.
Rust could really shine here.

  1. I still appreciate Iron as a foundation of web frameworks. It didn't attempt to copy those dynamic language frameworks and keep its core tiny, extensible and static typed. Added features to Iron is really enjoyable because you can write it as middleware and distribute separately. The problem with Iron is its development pace and ecosystem.

  2. Isn't adding async/await keywords bringing us to the green thread old days? Is it possible to do it with macro, at a library level? I'm afraid adding too much stuff to Rust language core conflicts with some original design principles.

  3. We do need a solid logging library to take rust to production, like log4j to Java. I cannot believe a serious application without log. The current log macro is good enough as API, but more appenders are needed.

2 Likes

I believe an actor library/framework similar to Akka (Scala, Java 8) on top of tokio would be helpful for some, particularly people coming from Scala or Erlang.

tokio itself is going in the right direction but it needs more docs/examples and some simpler APIs like the often mentioned async/await
It is a complex topic and the best would be to have something like the Rust Book for tokio/futures. But writing several hundred pages is of course a lot of work!

1 Like

I don't think I'll add anything new to the discussion, but nonetheless let me put a few of my own thoughts down.

  1. conservative_impl_trait needs to be stabilized, which I understand is happening so it's just a matter of time. A lot of futures/tokio code that I've seen is littered with boxed futures. This flies in the face of the "Zero Cost Futures" mantra, and even if futures themselves are zero cost, having to box them for ergonomical/practical reasons negates their efficiency on the whole.

  2. Better/more tokio examples. This has been mentioned multiple times already in this thread, but I'll echo that. In particular, examples of non-trivial scenarios would be helpful. It's a fine line between keeping examples simple enough yet useful, so I realize this requires a careful dance. The socks5 proxy example is pretty close to what I have in mind in terms of scope.

  3. Rust needs that "killer app" (or two :slight_smile:) to drive adoption further. By extension, a server written in Rust, by a team that understands Rust and tokio well, that gets the same type of attention and use as things like nginx, varnish, haproxy, kafka, hadoop, Cassandra/Scylla, etc would give a huge boost to that. That system could also be used as an example of how to use tokio in more complex scenarios. Of course this is a chicken and egg problem since someone would have to overlook the existing limitations/concerns, as discussed in this thread, and build such a thing in Rust in the first place.

  4. Also already mentioned, but the key libraries in the ecosystem that would be used to build a server (of some sort) all need to work on stable.

  5. One more vote for OOM handling. An abort that unconditionally brings down the entire daemon doesn't instill reliability confidence. It could be architected around, but that shouldn't be forced due to this.

  6. Relatedly, allocator APIs probably need to be stable. Inevitably someone will want to optimize/control/introspect memory management and not needing unstable/unsafe code for that would be nice.

  7. To echo @matthieum, lots of discussion thus far has been about http, database clients/ORM-like functionality, etc but raw high performance tcp/udp (uni, multi and broadcast) would be great. It'd be also nice if there were quality bindings to kernel networking bypass stacks, such as dpdk, efvi, and so on. This latter stuff is not on the critical path of course, and if adoption grows, someone will ultimately write such libs.

  8. Finally, just getting Rust adopted and incorporated into an existing corporate setting is challenging. In particular, cargo is great but it's unclear how one would go about using the crates.io ecosystem behind a firewall. Cleanly integrating rust into an existing build system is also a challenge.

15 Likes

I already have a server app in production, but I can tell what annoys me:

  1. Death by a thousand cuts. The biggest time sink with Rust turned not to be language learning or borrow checker
    or new concepts, but .. every little thing. Figuring out how to parse csv file, how to read a file, how to split a string...
    1000 trivial things that for every other language come up easily in search, require some amount of research for Rust.
    Is it in stdlib? Is there a create for it? Which one is the best? How do I use it? How do I achieve the goal?
    Lack of blogs, examples, documentation and so on costs a lot more time than I expected.

  2. Related, but I want to emphasize it. Rust libraries need more polish. Documentation, examples, clear indication of state of commonly used and requested features would help. This should be the focus of Rust team in my opinion.
    Having most common set of libraries on par with equivalents from other languages would help Rust a lot.

  3. Things like Tokio should be in the stdlib long time ago. Otherwise its hard to expect people to use it (some will, some won't).

  4. Web frameworks are lacking. There is nothing supporting HTTP1/2/Websockets out of the box. Documentation is limited and people frequently are confused how to do something (I've spent a lot of time on Iron channel).

5 Likes