I recently attempted to port Mononoke over from using
Runtimes to using
Runtimes, while cleaning up the test cases and fixing some of the tech debt in
hgcli. I have had to bring this work to a close without landing the stack), because the available backward compatibility shims aren't good enough to let me just swap out the
Runtimes, but also require me to rewrite all the code uses other parts of the
tokio Crates in use
In Mononoke, as well as the runtime, we use
tokio_openssl for SSL connections,
tokio_process for subprocesses,
tokio_timer for timers, and
tokio_net for TCP connections. This is not an exhaustive list of
tokio family crates we use, because I've ignored things like
tokio-core which have been folded into other crates in newer
While porting the runtime dependent bits wasn't challenging, the rest caused me various degrees of difficulty:
Underlying all this is that the various
tokio-* crates expect to be able to access a
tokio-0.1 runtime for calls like
tokio::spawn. As a result, if you swap out the runtime to a
tokio-0.2 runtime and add
compat() calls in the right places, the code compiles and runs, but fails at runtime with errors about
tokio being "shutdown".
Easy to handle
tokio_openssl are easy to port to the
tokio-0.2 versions - the API has changed, but for
timer, it's easy to do the conversion, and for
tokio_openssl, there are relatively few call sites that depend on the API directly.
Anything that uses
futures::sink ports in the obvious fashion.
oneshot are simple to port; our internal
FutureExt trait is either obsoleted by the new APIs, or trivial to replace with new code (
async/await and the new
_concurrent combinators in
StreamExt make this particularly easy).
Annoying, but manageable
tokio-0.1 depends on a
tokio-0.1 executor. The
tokio-0.2 version has an incompatible API; however, there are only a small number of users of
tokio_net, so porting is manageable. If we'd made heavier direct use of
tokio_net, the lack of a usable compatibility shim to the traits from
tokio-0.1 that the old
TcpStream implements would have caused fun.
tokio_codec is not significantly changed, but causes ripple-through with our internal
async_compression crate - this can be fixed by using the public
async_compression crate from crates.io.
tokio_process has migrated into
tokio_net (for some reason not clear to me), and is now undocumented as a result. This can be worked around by ample reading of the code, and I assume this will be fixed soon as an oversight (if it's not fixed up in the next alpha release of
tokio-0.2, I will raise an issue in the
tokio issue tracker to ensure that it doesn't get forgotten).
Hard, help needed.
futures::io in the new world have different
AsyncWrite traits; there is no compatibility shim between old (0.1)
Async* traits and new (0.2)
AsyncRead etc traits inherit from their
std::io sync counterparts - the
tokio-0.2 versions do not.
On the compatibility side, the
Async* traits are not compatible with the
Async* traits; as the compatibility shims are between
Async*, this requires a complete rewrite of all I/O code (including things that depend upon
tokio-0.2, and that then ripples through into
And finally, the
AsyncRead implementation does not also implement the various
std::io traits -
Read, etc - so code written against
tokio-0.1 that requires some of the
std::io traits as well as
AsyncWrite cannot be made to work with the compatibility shims.
There are two chunks of work that would make resuming this port to
tokio-0.2 much simpler:
- Write deeper compatibility shims between the
tokio-0.2runtimes, so that I can just swap out
Runtimeand not the other parts of our
tokio-0.1code in order to end up on
tokio-0.2. This would enable
tokio-0.1's versions of
tokio_netetc to run on
- Chase down the discrepancies between
futures-0.3so that I have a full set of compatibility shims available between
std::io::BufReadetc and modern
AsyncBufReadetc, and do not have to choose between
With those two in place, I could resume my efforts, and try to get Mononoke onto a modern
tokio - for now, though, the amount of code to rewrite is too high to be anything other than a whole-team "stop work and rewrite" effort.