Checklist for upgrading a project from tokio 0.1?

#1

Hello!

I recently found myself wanting to use a library that I'd need weeks if not months to reimplement myself. It's very well-written and async (which is what I am after; it involves sending requests to servers and waiting for events from the server: meaning one request can easily net anywhere from one to 200 events so there's no direct mapping) but is using tokio 0.1.

I have done a fair amount of reading and it seems that upgrading even to tokio 0.2 is not an easy thing to do and it involves:

  • Adding / removing / changing deps (e.g. remove the tokio-io crate);
  • Using the await / async syntax (currently it's not used in the library at all);
  • Changing features used of the new crates (I think? Probably the full tokio feature could suffice);
  • Others (?)

My question to you, and if you have it handy: can you please point me at a check-list of items to cover for upgrading a project from tokio 0.1 to 0.2 -- or directly to 1.X?

I already started sifting through projects in GitHub with related PRs and commits but so far I am only finding things like "introduce tokio 0.1" or "upgrade to tokio 1.0 from 0.2" which are not what I am looking for. I'll keep looking.

If you have something lying in your history or bookmarks that can help, I'll be very grateful!

Thank you, and have a lovely weekend.

#2

A data point which might not be directly helpful, but involves most items you've mentioned: two years ago, I ported my own LDAP crate from tokio-core (the predecessor of tokio 0.1) to tokio 0.2. I did it by throwing out all I/O code and rewriting it, keeping the protocol data structures and frame de- and encoding for the most part. The old chain-of-futures structuring is too alien compared to async/await to be salvageable, IMO. The bulk of porting work took a week or two, so it wasn't that bad.

Out of curiosity, what library are you trying to port?

ETA: I believe it would be best to port directly to tokio 1.0, which is the current and actively maintained version.

1 Like
#3

Tokio 0.1 is from before async await, and things have changed drastically since then. I don't think there's an easy check list for this anywhere.

#4

Tokio 0.1 upgrades are non-trivial. That said, you may consider doing the following:

  1. Migrate to tokio 0.2 with help of tokio-compat. At this point focus on getting your application running at all with tokio 0.2.

  2. Once you manage to get your application running, gradually migrate parts from futures 0.1 to standard library futures by updating your async dependencies to tokio 0.2 compatible versions and replacing future combinators with async/await.

    You may find versions page on lib.rs to be useful for determining which version of a given library introduced a tokio 0.2 dependency, for instance All releases of Reqwest // Lib.rs shows that first version of reqwest that used 0.2 was 0.10.0.

  3. Once you remove all usages of tokio 0.1, migrate to tokio 1 - skip 0.3. It may be as simple as updating dependencies to tokio 1 compatible versions and enabling necessary features, however in event you encounter any issues you can use tokio-compat-02.

1 Like
#5

For a data science project I would like to experiment with super parallel ingestion and basic level of analysis of stock data: GitHub - dailypips/async_ib: Tws api rust bind (that's basically a Rust port of the Python TWS API from InteractiveBrokers). There are old-school threaded libraries out there but they do have strange limits (like maximum connections to a server which is understandable but doesn't help me) and I want much more parallelism. Rust+tokio was a super obvious candidate for such a hobby project so here we go.

Plus I really want to learn Rust+tokio fully and to me that's a good way. Finally, I am a fan of evolving projects so upgrading from tokio 0.1 felt like a perfect exercise.

Basically, the stars aligned for me in my pursuit of a data science + learning project and I am willing to see it through to the end (I am doing it in my free time so no rush too).

I presume this is the commit you're mentioning?

#6

That's what I feared, thanks for confirming.

#7

That's extremely helpful, thank you!

I thought of migrating the futures crate from 0.1 to 0.3 together with upgrading tokio from 0.1 to 0.2 -- and you say that's not needed? Just use those in the stdlib?

#8

This is very rough and not very scientific:

cloc on that repo reports 8340 lines of Rust
cloc on the src/{domain, message} reports 7537 lines of Rust

a superficial less / grep on the src/{domain, message} directory shows:

  • no async
  • no await
  • a minor dependence on tokio for encoder / decoder

This seems to suggest there's ~7537 lines of IB datatypes that you can just use and at most < 800 lines of tokio 0.1 / networking / async code.

At this point, I'd just echo the suggestion of rewriting the remaining 800 lines in modern tokio.

Probably something like: 1) play with the modern tokio {tcp, udp} examples, 2) try to re-implement the remaining 800 lines.

This seems like the type of problem the type checker will really help you out and just "fix one more compiler error/warning" will take you quite far.

Cheers!

1 Like
#9

Most of the time futures is not really necessary now that async and .await exist. That said, migrating to futures 0.3 can be done, note however that the API between futures 0.1 and futures 0.3 did change a lot, so the migration isn't as straightforward as you would expect. Personally I would rather migrate to async/await than to a slightly different futures combinators API. This also has an advantage of making it apparent what was migrated as async/await looks very different to futures 0.1 combinators.

Of course, if you have Streams or Sinks in your program, they will have to be migrated to use those from futures 0.3 as standard library doesn't have Streams or Sinks. compat feature of futures can help gradually migrate those.

#10

That's the kind of advice I need here. Thank you!

Not sure how the type checker will help me exactly because the errors it spits out are confusing me but I guess trial and error will get me there.

#11

Okay, that's a weakness in my knowledge: haven't studied the combinators yet.

#12

I would try something like this:

  1. rip out the dependencies on tokio / async / futures in Cargo.toml
  2. get lots of compile errors
  3. replace all the async stuff with sync. i.e. : TcpStream in std::net - Rust channel in std::sync::mpsc - Rust

At the cost of losing lots of performance, this should be fairly straight forward and provide deep insights on how the IO / networking part of the code base works. Step (3) can almost be done mechanically: what does async struct X match to in sync Rust ? How do I fix this compile error?

Then:

  1. rewrite the sync stuff with latest tokio

So basically tokio 0.1 -> sync Rust -> latest tokio instead of tokio 0.1 -> latest tokio

1 Like
#13

That is... actually an excellent idea. Thank you again!

I'll do my best to start and report on my progress before this thread gets auto-locked -- but since it's a passion project and I have to progress at $dayjob in the meantime, no promises. I'll make sure to post after I finish however, one way or another.

1 Like