I recently found myself wanting to use a library that I'd need weeks if not months to reimplement myself. It's very well-written and async (which is what I am after; it involves sending requests to servers and waiting for events from the server: meaning one request can easily net anywhere from one to 200 events so there's no direct mapping) but is using tokio 0.1.
I have done a fair amount of reading and it seems that upgrading even to tokio 0.2 is not an easy thing to do and it involves:
Using the await / async syntax (currently it's not used in the library at all);
Changing features used of the new crates (I think? Probably the full tokio feature could suffice);
Others (?)
My question to you, and if you have it handy: can you please point me at a check-list of items to cover for upgrading a project from tokio 0.1 to 0.2 -- or directly to 1.X?
I already started sifting through projects in GitHub with related PRs and commits but so far I am only finding things like "introduce tokio 0.1" or "upgrade to tokio 1.0 from 0.2" which are not what I am looking for. I'll keep looking.
If you have something lying in your history or bookmarks that can help, I'll be very grateful!
A data point which might not be directly helpful, but involves most items you've mentioned: two years ago, I ported my own LDAP crate from tokio-core (the predecessor of tokio 0.1) to tokio 0.2. I did it by throwing out all I/O code and rewriting it, keeping the protocol data structures and frame de- and encoding for the most part. The old chain-of-futures structuring is too alien compared to async/await to be salvageable, IMO. The bulk of porting work took a week or two, so it wasn't that bad.
Out of curiosity, what library are you trying to port?
ETA: I believe it would be best to port directly to tokio 1.0, which is the current and actively maintained version.
Tokio 0.1 upgrades are non-trivial. That said, you may consider doing the following:
Migrate to tokio 0.2 with help of tokio-compat. At this point focus on getting your application running at all with tokio 0.2.
Once you manage to get your application running, gradually migrate parts from futures 0.1 to standard library futures by updating your async dependencies to tokio 0.2 compatible versions and replacing future combinators with async/await.
You may find versions page on lib.rs to be useful for determining which version of a given library introduced a tokio 0.2 dependency, for instance https://lib.rs/crates/reqwest/versions shows that first version of reqwest that used 0.2 was 0.10.0.
Once you remove all usages of tokio 0.1, migrate to tokio 1 - skip 0.3. It may be as simple as updating dependencies to tokio 1 compatible versions and enabling necessary features, however in event you encounter any issues you can use tokio-compat-02.
For a data science project I would like to experiment with super parallel ingestion and basic level of analysis of stock data: GitHub - dailypips/async_ib: Tws api rust bind (that's basically a Rust port of the Python TWS API from InteractiveBrokers). There are old-school threaded libraries out there but they do have strange limits (like maximum connections to a server which is understandable but doesn't help me) and I want much more parallelism. Rust+tokio was a super obvious candidate for such a hobby project so here we go.
Plus I really want to learn Rust+tokio fully and to me that's a good way. Finally, I am a fan of evolving projects so upgrading from tokio 0.1 felt like a perfect exercise.
Basically, the stars aligned for me in my pursuit of a data science + learning project and I am willing to see it through to the end (I am doing it in my free time so no rush too).
I thought of migrating the futures crate from 0.1 to 0.3 together with upgrading tokio from 0.1 to 0.2 -- and you say that's not needed? Just use those in the stdlib?
Most of the time futures is not really necessary now that async and .await exist. That said, migrating to futures 0.3 can be done, note however that the API between futures 0.1 and futures 0.3 did change a lot, so the migration isn't as straightforward as you would expect. Personally I would rather migrate to async/await than to a slightly different futures combinators API. This also has an advantage of making it apparent what was migrated as async/await looks very different to futures 0.1 combinators.
Of course, if you have Streams or Sinks in your program, they will have to be migrated to use those from futures 0.3 as standard library doesn't have Streams or Sinks. compat feature of futures can help gradually migrate those.
At the cost of losing lots of performance, this should be fairly straight forward and provide deep insights on how the IO / networking part of the code base works. Step (3) can almost be done mechanically: what does async struct X match to in sync Rust ? How do I fix this compile error?
Then:
rewrite the sync stuff with latest tokio
So basically tokio 0.1 -> sync Rust -> latest tokio instead of tokio 0.1 -> latest tokio
That is... actually an excellent idea. Thank you again!
I'll do my best to start and report on my progress before this thread gets auto-locked -- but since it's a passion project and I have to progress at $dayjob in the meantime, no promises. I'll make sure to post after I finish however, one way or another.