Coding in Rust on a plane

This is a very different topic than the ones I'm used to post here, but I have an upcoming international trip that will take around 8h and I would love to use some of this time to work on some of my Rust projects.

Since I will have no Internet access during this time, I was reflecting the things I rely the most on my development cycle and, of course, I do a lot of googling (which won't be possible) but I also rely a lot on crates.io and docs.rs.

I am looking to find if someone already attempted something similar and what the struggles were and availability of offline resources for download in preparation (like maybe the X most used crates for crates.io and some downloadable form of docs.rs archive).

Any tips or ideas on how to make something like this possible?

4 Likes

If you already have a project and downloaded its dependencies you can just do cargo doc --open to build and read your docs locally as you would on docs.rs.

7 Likes

Yeah this is very cool, because if I get another answer about an offline partial crates.io archive, I can derive the documentation from it. Thank you!

One way is to likewise have any dependencies already downloaded, e.g. with cargo fetch. While on the plane you can use cargo --offline. If there are other crates you anticipate using, you could make a dummy project relying on them and cargo fetch that.

Here's an article about all that.

Anticipating which crates you'll want is trickier. The playground uses the top 100 crates, by some metric. I have no idea how much space it takes up.

3 Likes

That's very, very interesting. I had no idea that could be done.

It would be lovely to know how much space it takes. I've been thinking about maybe having an external 1TB/2TB SSD just for that archive. Gonna start some googling to see if I find some more useful context.

Thank you!

I should have googled first, but I also wanted this "research" to be persisted somewhere online, so I think there's value on this thread.

I have just found this: GitHub - panamax-rs/panamax: Mirror rustup and crates.io repositories, for offline Rust and cargo usage..

Gonna explore it a bit and see what I can accomplish.

1 Like

OK, so basically panamax is the way to go.

Took me around 2.5-3h to download 132G which contains all the crates and rustup dependecies.

I also found that you can download an offline version of Stack Overflow to use with Kiwix. The whole archive is another 80Gb, but still considering if it's a good idea.

6 Likes

I was interested in your researches here having been a liberal user of offline docs and fetches because of unreliable bush internet. This was made even worse by floods that have washed away much local infrastructure twice this year. I've had several internet outages exceeding a week.

132GB is a lot of data though - it would probably take a couple of days to download, and use up more than half of my 200GB monthly cap! It'd be interesting to know how much the repos change. If panamax permits or makes it easy, would you be so kind as to report back sometime on how much extra data it takes to sync after a suitable interval?

2 Likes

This is a very short period for sure, but I just ran it now to see what would happen.

For context, nightly rust was updated, which took most of the 5m33s the whole process took.

After that, 23,175 items from crates were either updated or added. The initial execution had ~560,000 crate items downloaded, so we had around 5% increase/change in 24h give or take?

Disk usage went from 123GB to 143GB, a whopping 20GB delta.

Here's the starting point again:

Here's the sync output:

And here's the new disk usage report:

Hope this helps! :slight_smile:

Thanks @fcoury - that's interesting.

It tends to confirm what I suspected, which is that my existing targeted approach (anticipating outages as best I can, and selectively downloading as appropriate) is probably more feasible for me than downloading the entire rust corner of the internet ;). At least until I can afford Starlink (unlikely), or floods and fires drive me into an urban area (which is close to an 'over my dead body' kind of deal).

2 Likes

Out of curiosity, how did you go?

Back in September, I spent an entire week camping and fishing along the coast and if you wanted phone reception you'd need to hop in a 4WD and drive 10 minutes out to the cliff.

The only real preparation I did was to download a couple books for reference material, make sure any necessary dependencies were already downloaded (the usual suspects - serde, anyhow, rowan, salsa, clap, etc.), then set net.offline = true in my .cargo/config.toml file so builds wouldn't accidentally try to touch the network. This worked out pretty well because a compiler/VM is typically quite self-contained and I already put an emphasis on a low dependency count to keep my crate's build times snappy.

1 Like

I do not know if any of the off-the-shelf registry mirrors offer it, but a simple way to reduce storage usage of a mirror is to filter what crates you actually provide.

Picking the set of crates to vendor if (it's not just "the set of crates you're already using") is an art, but a reasonable cut would be to only mirror the most recent version in any semver-compatible range.

(e.g. you'd mirror serde 1.0.147 and serde 0.9.15, but not other 1.m.p or 0.9.p versions.)

A slight refinement of that scheme is to now query over your trimmed down registry mirror for any non- caret[1] dependency version and also vendor the top of that range. This should ensure that cargo update/cargo add will always choose versions that you've mirrored the source for.

You could also get all of the lock files and provide those versions such that --locked succeeds, but I suspect that this would result in mirroring enough of the full collection of crates that it wouldn't be worth over just mirroring the full repository.

From the other end, you can of course come up with some "significance" heuristic and only vendor crates which meet that heuristic. The most primitive version of which is just whatever's already in your Cargo.lock.


  1. The ^ver syntax is the "semver-compatible" operator, and the default operator if no operator is used. ↩︎

I wrote zerus for this use case, although it uses the rust http feature. The feature here is you don't need to download all the crates, just the ones you need.

I also added a feature like this to panamax: Use cargo vendor for mirroring selected crates by wcampbell0x2a · Pull Request #71 · panamax-rs/panamax · GitHub

2 Likes

Some fresh stats after a week:

Storage usage before sync

Sync process

Storage usage after sync

2 Likes

I am reminded of stack overflow's infamous "Boat Programming" question (fortunately this is not stack overflow)