Can a programmer use Rust and its packages without internet access?

joaocarvalhoopen · February 6, 2022, 10:11pm

Hello,

Today I was looking at the evening news and was seeing that many political commentators think that war is eminent in Ukraine, in my beloved Europe, I live in Portugal. I hope that all the parts talk things out and that war never begins. Historically wars are something that every part involved knows how to start but no part evolved in a war knows how it will end. Normally those are uncertain times. So please give humanity many years of peace, health and prosperity and not war and suffering .

But that makes me thinking about several scenarios, imagine a world without internet, a programmer as to make an income even in a war time, right? And a programmer, what he knows best is how to program. But languages like Rust, Python, Javascript and in certain ways Java, C# and C++, the normal usage of the language bring there packages/libs stuff from crates.io, npm, anaconda packages, right.

My question here is specific to Rust. Can the Rust latest versions of all packages in crates.io be easily downloadable and used from a crates.io disk copy in the case that internet is attacked and goes down?

Has the Rust language any degree of resilience, for the normal and usual programmer, against war times and war chaos?

Thank you,

Best regards,
João

Michael-F-Bryan · February 6, 2022, 11:18pm

Yes. The website has a page dedicated to exactly this.

Accessing the Crates.io Data

There are several ways of accessing the Crates.io data. You should try the options in the order listed.

The crates.io index. This git repository is updated by crates.io, and it is used by Cargo to speed up local dependency resolution. It contains the majority of the data exposed by crates.io and is cheap to clone and get updates.

The database dumps (experimental). The dump contains all information exposed by the API in a single download. It is updated every 24 hours. The latest dump is available at the address https://static.crates.io/db-dump.tar.gz. Information on using the dump is contained in the tarball. You can find the changelog for the database dumps in GitHub Issue #3617.

Crawl the crates.io API. This should be used as a last resort, and doing so is subject to our crawling policy. If the index and the database dumps do not satisfy your needs, we're happy to discuss solutions that don't require you to crawl the registry. You can email us at help@crates.io.

In particular you will probably want to check out that db-dump.tar.gz link.

If you just want to download a single crate and not half the crates.io registry (which will probably get you banned/rate limited because it's rude to waste all their bandwidth), metadata for the entire crates.io registry is available on GitHub.

At the top level you'll find a config.json file with a download URL that follows this format.

github.com

rust-lang/crates.io-index/blob/f46a475ef8ad16c6bbb4087cd66a2ba90e588337/config.json#L1-L4


      
          {
            "dl": "https://crates.io/api/v1/crates",
            "api": "https://crates.io"
          }

When you dive into the folders you'll see a JSON file that contains dependency information and a checksum for a particular crate.

github.com

rust-lang/crates.io-index/blob/b05a351819c326054b75dde7f2f9fb7a4f8bac6f/se/rd/serde#L222


      
          {"name":"serde","vers":"1.0.126","deps":[{"name":"serde_derive","req":"=1.0.126","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"ec7505abeacaec74ae4778d9d9328fe5a5d04253220a85c4ee022239fc996d03","features":{"unstable":[],"alloc":[],"derive":["serde_derive"],"rc":[],"std":[],"default":["std"]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.127","deps":[{"name":"serde_derive","req":"=1.0.127","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"f03b9878abf6d14e6779d3f24f07b2cfa90352cfec4acc5aab8f1ac7f146fae8","features":{"alloc":[],"derive":["serde_derive"],"rc":[],"default":["std"],"unstable":[],"std":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.128","deps":[{"name":"serde_derive","req":"=1.0.128","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"1056a0db1978e9dbf0f6e4fca677f6f9143dc1c19de346f22cac23e422196834","features":{"std":[],"unstable":[],"alloc":[],"default":["std"],"derive":["serde_derive"],"rc":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.129","deps":[{"name":"serde_derive","req":"=1.0.129","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"d1f72836d2aa753853178eda473a3b9d8e4eefdaf20523b919677e6de489f8f1","features":{"alloc":[],"derive":["serde_derive"],"default":["std"],"rc":[],"std":[],"unstable":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.130","deps":[{"name":"serde_derive","req":"=1.0.130","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"f12d06de37cf59146fbdecab66aa99f9fe4f78722e3607577a5375d66bd0c913","features":{"unstable":[],"default":["std"],"derive":["serde_derive"],"rc":[],"alloc":[],"std":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.131","deps":[{"name":"serde_derive","req":"=1.0.131","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"b4ad69dfbd3e45369132cc64e6748c2d65cdfb001a2b1c232d128b4ad60561c1","features":{"alloc":[],"derive":["serde_derive"],"rc":[],"default":["std"],"std":[],"unstable":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.132","deps":[{"name":"serde_derive","req":"=1.0.132","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"8b9875c23cf305cd1fd7eb77234cbb705f21ea6a72c637a5c6db5fe4b8e7f008","features":{"derive":["serde_derive"],"rc":[],"unstable":[],"alloc":[],"std":[],"default":["std"]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.133","deps":[{"name":"serde_derive","req":"=1.0.133","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"97565067517b60e2d1ea8b268e59ce036de907ac523ad83a0475da04e818989a","features":{"default":["std"],"unstable":[],"rc":[],"alloc":[],"std":[],"derive":["serde_derive"]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.134","deps":[{"name":"serde_derive","req":"=1.0.134","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"96b3c34c1690edf8174f5b289a336ab03f568a4460d8c6df75f2f3a692b3bc6a","features":{"unstable":[],"default":["std"],"alloc":[],"derive":["serde_derive"],"rc":[],"std":[]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.135","deps":[{"name":"serde_derive","req":"=1.0.135","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"2cf9235533494ea2ddcdb794665461814781c53f19d87b76e571a1c35acbad2b","features":{"derive":["serde_derive"],"alloc":[],"rc":[],"std":[],"unstable":[],"default":["std"]},"yanked":false,"links":null}
          {"name":"serde","vers":"1.0.136","deps":[{"name":"serde_derive","req":"=1.0.136","features":[],"optional":true,"default_features":true,"target":null,"kind":"normal"},{"name":"serde_derive","req":"^1.0","features":[],"optional":false,"default_features":true,"target":null,"kind":"dev"}],"cksum":"ce31e24b01e1e524df96f1c2fdd054405f8d7376249a5110886fb4b658484789","features":{"alloc":[],"default":["std"],"derive":["serde_derive"],"rc":[],"std":[],"unstable":[]},"yanked":false,"links":null}

This file has been truncated. show original

Using serde as an example, the latest version is 1.0.136 and the dl URL in config.json doesn't contain any replacement strings, therefore we can download the crate with

$ curl -L https://crates.io/api/v1/crates/serde/1.0.136/download > serde-1.0.136.crate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 76158  100 76158    0     0  84002      0 --:--:-- --:--:-- --:--:-- 84002

$ file serde-1.0.136.crate
serde-1.0.136.crate: gzip compressed data, was "serde-1.0.136.crate", max compression, original size modulo 2^32 525312

The *.crate file is just a gzipped tarball that we can extract.

$ tar -xvf serde-1.0.136.crate
serde-1.0.136/.cargo_vcs_info.json
serde-1.0.136/Cargo.toml
serde-1.0.136/Cargo.toml.orig
serde-1.0.136/LICENSE-APACHE
serde-1.0.136/LICENSE-MIT
serde-1.0.136/README.md
serde-1.0.136/build.rs
serde-1.0.136/crates-io.md
serde-1.0.136/src/de/format.rs
serde-1.0.136/src/de/ignored_any.rs
serde-1.0.136/src/de/impls.rs
serde-1.0.136/src/de/mod.rs
serde-1.0.136/src/de/seed.rs
serde-1.0.136/src/de/utf8.rs
serde-1.0.136/src/de/value.rs
serde-1.0.136/src/integer128.rs
serde-1.0.136/src/lib.rs
serde-1.0.136/src/macros.rs
serde-1.0.136/src/private/de.rs
serde-1.0.136/src/private/doc.rs
serde-1.0.136/src/private/mod.rs
serde-1.0.136/src/private/ser.rs
serde-1.0.136/src/private/size_hint.rs
serde-1.0.136/src/ser/fmt.rs
serde-1.0.136/src/ser/impls.rs
serde-1.0.136/src/ser/impossible.rs
serde-1.0.136/src/ser/mod.rs
serde-1.0.136/src/std_error.rs

CAD97 · February 6, 2022, 11:30pm

Some old blog posts which might also be interesting:

https://gmjosack.github.io/posts/dissecting-cratesio-minimum-mirror/

crates-io also has a doc on setting up a mirror:

github.com

rust-lang/crates.io/blob/master/docs/MIRROR.md

# Deploying & using a mirror

**DISCLAIMER: The process of setting up a mirror is a work-in-progress and is
likely to change. It is not currently recommended for mission-critical
production use. It also requires Cargo from Rust distribution 1.12.0 or
later.**

## Current functionality: a read-only, download-API-only mirror

This mirror will function as a read-only duplicate of crates.io's API. You will
be able to download crates using your index and your mirror, but the crate files
will still come from crates.io's S3 storage.

Your mirror will not:

- Allow users to sign up/sign in
- Allow crate publish
- Keep track of any statistics
- Display available crates in its UI

This file has been truncated. show original

And there's a crate that claims to set up a caching mirror (downloads when-first-needed, doesn't grab everything up front):

Also, cargo vendor for a specific dependency tree.

joaocarvalhoopen · February 7, 2022, 12:00am

Tank you both, I will study all the info that you sent me.

Best regards,
João

system · May 8, 2022, 12:01am

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.

Topic		Replies	Views
Crates.io is down community	6	854	June 22, 2021
Offline Rust install	3	7440	January 12, 2023
Does Rust support non-internet connected compilation? help	2	726	January 12, 2023
Question about crates.io	6	627	April 21, 2022
Would an internal crates server make your company able to adopt Rust? Integer 32 wants to know!	8	1197	January 12, 2023

Can a programmer use Rust and its packages without internet access?

Accessing the Crates.io Data

Related Topics