Using Crates in offline environment

#1

Hi everyone,

I am new to Rust programming and I cannot figure out how to use crates in offline environment. For some security reasons, I have no connection to internet. What I would like to do is, download crates from GitHub and want to give it as a path dependency to my Cargo.toml file. I thought, this is working because I have tried it for "libc" crate. I download the libc crate from github (GitHub - rust-lang/libc: Raw bindings to platform APIs for Rust). Then, give the path to Cargo.toml file under dependency title. And it worked.

[dependencies]
libc = {path = "/home/CRATES/libc"}

However, when I tried to use "quick-xml" and "serde" crates, this approach seems useless. In my opinion, the problem was each crate has different dependencies. I have download every crate that dependent both to quick-xml and serde crates and put them into CRATES folder. Then give the path in Cargo.toml file but it was still trying to connect the crates.io.

[dependencies]
quick-xml = {path = "/home/CRATES/quick-xml"}
serde = {path = "/home/CRATES/serde/serde"}

Then I add .cargo/config.toml to my root folder ($CARGO_MANIFEST_DIR of my project) and change the source as follows.

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "home/CRATES"

And cargo gives error of no such file or directory about a file inside a crate folder .cargo-checksum.json. I cant give the name which crate folder it is because every time I play with Cargo.toml file it changes. In addition, I cannot also use cargo vendor since It is also trying to connect internet .

So simplify my question; Is there any way to use the crates by downloading them from GitHub and use them by editing Cargo.toml file of my project. If it possible can you guys show me how to? If it is not possible, can you suggest any other way to use crates without connecting internet at all. Any help would welcome. I appreciate for your time.

#2

The best way to prepare a full downloaded set of crates is to run cargo vendor and add a directory source to use it.

6 Likes
#3

When I try to cargo vendor, it is still trying to connect to crates.io where I cannot. From my experiments, to vendor a project, it requires *.crate files under the folder /home/.cargo/registry/cache. What I mean is if *.crate files are exists under cache folder then cargo vendor --offline is working. However if these files are not present, then vendoring isnt working. Is it possible to vendor crates that I downloaded from github without connecting crates.io ? Or Do I have to connect to internet once and then go to offline?

#4

If you want to replace upstream crates with Git repositories, you’re going to need to use patch directives.

#5

To use external crate you need to download it in some way. Once you've downloaded it you can move them freely as local files/directories.

It is not. Let's vendor some sample project with extreme dependency freshly made from cargo new vdr.

Cargo.toml

[package]
name = "vdr"
version = "0.1.0"
edition = "2021"

[dependencies]
extreme = "666"

Terminal

$ cargo vendor
    Updating crates.io index
   Vendoring extreme v666.666.666666 (/path/to/home/.cargo/registry/src/github.com-1ecc6299db9ec823/extreme-666.666.666666) to vendor/extreme
To use vendored sources, add this to your .cargo/config.toml for this project:

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"

$ ls
Cargo.lock  Cargo.toml  src  vendor

$ ls vendor
extreme

$ mkdir .cargo
$ touch .cargo/config.toml

Note that the .cargo directory containing config.toml file should be placed at the next to the project's Cargo.toml file.

.cargo/config.toml

[source.crates-io]
replace-with = "vendored-sources"

[source.vendored-sources]
directory = "vendor"

And it's done! Copy your project directory including the vendor directory which contains all your dependencies.

1 Like
#6

Another interesting and working project for downloading the entire crates.io is GitHub - panamax-rs/panamax: Mirror rustup and crates.io repositories, for offline Rust and cargo usage.

#7

I have tried it already. First I added my two main dependencies as follows to Cargo.toml file

[dependencies]
quick-xml = { version="0.23.0",  features=["serialize"] }
serde = {version = "1.0.136",  features = [ "derive" ]}

[patch.crates-io]
quick-xml = { path = "/home/CRATES/quick-xml" }
serde = { path = "/home/CRATES/serde/serde" }

Then it returns the following message,

failed to get memchr as a dependency of package quick-xml v0.23.0 (/home/CRATES/quick-xml)
... which satisfies dependency quick-xml = "^0.23.0" (locked to 0.23.0) of package xml_project v0.1.0 (/home/byilmaz/Desktop/Projects/xml_project)

Caused by:
failed to load source for dependency memchr

Caused by:
Unable to update registry crates-io

Caused by:
failed to fetch https://github.com/rust-lang/crates.io-index

Then I update my Cargo.toml file and add the memchr dependency.

[dependencies]
quick-xml = { version = "0.23.0", features = [ "serialize" ] }
serde = {version = "1.0.136",  features = [ "derive" ]}
memchr = "2.4.1"

[patch.crates-io]
quick-xml = { path = "/home/CRATES/quick-xml" }
serde = { path = "/home/CRATES/serde/serde" }
memchr = { path = "/home/CRATES/memchr" }

Then it yields same error message for "proc-macro2" crate. So, I did the same for proc-macro2. Then the error message came up for quote crate. So, I also added quote crate to my Cargo.toml file. Final version of my Cargo.toml file is as follows

[dependencies]
quick-xml = { version = "0.23.0", features = [ "serialize" ] }
serde = {version = "1.0.136",  features = [ "derive" ]}
memchr = "2.4.1"
proc-macro2 = "1.0.36"
qutoe = "1.0.15"

[patch.crates-io]
quick-xml = { path = "/home/CRATES/quick-xml" }
serde = { path = "/home/CRATES/serde/serde" }
memchr = { path = "/home/CRATES/memchr" }
proc-macro2 = { path = "/home/CRATES/proc-macro2" }
quote = { path = "/home/CRATES/quote" }

Then this will return following error message;
failed to get proc-macro2 as a dependency of package xml_project v0.1.0 (/home/byilmaz/Desktop/Projects/xml_project)

Caused by:
failed to load source for dependency proc-macro2

Caused by:
Unable to update registry crates-io

Caused by:
failed to fetch https://github.com/rust-lang/crates.io-index

What I have noticed is that, at this final error message, the missing dependencies are for the crates that I want to implement on the project. But for this error, it couldnt find "proc-macro2" crate which is dependent for my project and I already add it to my Cargo.toml file. What am I missing? Am I try to do something silly?

#8

Well, I actually didn't try to vendor a crate which does not have any dependency. As I checked from crates-io this crate does not have any dependency. Can you manage to vendor the crates which is dependent to other crates? If you can can you make an example for it also ?

#9

The thing that confuses me is that I’m not sure if you really want to download from crates.io, or from GitHub?

If you want to download them from crates.io, then it should be easy. The cargo vendor command does all the work for you, downloading .crate files into the cache just like you described.

If you want to download from GitHub, it’ll be harder, because version numbers aren’t necessarily unique on there, you have to check out each crate yourself.

#10

Nothing different. Dependencies of dependencies only make the output from the commands slightly longer. All you need to do remains identical. You can try with deps like serde_json = "1.0" which has its own dependencies.

#11

Well actually, I just examine your output. It says that it is taking the crate from your $HOME/.cargo/registry folder and copy it to your $CARGO_MANIFEST_DIR/vendor folder. Therefore, the crate was downloaded once. As I told I can not connect to internet. So, the crate was not present at that folder. What I did was download the crate from github and give the path and it worked because this crate has no dependency. Now, I tried the dependency you said 'serde_json', it yields the same errors that I mentioned above.

#12

I am sorry if I say something meaningless, I am just new to Rust. What I want to use is GitHub repos. I cannot download anything from crates.io (*.crate files etc.).

What I try so far was downloading crates from GitHub and put them into CRATES folder. Then I edit Cargo.toml file such as;

  • Giving paths of each crate under [dependency] section which still tries to connect to crates.io (which I don't want and get the error).

  • Giving the dependencies with their versions under [dependency] section and give the paths of crates under [path.crates-io] section which again tries to connect to crates-io (which I don't want and get the error).

Then I create .cargo/config.toml file at my manifest directory and do the source replacement and give the path of my CRATES folder as a vendor path. However, when I try to cargo run It says that .cargo-checksum.json file cannot find. I think the reason of that is I didn't use cargo vendor command in my project so these files are not created at all. When I try to cargo vendor my project, it again tries to connect to crates-io (which I don't want and get the error).

So I am not figured out how to use the crates (which has dependencies to other crates) just download them from GitHub and use it locally. I don't know if I should edit the Cargo.toml files of the crates that I downloaded from GitHub.

#13

If you're not connected to the internet, how did you download crates from the github?

I guess you have 2 machines - one has internet connection but you can't code on it, and one without internet connection and where your code lives. Like downloading from the github, the vendoring step only requires the Cargo.toml file to list required dependencies, and should be performed in the machine with internet access. After the vendoring you have a directory with all the files needed. Copy them to the machine with your code.

4 Likes
#14

I understand that you want to be able to work on it offline. I don't understand what use case you have that allows you to connect to GitHub, but not to crates.io.

Is there a special problem that prevents you from connecting to crates.io, but allows you to connect to GitHub? Or are you just trying to prepare all the crates you need so that you can do work for an extended time without an internet connection?

#15

Exactly. I can download the crates from GitHub but cargo install doesn't work in my coding environment. This may solve the problem, but when I want to add a new crate at the middle of the project this all process should done again and again which will not something I want actually.

#16

Yes. My working environment has no internet connection. I download the crates from GitHub and put them into my coding machine via USB. So I want to download all dependencies from GitHub and put them into my working environment and work offline.

#17

Well I think that's one of the fundamental problem of your environment. If you want new code from the internet, you need to download it. And the download should always happen from the internet pc and manually transferred to the code pc. Unless you can automate this manual part with whatever side channel exploit including audio jack, screen-camera pair or bluetooth/wifi direct.. but it doesn't seems you're allowed to do so.

#18

Wanting an airgapped build environment seems pretty reasonable.

But if you want to download all the dependencies, you're better off grabbing them using Cargo. Basically, the steps are:

  • Install a Rust toolchain onto the computer with Internet access.
  • Use cargo vendor to generate a vendor folder. You'll need the Cargo.toml file to actually specify the dependencies to download, but you won't need any of the actual code.
  • Copy that vendor folder to the airgapped machine, and patch its Cargo.toml file to use it.

And that should do it. It's basically how it's done in applications like Firefox with isolated build environments, and it should work for you, too.

7 Likes
#19

So, if this is the only solution, should I copy the folders $HOME/.cargo/register/src (which has the crate folders) and $HOME/.cargo/cache (which has the *.crates files) to the airgapped machine ?

#20

As far as I understand its documentation

cargo vendor - The Cargo Book

you only need to copy the folder that cargo vendor creates, by default called “vendor”, but you can pass any other path if you want it in a different location; and you need to modify the .cargo/config.toml accordingly in the project on the offline machine.

To add or change a dependency later, transfer the updated Cargo.toml back to the online machine, run cargo vendor again, and transfer the new vendor folder back. (The config.toml changes should stay the same.)

2 Likes