Manully Download Crate from crates.io

help
22

How does that work? If the browser uses a proxy, you may be able to tell cargo to use the same proxy? Look at the [http] section in Configuration - The Cargo Book .

2 Likes
23

I don't know if that works, but if it does, it'd certainly be against Company policy so I'd rather not

24

Unfortunately, using cargo fetch isn't sufficient — it won't download the dev-dependencies, which the registry is required to have even if you'll never need them. This feels like an artificial limitation of Cargo, as it shouldn't care about the dev-dependencies, but that's the case today.

I honestly don't know the right solution here, because I'm pretty sure that the "right" solution requires reimplementing a reasonable chunk of Cargo's resolver in order to figure out which dependencies (regular, and dev) are needed for a given version and then doing that recursively and iteratively.

You could probably get 80% of the way by grabbing the latest version of a crate and then assuming the latest version of all the dependencies, but that will break as soon as something needs an older version. I even started a little bash script to do this and quickly hit limitations around renamed crates.

It seems like the hard part is figuring out the list of crates to download to put into your mirror.

1 Like
25

Ah, I think it might actually be fine, but there was a subtle edge case in how Cargo deals with crates with features that enable features of thier dev-dependencies. I've made some changes to Margo to address that, which should be available if you grab version 0.1.5. With those changes, I did the same process as before:

  1. Create a new repo
  2. Add a crate to it with cargo add (I chose serde_json)
  3. Set an alternate CARGO_HOME directory
  4. Run cargo fetch
  5. Get the crates from the alternate CARGO_HOME and add them to a Margo registry.

Then I set up my usage crate with a .cargo/config.toml to use Margo instead of crates.io:

[registries]
my-awesome-registry = { index = "sparse+http://[::1]:9999/" }

[source.crates-io]
replace-with = "my-awesome-registry"

With that, I was able to build:

% cargo build
    Updating `my-awesome-registry` index
  Downloaded itoa v1.0.11 (registry `my-awesome-registry`)
  Downloaded memchr v2.7.4 (registry `my-awesome-registry`)
  Downloaded ryu v1.0.18 (registry `my-awesome-registry`)
  Downloaded serde v1.0.209 (registry `my-awesome-registry`)
  Downloaded serde_json v1.0.127 (registry `my-awesome-registry`)
26

For posterity, this isn't fully accurate, but it is something that crates.io will enforce (and Margo doesn't). As a bit of exploring, I checked to see how many crates are needed to have serde_json exist on crates.io. This means finding every dependency recursively — regular, build-, and dev-dependencies. The grand total is 2109 crates, including some of my own!

Thank goodness that we only have to build 5 or so crates to use it in a normal case :wink:

5 Likes
27

may be useful. please do not put it in your project directory, you should understand this code first before using it because I will not be responsible if you experience problems because of this bash script because I put it in the folder that stores crates "sdk/crates.sh"

#!/bin/bash

#
# sdk/crates.sh
#

CURRENT_PATH=$(pwd)

cd $(dirname $(readlink -f $0))

find . -maxdepth 1 -type d -exec rm -rf {} +
find . -maxdepth 1 -type f \( ! -name '.sh' ! -name '*.sh' \) -exec rm -rf {} +

CRATES=(
    'tokio' # Latest
    'tokio=1.0.0' # Specific version
    'serde' # Latest
    'serde_json' # Latest
    'reqwest' # Latest
    'rayon' # Latest
    'actix-web' # Latest
    'sqlx' # Latest
    'async-std' # Latest
    'chrono' # Latest
    'hyper' # Latest
    'websocket' # Latest
    'libc' # Latest
)

for i in ${CRATES[@]}; do
    if [[ $i == *"="* ]]; then
        V=${i##*=}
    else
        V=$(curl -s https://crates.io/api/v1/crates/${i%%=*} | grep -oP '"num":\s*"\K[0-9.]+(?=")' | head -n 1)
    fi
    wget -c -O ${i%%=*}.crate https://crates.io/api/v1/crates/${i%%=*}/$V/download
done

find . -maxdepth 1 -type f -name '*.crate' -exec tar -xzf {} \;
find . -maxdepth 1 -type d | while read dir; do
    mv $dir ${dir%-*} || echo "Failed to rename $dir"
done

# remove garbage (optional)
find . -type f \( ! -name '*.rs' ! -name '*.toml' ! -name '.sh' ! -name '*.sh' \) -exec rm -rf {} +
find . -type l -exec rm -f {} +
find . -type d -empty -print -delete
find . -type d \( -name 'examples' -or -name '*examples*' -or -name 'tests' -or -name '*tests*' -or -name 'docs' -or -name '*docs*' \) -exec rm -rf {} +

# updating crates dependencies
cargo update --verbose

cd $CURRENT_PATH

manual from browser
https://crates.io/api/v1/crates/{NAME}/{VERSION}/download

override dependencies

[patch.crates-io]
actix-web  = { path = 'sdk/actix-web',  default-features = false }
chrono     = { path = 'sdk/chrono',     default-features = false }
hyper      = { path = 'sdk/hyper',      default-features = false }
libc       = { path = 'sdk/libc',       default-features = false }
rayon      = { path = 'sdk/rayon',      default-features = false }
reqwest    = { path = 'sdk/reqwest',    default-features = false }
serde      = { path = 'sdk/serde',      default-features = false }
serde_json = { path = 'sdk/serde_json', default-features = false }
sqlx       = { path = 'sdk/sqlx',       default-features = false }
tokio      = { path = 'sdk/tokio',      default-features = false }
websocket  = { path = 'sdk/websocket',  default-features = false }