Manually vendoring dependencies (work around cargo vendor bug)

help
1

I'm trying to manually vendor some of the dependencies of my project to work around Support `cargo vendor` · Issue #23 · rust-lang/wg-cargo-std-aware · GitHub.

I have a simple script that when passes a package name and a version, downloads the package tarball from crates.io. Using it I was able to vendor all dependencies of my project.

Now the next issue is generating the .cargo-checksum.json files. The package field in that file is the SHA256 of the tarball, so that's easy to generate. However I couldn't figure out how to generate hashes of the files, and which files to include. It seems like hashes of the files are not SHA256, for example, for a package with matching "package" SHA256 (i.e. if I download the tarball and run sha256sum <file> I get the same hash as cargo vendor-generated package hash in .cargo-checksum.json), but if I compute SHA256 hashes of the files, they don't match the ones with cargo vendor-generated file hashes.

So I'm wondering how to generate hashes of files when vendoring.

Secondly, I'm wondering which files to include, and whether having more files listed in files in the .cargo-checksum.json file is a problem. I realized that if I vendor a package with cargo vendor, and fetch the tarball myself using my own script, cargo vendor generated checksum omits some of the files that I see in the tarball downloaded by my script. Is this a problem? If I generate hashes of all files in the tarball, will cargo be able to use it, or do I have to omit some of the files as cargo vendor do?

Thanks.

Here's the script I'm using to download packages from crates.io. Note that this version does not write file hashes yet to .cargo-checksum.json, only writes the package hash (which matches with the hashes generated by cargo vendor):

import os
import ssl
import subprocess
import sys
import urllib.request

def sha256sum(path):
    output = subprocess.run(['sha256sum', path], capture_output=True).stdout
    print(path + " output: " + str(output))
    return output.split()[0]

package_name = sys.argv[1]
package_version = sys.argv[2]

package_url = f'https://crates.io/api/v1/crates/{package_name}/{package_version}/download'
package_file_name = f'{package_name}-{package_version}.gz'

print(f'Vendoring package "{package_name}" version = "{package_version}"')
print(f'Package URL: {package_url}')

urllib.request.urlretrieve(package_url, package_file_name)

print(f'Done downloading package: {package_file_name}')

package_checksum = sha256sum(package_file_name)

print(f'Package checksum: {package_checksum}')

subprocess.run(['tar', '-xvf', package_file_name])

package_dir_path = f'{package_name}-{package_version}'

checksum_json_file_path = f'{package_dir_path}/.cargo-checksum.json'

print(f'Creating file "{checksum_json_file_path}"')

with open(checksum_json_file_path, 'w') as cargo_checksum:
    cargo_checksum.write('{"files":{},"package":"{%s}"}' % package_checksum)
1 Like
2

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.