Publish dependent crates together (w/o sleep hack)

Yandros · October 17, 2020, 11:49am

I was gonna post this to answer to this post:

but then I thought it was kind of out of topic so I guess a dedicated thread makes more sense.

I noticed:

matklad/xshell/blob/a419eb5/examples/ci.rs#L63


            if tag_exists || &current_branch != "master" { &["--dry-run"][..] } else { &[] };

        if dry_run.is_empty() {
            cmd!("git tag v{version}").run()?;
        }

        let token = env::var("CRATES_IO_TOKEN").unwrap_or("DUMMY_TOKEN".to_string());
        {
            let _p = pushd("xshell-macros")?;
            cmd!("cargo publish --token {token} {dry_run...}").run()?;
            std::thread::sleep(std::time::Duration::from_secs(60));
        }
        cmd!("cargo publish --token {token} {dry_run...}").run()?;
        cmd!("git push --tags {dry_run...}").run()?;
    }
    Ok(())
}

struct Section {
    name: &'static str,
    start: Instant,

So I see I'm not the only one using sleep to "chain" publications when crates depend on each other

The question then is:

do people know of a less hacky way of achieving this?

matklad · October 17, 2020, 11:55am

Lol, thanks for asking, I wanted to do the same, but got distracted

matklad · October 17, 2020, 12:01pm

Some history: I believe a long time ago, publishing API on crates.io was synchronous -- it only return an http response to cargo publish after publishing was fully finished. This then was changed on serve side to async publishing for some technical reason I don't remember.

Riateche · October 17, 2020, 1:51pm

I'd like to share my script for this: gist (requires cargo-add). Usage: sync_publish path_to_crate -f

It creates a temp package using the published crate as a dependency and waits until cargo can successfully generate a lock file for it. That usually takes a few seconds.

As a bonus, it skips already published versions, so you can run it for every package in your workspace (in an outer script), and it will only try to publish the crates when you bump their versions.

ratmice · October 17, 2020, 4:19pm

While I haven't tried it myself, it looks like cargo-release has a similar loop which updates metadata, looks for the published crate, instead of trying to create a temp package in its wait_for_publish function. There is a sleep there, but just for a second presumably to avoid being too busy.

Michael-F-Bryan · October 17, 2020, 4:51pm

Assuming we don't want to make changes to how crates.io the wait_for_publish() function mentioned by @ratmice is probably the best you can do.

The underlying problem is that we don't have a mechanism to let the server asynchronously push events down to us. One possible idea is for crates.io to gain a pub-sub service where you can subscribe to be notified about certain events, then cargo-release and friends would temporarily subscribe to notifications for just that package, do the release, then go to sleep until crates.io says it's been released.

How does docs.rs know when a new package has been published so it can generate docs? Is it just polling for changes every X minutes, too?

matklad · October 17, 2020, 6:18pm

Added worse-is-better version via cargo install

for _ in 0..100 {
    thread::sleep(Duration::from_secs(3));
    let err_msg =
        cmd!("cargo install xshell-macros --version {version} --bin non-existing")
            .ignore_status()
            .read_stderr()?;

    if err_msg.contains("no bin target") {
        break;
    }
}

Nemo157 · October 17, 2020, 9:06pm

Currently yes, every minute (maybe slightly longer, I didn’t check) it updates a local index clone and checks if there are new crates. There’s plans to move that to a webhook, but that’d be a rust-lang internal infrastructure thing.