Why is cargo patch not delivering consistent results? What is broken in this example?

Hi guys :slight_smile:

So I'm working on this project that uses relatively complex source code, and I'm trying to understand why when patching one dependency by using cloned code in the same branch it's causing errors. It logically doesn't add up.

Can someone help me understand whether this is a bug in rust or something wrong I'm doing?

Here's how to reproduce the issue:

Build the project to ensure it works fine:

git clone https://github.com/PureStake/moonbeam/ && cd moonbeam/
cd node/standalone/
cargo build

this builds fine. Great. Now let's patch the "frontier" components:

mkdir temp && cd temp
git clone https://github.com/purestake/frontier && cd frontier
git checkout v0.2-hotfixes  # switch to the branch used in Cargo.toml
cd ../../  # go back to Cargo.toml directory

Now add these lines to Cargo.toml (in the one in node/standalone/) to use the patch we cloned above:

[patch."https://github.com/purestake/frontier"]
ethereum = { package = "pallet-ethereum", path = "temp/frontier/frame/ethereum" }
frontier-consensus = { path = "temp/frontier/consensus" }
frontier-rpc = { path = "temp/frontier/rpc" }
frontier-rpc-primitives = { path = "temp/frontier/rpc/primitives" }

Now build with cargo build, and it builds fine.

However, if you only patch ethereum, so you use only these lines, and comment out the others:

[patch."https://github.com/purestake/frontier"]
ethereum = { package = "pallet-ethereum", path = "temp/frontier/frame/ethereum" }

Then you run cargo build, you get build errors!!!!

Notice that this means that we use the given path for ethereum, but the others should be taken from the repo, which practically should be the exact same code because it's the same branch used at the same time... we just cloned and switched to that branch!

How can this be explained? Aren't both using the same branch and the same repository? Why are we getting broken code? Is this a bug in rust where it's taking a different branch or something, or am I doing something wrong?

Thank you in advance, and sorry for the long question.

If the frontier crates are not patched, then you depend on the frontier crates from crates.io, but ethereum depends on the frontier crates from temp/frontier. Even if the source would be the same, they are treated as separate crates by crates.io and thus get compiled individually. The compile errors likely come from attempting to pass a value with a type from one of the versions of the crate to a function that expects a type of the other version of the crate.

So when I implement a patch, it ignores all the crate mentions in Cargo.toml (which include the correct repo and branch), and just uses the ones from crates.io?

Is there a way I could tell it to not ignore these entries in Cargo.toml?

If this is the case, is there a way to tell Cargo to just replace everything that uses that repo? The motivation for this question is that I'm having tons of other crates that need to be replaced (while I only need one replaced), even many deep in the project that I'm not aware of, and having to replace them manually one by one is not really practical.

Oh, you are using a git dependencies. Then replace crates.io with git repo in my answer.

I see. This is really a problem for the reasons I mentioned in my last reply. Is there a way to tell Cargo to just replace everything? Or do I have to run some kind of find and replace over the whole project directory? Even the latter is not necessarily simple because I can imagine different branches being used for the same repo in different places.

No, I don't think there is a way to replace everything.

What would you do if you were in my place? Is there anything I can do to fix this issue? (this is not only a question to @bjorn3 , but anyone who comes by and could help)

I would just list all crates. You said this worked fine:

[patch."https://github.com/purestake/frontier"]
ethereum = { package = "pallet-ethereum", path = "temp/frontier/frame/ethereum" }
frontier-consensus = { path = "temp/frontier/consensus" }
frontier-rpc = { path = "temp/frontier/rpc" }
frontier-rpc-primitives = { path = "temp/frontier/rpc/primitives" }

Yes, but the real problem is not these. I have bigger problems with other crates (this was just a minimal example to understand the issue). For example, try to replace frame-system and frame-executive and you'll see you'll need to replace a gazillion other dependencies and I never saw the end of it. I actually, at some point, totally failed at recognizing what I should replace next to make it work after having replaced over 30 dependencies. That's what I'm asking for "What would you do if you were in my place".

https://github.com/PureStake/moonbeam/blob/master/node/standalone/Cargo.toml#L23-L67

Yeah, that is a lot of dependencies. I am not quite sure what to do.

Usually, it's enough to patch one crate, and it will be replaced with your source without impacting any of the other crates. But your situation is a bit unusual because pallet-ethereum only specifies path for its frontier-consensus-primitives and frontier-rpc-primitives dependencies. When pallet-ethereum is specified with git URL, cargo will automatically resolve relative paths inside that repository. But when you add a patch and cargo uses the local version, there is no more an associated repository, so cargo resolves them by local path and these dependencies are no longer the same as the dependencies specified with git URL.

In fact, it's enough to only patch pallet-ethereum if you also change the local temp/frontier/frame/ethereum/Cargo.toml file like so:

frontier-consensus-primitives = { git = "https://github.com/purestake/frontier", branch = "v0.2-hotfixes", path = "../../consensus/primitives", default-features = false }
frontier-rpc-primitives = { git = "https://github.com/purestake/frontier", branch = "v0.2-hotfixes", path = "../../rpc/primitives", default-features = false }

This forces cargo to use the same version of these dependencies (obtained from git) everywhere.

I understand, and that sounds 100% reasonable. However, please remember that my original question is just a minimal example to present the problem. My issue now is that I have to manually change something in all relevant repositories' entries. This is extreme. Like I said above, try replacing frame-system and frame-executive and see what happens... Tons of dependencies have to be updated, many of them are inside other cargo toml files, and I don't even have a formal way of automatically detecting them. So I have to guess why the code is broken.

It's totally untenable for team development, because not everyone understands everything in such a complex source code. There had to be a way to just replace a repository + branch in the whole dependency tree by adding one line in the root cargo file. Currently I'm writing a Python script to do this for me. But rust has to have such a thing inside of it.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.