It seems that I need artifact dependencies for this.
FWIW, I found this reddit thread where OP asked the same question... and artifact dependencies seem to be the solution here (although I am not able to use them, as I am running on stable rust).
So I have to find a way to fiddle myself around the issue somehow...