Using custom rustc + std for a project

visigoth · December 20, 2020, 10:02am

I am trying to build a rust project against the x86_64-apple-ios-macabi target. I know the target is tier 3, and as such, need to build my own std. With nightly cargo and toolchain, I am able to do that with the -Z build-std flag.

However, it turns out rustc has a bug when trying to build binaries for x86_64-apple-ios-macabi (https://github.com/rust-lang/rust/issues/80202). i have a custom rustc that fixes the issue (https://github.com/rust-lang/rust/pull/80215). However, when I try to use that rustc, I get an error stating that the std crate could not be found.

I specifically have done this:

built rustc with ./x.py build -i with the targets entry set in config.toml to include both my host triple (x86_64-apple-darwin) and the catalyst triple (x86_64-apple-ios-macabi) and the build-stage entry set to 2
linked this build to rustup with rustup toolchain link stage2 build/x86_64-apple-darwin/stage2
used rustup override to use my custom rustc as the compiler in my sample app
setup Xargo.toml with:

[target.x86_64-apple-ios-macabi.dependencies]
std = {path = "../rust/library/std"}

and tried building with xargo build --target x86_64-apple-ios-macabi $*

however, i get the following error:

error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-apple-ios-macabi` target may not be installed

when that didn't work, i tried building rustc with: ./x.py -i --target x86_64-apple-ios-macabi --stage 2 compiler/rustc and did the rest as well, but i still got the same error.

how can i get my sample app to see my custom built std crate?

jyn514 · December 20, 2020, 2:10pm

Try changing compiler/rustc to library/std and see if it helps.

visigoth · December 20, 2020, 7:35pm

unfortunately that does not change the outcome. here is the full output:

 ~/code/rust   issue-80202-fix  ./x.py build -i --target x86_64-apple-ios-macabi --stage 2 library/std                               ✔  10265  10:35:31
Updating only changed submodules
Submodules updated in 0.06 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.84s
Building stage0 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.18s
Copying stage0 std from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.44s
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage1 compiler (x86_64-apple-darwin)
Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.21s
Copying stage1 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage1 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.39s
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage2 compiler (x86_64-apple-darwin)
Building stage2 std artifacts (x86_64-apple-darwin -> x86_64-apple-ios-macabi)
   Compiling cc v1.0.60
   Compiling core v0.0.0 (/Users/shaheen/code/rust/library/core)
   Compiling libc v0.2.79
   Compiling std v0.0.0 (/Users/shaheen/code/rust/library/std)
   Compiling compiler_builtins v0.1.36
   Compiling unwind v0.0.0 (/Users/shaheen/code/rust/library/unwind)
   Compiling rustc-std-workspace-core v1.99.0 (/Users/shaheen/code/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/Users/shaheen/code/rust/library/alloc)
   Compiling cfg-if v0.1.10
   Compiling adler v0.2.3
   Compiling rustc-demangle v0.1.18
   Compiling panic_abort v0.0.0 (/Users/shaheen/code/rust/library/panic_abort)
   Compiling rustc-std-workspace-alloc v1.99.0 (/Users/shaheen/code/rust/library/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/Users/shaheen/code/rust/library/panic_unwind)
   Compiling gimli v0.23.0
   Compiling miniz_oxide v0.4.0
   Compiling object v0.22.0
   Compiling hashbrown v0.9.0
   Compiling addr2line v0.14.0
warning: dropping unsupported crate type `dylib` for target `x86_64-apple-ios-macabi`

warning: 1 warning emitted

   Compiling rustc-std-workspace-std v1.99.0 (/Users/shaheen/code/rust/library/rustc-std-workspace-std)
   Compiling term v0.0.0 (/Users/shaheen/code/rust/library/term)
   Compiling proc_macro v0.0.0 (/Users/shaheen/code/rust/library/proc_macro)
   Compiling unicode-width v0.1.8
   Compiling getopts v0.2.21
   Compiling test v0.0.0 (/Users/shaheen/code/rust/library/test)
warning: dropping unsupported crate type `dylib` for target `x86_64-apple-ios-macabi`

warning: 1 warning emitted

    Finished release [optimized] target(s) in 1m 15s
Copying stage2 std from stage2 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-ios-macabi)
Build completed successfully in 0:01:18
 ~/code/rust   issue-80202-fix  cd ../catalystsample                                                                                 ✔  10266  10:36:54
 ~/code/catalystsample   master ✚ ● ? ⍟1  cat build.sh                                                                               ✔  10267  11:34:01
#!/bin/sh

# Add --release for release builds
xargo build --target x86_64-apple-ios-macabi $*
#cargo +nightly run -Z build-std --target x86_64-apple-ios-macabi $*
#cargo build -Z build-std --target x86_64-apple-ios-macabi $*
 ~/code/catalystsample   master ✚ ● ? ⍟1  ./build.sh                                                                                 ✔  10268  11:34:05
   Compiling compiler_builtins v0.1.36
   Compiling rustc-std-workspace-core v1.0.0
   Compiling libc v0.2.79
   Compiling cc v1.0.60
   Compiling core v0.0.0 (/Users/shaheen/code/rust/library/core)
   Compiling rustc-std-workspace-alloc v1.0.0
   Compiling std v0.0.0 (/Users/shaheen/code/rust/library/std)
error[E0463]: can't find crate for `std`

error: aborting due to previous error

error[E0463]: can't find crate for `std`

For more information about this error, try `rustc --explain E0463`.
error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: could not compile `libc`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-apple-ios-macabi` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-apple-ios-macabi` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: build failed
error: `"cargo" "build" "--release" "--manifest-path" "/var/folders/s1/mypqqscx3sd1czdv039rsth80000gn/T/xargo.GEEjw8KTu2aG/Cargo.toml" "--target" "x86_64-apple-ios-macabi" "-p" "std"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace

visigoth · December 20, 2020, 7:43pm

actually, i guess i don't need Xargo if i'm using my own rustc since I've built std for the target from scratch already - regular cargo should be able to find it without having to build it, so maybe just cargo build --target x86_64-apple-ios-macabi will work?

visigoth · December 20, 2020, 8:33pm

Okay, doing that for my sample application works. However, doing the same for my actual use case fails. if i add dependencies to my sample application, i can repro this new problem.

i added dependencies to my sample project like so (previously, it had no dependencies):

[dependencies]
 prost = "0.6.1"
 protobuf = "2.18.1"
 tokio = { version = "0.2.24", features = ["full"] }
 tonic = { version = "0.3.1", features = ["transport", "tls", "codegen"] }

here is the output of cargo build --target x86_64-apple-ios-macabi:

  ~/code/catalystsample   master ● ? ↑1 ⍟1  cargo build --target x86_64-apple-ios-macabi
    Updating crates.io index
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.54
   Compiling libc v0.2.81
   Compiling cfg-if v0.1.10
   Compiling log v0.4.11
   Compiling futures-core v0.3.8
   Compiling bytes v0.5.6
   Compiling lazy_static v1.4.0
   Compiling pin-project-internal v0.4.27
   Compiling slab v0.4.2
   Compiling memchr v2.3.4
   Compiling fnv v1.0.7
   Compiling pin-project-lite v0.1.11
   Compiling tower-service v0.3.0
   Compiling cfg-if v1.0.0
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error[E0463]: can't find crate for `std`

...

visigoth · December 21, 2020, 1:19am

i tried debugging this a bit myself by enabling logging on the part of rust attempting to match std. here's what i get when i turn on logging in the locator:

RUSTC_LOG=rustc_metadata::locator rustc +stage2 --crate-name prost --edition=2018 /Users/shaheen/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-0.6.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="prost-derive"' -C metadata=7eb4e11f45b48716 -C extra-filename=-7eb4e11f45b48716 --out-dir /Users/shaheen/code/catalystsample/target/x86_64-apple-ios-macabi/debug/deps --target x86_64-apple-ios-macabi -L dependency=/Users/shaheen/code/catalystsample/target/x86_64-apple-ios-macabi/debug/deps -L dependency=/Users/shaheen/code/catalystsample/target/debug/deps --extern bytes=/Users/shaheen/code/catalystsample/target/x86_64-apple-ios-macabi/debug/deps/libbytes-d5ed15f5286ca0ad.rmeta --extern prost_derive=/Users/shaheen/code/catalystsample/target/debug/deps/libprost_derive-a57ee8557424ef6e.dylib --cap-lints allow
 INFO rustc_metadata::locator lib candidate: /Users/shaheen/code/rust/build/x86_64-apple-darwin/stage2/lib/rustlib/x86_64-apple-darwin/lib/libstd-7e7cd4f74fc23894.rlib
 INFO rustc_metadata::locator lib candidate: /Users/shaheen/code/rust/build/x86_64-apple-darwin/stage2/lib/rustlib/x86_64-apple-darwin/lib/libstd-7e7cd4f74fc23894.dylib
 INFO rustc_metadata::locator rlib reading metadata from: /Users/shaheen/code/rust/build/x86_64-apple-darwin/stage2/lib/rustlib/x86_64-apple-darwin/lib/libstd-7e7cd4f74fc23894.rlib
 INFO rustc_metadata::locator Rejecting via proc macro: expected true got false
 INFO rustc_metadata::locator metadata mismatch
 INFO rustc_metadata::locator dylib reading metadata from: /Users/shaheen/code/rust/build/x86_64-apple-darwin/stage2/lib/rustlib/x86_64-apple-darwin/lib/libstd-7e7cd4f74fc23894.dylib
DEBUG rustc_metadata::locator checking 8 bytes of metadata-version stamp
DEBUG rustc_metadata::locator inflating 3131592 bytes of compressed metadata
 INFO rustc_metadata::locator Rejecting via proc macro: expected true got false
 INFO rustc_metadata::locator metadata mismatch

it's unclear to me whether it's look at the right candidate files. rustc is invoked with x86_64-apple-ios-macabi. i see build/x86_64-apple-darwin/stage2-std/x86_64-apple-ios-macabi, but that doesn't seem to have any libstd files in it.

not sure where to look at this point.

jyn514 · December 23, 2020, 8:36pm

Compiling cfg-if v1.0.0
error[E0463]: can't find crate for std

Is cfg-if a proc-macro? Proc macros use the standard library of the host, not of the target. Try x.py build library/std --stage 2 (without passing --target) and see if it helps.

visigoth · December 23, 2020, 4:45am

that surprisingly worked! i think that's surprising because i specifically did ./x.py build -i --target x86_64-apple-darwin --stage 2 library/std - i thought this would build the std library that is compatible with the host. i would have thought ./x.py build --stage 2 library/std would implicitly have --target x86_64-apple-darwin since that is my host machine's target triple, and that -i (incremental) would not be particularly meaningful.

jyn514 · December 23, 2020, 2:25pm

Are you sure that's your host triple? If you run x.py check -vv | head there should be a snippet at the very top that says "detected default triple", what does that say?

visigoth · December 23, 2020, 8:36pm

yes, that's my host triple:

 ~/code/rust   issue-80202-fix  ./x.py check -vv
Updating only changed submodules
Submodules updated in 0.06 seconds
detected default triple x86_64-apple-darwin