rules_rust can't bootstrap using remote execution

[rejuvenile]

1. The bootstrap phase can't find Xcode unless SDKROOT and DEVELOPER_DIR are set as host action_envs.
2. The bootstrap phase assumes it can find rust-objcopy, which isn't supplied by rules_rust's nightly toolchain.

[UebelAndre]

By bootstrap do you mean bootstrap the process wrapper?

2. The bootstrap phase assumes it can find rust-objcopy, which isn't supplied by rules_rust's nightly toolchain.

interesting, where does rust-objcopy come from?

[DoDoENT]

I'm just in the process of setting up remote build for my company and stumbled upon this issue. Whenever there is a remote execution on mac worker, build fails with:

(15:52:00) ERROR: /private/var/tmp/_bazel_****/eb0b148894ba6d64194ea7ef59d6bc51/external/rules_rust+/util/process_wrapper/BUILD.bazel:30:36: Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool] failed: (Exit 1): bootstrap_process_wrapper.sh failed: error executing Rustc command (from target @@rules_rust+//util/process_wrapper:process_wrapper) 

  (cd /private/var/tmp/_bazel_****/eb0b148894ba6d64194ea7ef59d6bc51/execroot/_main && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=26.2 \
    CARGO_CFG_TARGET_ARCH=aarch64 \
    CARGO_CFG_TARGET_OS=macos \
    CARGO_CRATE_NAME=process_wrapper \
    CARGO_MANIFEST_DIR='${pwd}/external/rules_rust+/util/process_wrapper' \
    CARGO_PKG_AUTHORS='' \
    CARGO_PKG_DESCRIPTION='' \
    CARGO_PKG_HOMEPAGE='' \
    CARGO_PKG_NAME=process_wrapper \
    CARGO_PKG_VERSION=0.0.0 \
    CARGO_PKG_VERSION_MAJOR=0 \
    CARGO_PKG_VERSION_MINOR=0 \
    CARGO_PKG_VERSION_PATCH=0 \
    CARGO_PKG_VERSION_PRE='' \
    PATH=/bin:/usr/bin:/usr/local/bin \
    REPOSITORY_NAME=rules_rust+ \
    XCODE_VERSION_OVERRIDE=26.2.0.17C52 \
    ZERO_AR_DATE=1 \
  bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust+/util/process_wrapper/bootstrap_process_wrapper.sh -- bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust++rust+rust_macos_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain/bin/rustc external/rules_rust+/util/process_wrapper/main.rs '--crate-name=process_wrapper' '--crate-type=bin' '--error-format=human' '--out-dir=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust+/util/process_wrapper' '--codegen=opt-level=3' '--codegen=debuginfo=0' '--codegen=strip=debuginfo' '--remap-path-prefix=${pwd}=.' '--emit=link=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust+/util/process_wrapper/process_wrapper' '--emit=dep-info' '--color=always' '--target=aarch64-apple-darwin' -L bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust++rust+rust_macos_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain/lib/rustlib/aarch64-apple-darwin/lib '--edition=2018' '-Cembed-bitcode=no' '--codegen=linker=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/apple_support+/crosstool/cc_wrapper.sh' '--codegen=link-arg=-lc++' '--codegen=link-arg=-Wl,-oso_prefix,__BAZEL_EXECUTION_ROOT__/' '--codegen=link-arg=-headerpad_max_install_names' '--codegen=link-arg=-no-canonical-prefixes' '--codegen=link-arg=-target' '--codegen=link-arg=arm64-apple-macosx26.2' '--codegen=link-arg=-fobjc-link-runtime' '--codegen=link-arg=-dead_strip' '--codegen=link-arg=-framework' '--codegen=link-arg=Foundation' '--codegen=link-arg=-Wl,-no_warn_duplicate_libraries' '--extern=tinyjson=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust++i+rules_rust_tinyjson/libtinyjson-1317710085.rlib' '-Ldependency=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust++i+rules_rust_tinyjson' '--sysroot=bazel-out/darwin_arm64-opt-exec-ST-a8947299217b/bin/external/rules_rust++rust+rust_macos_aarch64__aarch64-apple-darwin__stable_tools/rust_toolchain' '-Cstrip=debuginfo')

# Configuration: 8e7f5a340e5608639b795cfdb4babdca2f999363fa8d2ed038a835a53dc86948
# Execution platform: //platforms/macos:arm64

Remote server execution message: Action details (uncached result): https://browser.build.microblink.com/blobs/sha256/historical_execute_response/1bdc093b04c5e73496feb29aa7ff86b6190efc5b882c908e06e58872ad90bd6f-1273/

error: unable to run `rust-objcopy`: No such file or directory (os error 2)

error: aborting due to 1 previous error

If I build the same target with --spawn_strategy=local or without --remote_executor, so that it runs locally on my machine, then it works.

[rejuvenile]

@DoDoENT I have managed to make it work by adding these to .bazelrc:

common:macos --action_env=DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
common:macos --host_action_env=DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer

... and adding "lib/rustlib/aarch64-apple-darwin/bin/rust-objcopy" to my toolchain's filegroup.

[rejuvenile]

By bootstrap do you mean bootstrap the process wrapper?

2. The bootstrap phase assumes it can find rust-objcopy, which isn't supplied by rules_rust's nightly toolchain.

interesting, where does rust-objcopy come from?

Yes, the process wrapper. I'm not sure what uses rust-objcopy, but adding it to my toolchain fixes the issue.

[DoDoENT]

I have managed to make it work by adding these to .bazelrc

I used to have those before but I had to remove them because this then interferes with remote build (different builds on different branches can use different versions of Xcode).

DEVELOPER_DIR needs to be automatically calculated based on the selected xcode_config. See here, here, and here for more info.

[DoDoENT]

Until this gets fixed, as a workaround I added this to my .bazelrc:

common:macos --strategy=Rustc=local

This forces Rust compilations to always be local, while everything else (java, c, c++, python, ...) can safely run remotely.

[illicitonion]

@keith Does rules_apple have any handy utilities or recommendations for how $DEVELOPER_DIR should be set when doing remote execution? Should this be configured on the remote executor itself, or in general bazel config, or do rules need to integrate this directly themseves?

[keith]

typically the remote execution services duplicates the logic in bazel that takes the hermetic env variables:

    APPLE_SDK_VERSION_OVERRIDE=26.2 \
    APPLE_SDK_PLATFORM=MacOSX \
    XCODE_VERSION_OVERRIDE=26.2.0.17C52 \

and uses that to find the actual absolute path to Xcode. for example here's buildbuddy's implementation: https://github.com/buildbuddy-io/buildbuddy/blob/bde95f45f3661c4a418823490bd862d512a93d48/enterprise/server/remote_execution/executorplatform/executorplatform.go#L213-L241 and here's buildbarn's https://github.com/buildbarn/bb-remote-execution/blob/59f12f4e5145054474423b8a5214ef4151e9eee9/cmd/bb_runner/main.go#L138-L148

it is never expected that you set DEVELOPER_DIR / SDKROOT. the expectation is just that the remote executors have the right version of Xcode (easiest to just use the same locally and remotely)

[UebelAndre]

Sounds like this is not a rules_rust issue then? @DoDoENT @rejuvenile does that context help you out?

[keith]

given that action above has the env vars already it doesn't seem like it. often one issue is needing to get those xcode specific env vars into the action, same with execution_info in the case you have a heterogenous remote exec cluster and you respect requires-darwin (I don't think many remote exec clusters work that way though)