[Core][Jobs][runtime_env] worker_process_setup_hook runs in driver instead of worker processes

[TheGreatFrankie]

What happened + What you expected to happen

Description

When submitting a Ray job via the Ray Jobs REST API, the
worker_process_setup_hook specified in runtime_env is executed in the
driver process, rather than in the worker processes.

According to the documentation:
https://docs.ray.io/en/latest/ray-core/api/doc/ray.runtime_env.RuntimeEnv.html

worker_process_setup_hook: A callable that runs in the worker process on each worker node.

Expected behavior

worker_process_setup_hook should run in the worker process on each worker node, regardless of whether the job is submitted via the Python API or the Jobs REST API.

Actual behavior

When the job is submitted using the Ray Jobs REST API, the hook runs in the
driver process, not in worker processes.

Impact

This blocks per-worker initialization (e.g. environment setup, local resource
configuration) for jobs submitted via the Jobs API.

Versions / Dependencies

Environment:

• Ray version: 2.35.0
• Python version: 3.9 (container), 3.11.2 (development)
• Submission method: Ray Jobs REST API (via ray.run_job())
• OS: Debian 11 (in Docker container)
• Installation: Bazel-managed dependencies in uber-one monorepo

Reproduction script

Reproduction (high level)

1. Submit a Ray job via the Jobs REST API
2. Provide worker_process_setup_hook in runtime_env
3. Log process context (PID / role) inside the hook
4. Observe the hook runs in the driver process

We can provide a minimal reproduction script if helpful.

Issue Severity

None

[sampan-s-nayak]

thanks for reporting this issue, could you please share a repro script if possible? I tried following the steps mentioned on ray version > 2.53 and I dont seem to be facing any issue. could you try updating ray version to see if you are still facing this issue?.

[TheGreatFrankie]

Internally, we’re currently on Ray 2.35, and upgrading will take some effort. Could you confirm whether this is a known issue that was fixed upstream, and if so, which Ray version or PR addressed it? That would help us assess the upgrade path on our side.

[sampan-s-nayak]

I dont see any significant changes in this code path since ray 2.10.0. could you share repro steps, I can spend some time debugging with it on my local machine.