bug: Unable to run vllm performance in uv native environment

[christian-pinto]

Issue Description

When running the vllm-performance actuator, a ray task is generated for executing the benchmark and dependencies (e.g., vllm) are installed on the fly.

However, due to a bug in Ray, even though uv is installed in the venv it is not found and Ray tries to install it with pip. Pip is in turn not found and the execution hangs indefinitely.

Only inspecting the ray logs one can figure it out

2026-04-21 10:27:16,112 ERROR runtime_env_agent.py:404 -- Failed to create runtime env {"uv": {"packages": ["ado-vllm-performance[vllm]"], "uv_check": false, "uv_pip_install_options": ["--no-cache"]}}.
Traceback (most recent call last):
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/agent/runtime_env_agent.py", line 396, in _create_runtime_env_with_retry
    runtime_env_context = await asyncio.wait_for(
                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/agent/runtime_env_agent.py", line 354, in _setup_runtime_env
    await create_for_plugin_if_needed(
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/plugin.py", line 255, in create_for_plugin_if_needed
    size_bytes = await plugin.create(uri, runtime_env, context, logger=logger)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 316, in create
    uv_dir_bytes = await task
                   ^^^^^^^^^^
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 296, in _create_for_hash
    await UvProcessor(
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 206, in _run
    await self._install_uv_packages(
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 158, in _install_uv_packages
    await self._install_uv(path, cwd, pip_env, logger)
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/uv.py", line 103, in _install_uv
    await check_output_cmd(uv_install_cmd, logger=logger, cwd=cwd, env=pip_env)
  File "/workspace/venvs/test/lib/python3.12/site-packages/ray/_private/runtime_env/utils.py", line 105, in check_output_cmd
    raise SubprocessCalledProcessError(
ray._private.runtime_env.utils.SubprocessCalledProcessError: Run cmd[4] failed with the following details.
Command '['/tmp/ray/session_2026-04-21_10-26-48_683153_1748541/runtime_resources/uv/49299b5d5c8efb07187427cfd44e9b6431b8fd09/virtualenv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--no-cache-dir', 'uv']' returned non-zero exit status 1.
Last 50 lines of stdout:
    /tmp/ray/session_2026-04-21_10-26-48_683153_1748541/runtime_resources/uv/49299b5d5c8efb07187427cfd44e9b6431b8fd09/virtualenv/bin/python: No module named pip

This issue in Ray is already known in Ray ray-project/ray#60570 and there is a commit fixing it since Feb 2026. However, it has not been included in any PR, nor it is merged yet.

How to reproduce

save the snippet below as point.yaml. The values of endpoint and model don't really matter because the execution stops way before they are used.

entity:
  endpoint: "http://localhost:8000"
  model: Qwen/Qwen3.5-27B
  request_rate: 10

experiments:
  - actuatorIdentifier: vllm_performance
    experimentIdentifier: test-endpoint-v1

Then execute the command below

$ uv venv test
$ source test/bin/activate
$ uv pip install ado-core ado-vllm-performance
$ LOGLEVEL=DEBUG run_experiment point.yaml
...
Executing: vllm_performance.test-endpoint-v1
(VLLMPerformanceTest pid=1893418) 2026-04-21 13:20:28,408 WARNING   AsyncIO Thread: default actuator       : __init__            : No namespace set in acutator configuration - will not be able to create deployments
(VLLMPerformanceTest pid=1893418) 2026-04-21 13:20:28,414 DEBUG     AsyncIO Thread: default ado_actuators.vllm_performance.actuator: catalog             : Experiments dir /workspace/venvs/test/lib/python3.12/site-packages/ado_actuators/vllm_performance/experiments
(VLLMPerformanceTest pid=1893418) 2026-04-21 13:20:28,414 DEBUG     AsyncIO Thread: default ado_actuators.vllm_performance.actuator: catalog             : Loading experiments from performance_testing.yaml
(VLLMPerformanceTest pid=1893418) 2026-04-21 13:20:28,485 DEBUG     AsyncIO Thread: default ado_actuators.vllm_performance.actuator: submit              : Experiment (test-endpoint-v1) - Ray task environment: {'uv': ['ado-vllm-performance[vllm]']}

run_experiment gets stuck indefinitely on the last line of the log above.

Expected behaviour

Screenshots/Logs

Python/ado/system info

Additional information

Add any other context about the problem here.

[AlessandroPomponio]

I'm marking this as externally blocked at the moment. In general, you can always use uv venv --seed to create a virtual environment that has pip available

[christian-pinto]

I think this might require adding a temporary warning in the documentation

[AlessandroPomponio]

As far as I remember in our documentation we never suggest using uv to create virtual environments, only in the development setup, where pip would be pulled automatically. If you find a place where that isn't the case, however, or see a place where that could go, feel free to open a PR. It's definitely better to have it

[VassilisVassiliadis]

This sounds like a problem I ran into a while ago for which I submitted a fix to Ray ray-project/ray#54141 it's supposed to be fixed in ray 2.49. Which version of ray are you using ?

[AlessandroPomponio]

The issue is that https://github.com/ray-project/ray/blob/30482d19df47d36da6313ffbdd1eff7fbed0e269/python/ray/_private/runtime_env/uv.py#L105-L123 uses python -m uv version to check for uv version and it doesn't work anymore. A fix has already been proposed in ray-project/ray@f895148. Another option would be to use python -m uv self version

[VassilisVassiliadis]

ah there you go, that makes sense. Thanks!

[christian-pinto]

I have opened a PR to speedup the process: ray-project/ray#62818

[christian-pinto]

The above PR got merged so we can expect the fix to land in one of the next Ray releases.

[AlessandroPomponio]

I'll keep this open until it's released and we have updated our requirements to be at least the ray version that implements the fix