Tests(py): add runtime profiles and endpoint-aware fixtures#10500
Tests(py): add runtime profiles and endpoint-aware fixtures#10500nmanovic wants to merge 4 commits intov3-pr4b-fast-restorefrom
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 993bf99607
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| invocation_args = tuple(str(arg) for arg in config.invocation_params.args) | ||
| explicit_profile_requested = any( | ||
| arg == "--infra-profile" or arg.startswith("--infra-profile=") | ||
| for arg in invocation_args |
There was a problem hiding this comment.
explicit_profile_requested is derived only from config.invocation_params.args, but pytest does not include options injected via addopts/PYTEST_ADDOPTS there. As a result, a user-provided --infra-profile from those standard config paths is treated as implicit and can be silently replaced by required_profile, so an intentionally fixed (or intentionally too-small) profile no longer enforces the expected fail-fast behavior and can start a different stack than requested.
Useful? React with 👍 / 👎.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd233ad3c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| elif platform == "local": | ||
| return |
There was a problem hiding this comment.
The unconditional local-mode early return means instance.start() is deferred until pytest_runtestloop, so --cleanup / --dumpdb no longer execute during session startup and now depend on successful test collection. If collection/import fails first, these maintenance commands never run, leaving generated compose artifacts or DB dumps unchanged even though the user explicitly requested them.
Useful? React with 👍 / 👎.
| marker = item.get_closest_marker("infra_profile") | ||
| if marker and marker.args: | ||
| item_profile = str(RuntimeInfraConfig.parse_infra_profile(marker.args[0])) | ||
| else: | ||
| item_profile = str(InfraProfile.SIMPLE) |
There was a problem hiding this comment.
Profile detection only reads marker.args, so tests marked as @pytest.mark.infra_profile(name="full") are treated as unmarked (simple) because marker.args is empty. Since pytest.ini documents the marker as infra_profile(name), this silently underestimates required services and can start an undersized runtime for affected tests.
Useful? React with 👍 / 👎.
nmanovic
force-pushed
the
v3-pr4b-fast-restore
branch
from
b8f310d to
fec0185
Compare
nmanovic
force-pushed
the
v3-pr4c-profiles-runtime-urls
branch
from
dd233ad to
3f3c5a8
Compare
nmanovic
force-pushed
the
v3-pr4b-fast-restore
branch
from
fec0185 to
6719c1e
Compare
nmanovic
force-pushed
the
v3-pr4c-profiles-runtime-urls
branch
from
3f3c5a8 to
b1c63cd
Compare
nmanovic
force-pushed
the
v3-pr4c-profiles-runtime-urls
branch
from
b1c63cd to
987e99a
Compare
nmanovic
force-pushed
the
v3-pr4b-fast-restore
branch
from
6719c1e to
943f5b3
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1c63cdb10
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| required_profile = str(InfraProfile.SIMPLE) | ||
| for item in items: | ||
| marker = item.get_closest_marker("infra_profile") | ||
| if marker and marker.args: | ||
| item_profile = str(RuntimeInfraConfig.parse_infra_profile(marker.args[0])) |
There was a problem hiding this comment.
collection_modifyitems computes required_profile from every collected item, but this hook runs before pytest's built-in -k/-m deselection handlers, so filtered-out high-profile tests still influence the result. In practice, selecting a small subset can still force standard/full startup or trigger a false --infra-profile ... is too small error even though the deselected tests would not run. Move this logic to run after deselection (for example via trylast) or recompute from the final selected item set.
Useful? React with 👍 / 👎.
| if runtime.port: | ||
| netloc = f"{netloc}:{runtime.port}" | ||
| elif current.port: | ||
| netloc = f"{netloc}:{current.port}" |
There was a problem hiding this comment.
The normalization logic always replaces the endpoint port with runtime.port when CVAT_TEST_DB_MINIO_ENDPOINT_URL is set, which collapses intentionally distinct fixture ports (including broken endpoints used for negative scenarios) into the runtime default. That can turn fixtures like the minio:9777 case into a valid endpoint and invalidate status/error expectations when endpoint remapping is enabled. The rewrite should avoid overriding an explicit fixture port indiscriminately.
Useful? React with 👍 / 👎.
nmanovic
force-pushed
the
v3-pr4b-fast-restore
branch
from
943f5b3 to
86c7dba
Compare
nmanovic
force-pushed
the
v3-pr4c-profiles-runtime-urls
branch
from
987e99a to
6e83eae
Compare
nmanovic
force-pushed
the
v3-pr4b-fast-restore
branch
from
86c7dba to
753f499
Compare
nmanovic
force-pushed
the
v3-pr4c-profiles-runtime-urls
branch
from
6e83eae to
6cfb20b
Compare
1 participant
Summary
This is the next stacked PR after
v3-pr4b-fast-restore.It introduces explicit local runtime profiles and runtime-aware endpoint handling for Python tests.
Scope:
simple,standard, andfulllocal runtime profiles@pytest.mark.infra_profile(...)markers in test files--infra-profileis too smallOut of scope:
Why
After the runtime foundation and fast restore PRs, the remaining local-runtime gap was test selection and endpoint assumptions.
Some Python tests need a heavier local stack than others:
At the same time, some fixture-backed payloads still assumed fixed legacy internal URLs.
This PR makes those requirements explicit and keeps the runtime/test data aligned without introducing dynamic ports yet.
What changed
Local runtime profiles
Added 3 local profiles:
simplestandardfullSelection rules:
simple@pytest.mark.infra_profile("standard")or@pytest.mark.infra_profile("full")--infra-profileis not explicitly provided, the local runtime selects the smallest sufficient profile for the collected test set--infra-profileis explicitly too small, pytest fails before starting the stackCompose layering
Added only valid local profile overlays:
tests/docker-compose.simple.profile.ymltests/docker-compose.standard.profile.ymlImportant:
tests/docker-compose.core.profile.ymltests/docker-compose.extended.profile.ymlThe
fullprofile is assembled from valid existing compose inputs only.Runtime-aware endpoints
Added shared helper support so runtime-facing URLs adapt without changing committed fixture assets:
CVAT_BASE_URLfor SDK / CLI / shared helpersThis keeps repo-tracked fixtures stable while still matching the active runtime.
Test classification cleanup
infra_profile(...)markerswith_external_servicesmarker usage fromtests/pythonValidation
Static checks:
py_compileblack --checkisort --check --diff --resolve-all-configspylinttyposremarkProfile selection checks:
simplelocal smoke: passedstandardlocal smoke: passedfulllocal smoke: passedTargeted profile-sensitive suites:
tests/python/rest_api/test_cloud_storages.py52 passedtests/python/rest_api/test_webhooks.py94 passed, 1 skippedFull local Python suite:
pytest tests/python --platform=local2178 passed, 20 skipped, 1 xfailedCI-like Django unit tests:
docker compose -f docker-compose.yml -f docker-compose.dev.yml -f docker-compose.ci.yml run --rm cvat_ci python manage.py test cvat/apps -v 2412 tests,OK (skipped=58)Review notes
infra_profile(...)markers but does not create a second synthetic marker system.