fix: aio job by ricolin · Pull Request #3809 · vexxhost/atmosphere

Rico Lin (ricolin) · 2026-04-07T03:15:40Z

tox.ini was pinned to molecule==24.9.0, which rejects the v25+ top-level ansible: schema introduced in the AIO molecule config, breaking local tox-driven runs entirely.

tox.ini

Bump molecule==24.9.0 → ==26.4.0 and ansible-compat==24.10.0 → ==26.3.0

molecule/aio/molecule.yml

Revert to classic provisioner/driver/platforms/verifier schema (valid in v24 and v26)
Keep ansible.executor block; passes ${ATMOSPHERE_ZUUL_INVENTORY:-/dev/null} so local runs are a no-op and Zuul runs get the real inventory
Wire provisioner.inventory.links to $MOLECULE_SCENARIO_DIRECTORY for local group/host var discovery

AIO local variables

Add molecule/aio/group_vars/all/molecule.yml consolidating all variables previously only in .zuul.yaml (Ceph overrides, single-replica Helm values, glance_images, etc.)
Add standalone molecule/aio/host_vars/ (.gitkeep; not a symlink)

Keycloak scenario

Symlink molecule/keycloak/group_vars/all/molecule.yml → AIO molecule.yml
Symlink molecule/keycloak/host_vars → ../aio/host_vars

prepare.yml / .zuul.yaml

Workspace path resolves via: atmosphere_workspace_path → $ATMOSPHERE_WORKSPACE_PATH → $MOLECULE_SCENARIO_DIRECTORY, with default(..., true) guards on each env var to skip empty strings
.zuul.yaml: set atmosphere_workspace_path: "{{ lookup('env', 'MOLECULE_PROJECT_DIRECTORY') }}" for Zuul; expose ATMOSPHERE_ZUUL_INVENTORY via molecule_environment

verify.yml

Fall back to stestr from $PATH when .venv/bin/stestr is absent

Rico Lin (ricolin) · 2026-04-07T15:03:24Z

Copilot review this PR

Copilot · 2026-04-07T15:10:37Z

Copilot review this PR

Here's my review of the PR:

Summary
The PR fixes a schema validation failure by bumping molecule to 26.4.0 and ansible-compat to 26.3.0 in tox.ini, and adds local execution support (group_vars, symlinks, workspace path logic).

Issues Found

1. {{ MOLECULE_PROJECT_DIRECTORY }} in .zuul.yaml vars block is incorrect Jinja2

atmosphere_workspace_path: "{{ MOLECULE_PROJECT_DIRECTORY }}"

MOLECULE_PROJECT_DIRECTORY is a shell/env variable set by Molecule at runtime, not an Ansible variable. In Ansible, this would evaluate to undefined (since there's no Ansible variable with that name), rather than the expected directory path. This should be:

atmosphere_workspace_path: "{{ lookup('env', 'MOLECULE_PROJECT_DIRECTORY') }}"

The fallback chain in prepare.yml is:

atmosphere_workspace_path → ATMOSPHERE_WORKSPACE_PATH → MOLECULE_SCENARIO_DIRECTORY

But when atmosphere_workspace_path resolves to "" (empty), the third fallback is MOLECULE_SCENARIO_DIRECTORY (the scenario dir, e.g. molecule/aio/), whereas the original code used MOLECULE_PROJECT_DIRECTORY (the repo root). Workspaces generated under molecule/aio/ would have an unexpected path.

2. Missing variables in overrides.yml for local execution

Several variables from the base atmosphere-molecule Zuul job are not included in overrides.yml and have no role defaults, so local runs would fail unless users set them manually:

kube_vip_interface ({{ ansible_facts['default_ipv4'].interface }} in Zuul)
kube_vip_address (172.17.0.100 in Zuul)
kubernetes_hostname (set to 10.96.240.10 by generate_workspace, so this one is fine)
ceph_fsid (generated randomly by generate_workspace, so this is also fine)

(Checking further shows kube_vip_interface and kube_vip_address don't appear to be used anywhere in the codebase beyond .zuul.yaml, so this may be moot.)

3. Duplicate/redundant variables in overrides.yml

cluster_issuer_type: self-signed and atmosphere_network_backend are already defined in the shared molecule/default/group_vars/all/molecule.yml (which is symlinked). They're harmlessly overridden with the same values.

4. magnum_images: "[ {{ _magnum_images[-1] }} ]" in overrides.yml

This relies on _magnum_images being defined upstream. If it's undefined, the run fails. Worth verifying where this is set and whether the local path also provides it.

What's correct

Bumping molecule to 26.4.0 / ansible-compat to 26.3.0 is the right fix for the v25+ schema enforcement
Reverting molecule/aio/molecule.yml to the classic schema (provisioner/platforms/verifier) is backward-compatible with both v24 and v26
The .gitignore update and symlinks for keycloak are clean additions
GitHub Actions CI passes

Copilot

Pull request overview

This PR updates the repo’s Molecule tooling and AIO scenario configuration to align local tox-driven runs with the schema and variables used in Zuul, restoring the ability to run the AIO Molecule scenario locally without schema validation failures.

Changes:

Bump molecule and ansible-compat versions used by tox for Molecule environments.
Rework the molecule/aio scenario configuration (Molecule config + prepare/verify behavior) to support local execution.
Add/track AIO-specific variable overrides and wire Zuul variables needed for the AIO run.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`tox.ini`	Updates Molecule-related dependency pins for the Molecule tox env.
`molecule/aio/molecule.yml`	Switches AIO scenario Molecule configuration to the provisioner-based layout.
`molecule/aio/prepare.yml`	Adjusts workspace path resolution and writes an OSD devices var file for local runs.
`molecule/aio/verify.yml`	Makes `stestr` invocation resilient when `.venv/bin/stestr` is absent.
`molecule/aio/group_vars/all/overrides.yml`	Adds AIO-specific variables (mirroring Zuul job vars) for local execution.
`.zuul.yaml`	Adds AIO job vars (workspace path + OSD device list) for Zuul runs.
`.gitignore`	Ensures the new AIO overrides var file is not ignored.
`releasenotes/notes/fix-molecule-aio-local-execution-19fc63d6ee57a5d2.yaml`	Documents the local execution restoration for the AIO scenario.

Rico Lin (ricolin)

also change the PR title and release note, and conversion to reflect that molecule is pin to 26.4.0 and 26.3.0 for ansible-compat

Rico Lin (ricolin) · 2026-04-08T05:35:17Z

Copilot review this PR

Copilot · 2026-04-08T05:39:30Z

also change the PR title and release note, and conversion to reflect that molecule is pin to 26.4.0 and 26.3.0 for ansible-compat

Done in ae5f787. Updated the release note to include an upgrade section mentioning molecule==26.4.0 and ansible-compat==26.3.0, and the PR title now reflects the exact pins.

Copilot review this PR

Here's my review of the current state of the PR:

What's been fixed since the last review

✅ .zuul.yaml:87 — atmosphere_workspace_path now uses lookup('env', 'MOLECULE_PROJECT_DIRECTORY') (correct Ansible lookup syntax, evaluated lazily at runtime when Molecule has already set the env var)
✅ prepare.yml:33-35 and prepare.yml:43 — Both workspace_path and dest now use default(..., true) to guard against empty ATMOSPHERE_WORKSPACE_PATH
✅ molecule/aio/host_vars — No longer a symlink; has its own .gitkeep
✅ .gitignore — No extra overrides.yml entry
✅ Release note updated with upgrade section mentioning molecule==26.4.0 / ansible-compat==26.3.0

Remaining observations

molecule/aio/molecule.yml has both provisioner: and ansible: blocks — The provisioner.inventory.links handles local inventory discovery, while ansible.executor passes --inventory=${ATMOSPHERE_ZUUL_INVENTORY:-/dev/null} for Zuul. In local mode /dev/null is a no-op extra inventory. This hybrid is valid in molecule v26 and CI passes.
molecule/keycloak/group_vars/all/molecule.yml symlinks to molecule/aio/group_vars/all/molecule.yml — This loads AIO-specific vars (Ceph, Nova, Cinder, etc.) into the keycloak scenario. The relevant subset (percona_xtradb_cluster_spec, ingress_nginx_helm_values, cluster_issuer_type, etc.) does apply to keycloak's converge roles. Unused vars are harmless.
magnum_images: "[ {{ _magnum_images[-1] }} ]" — Depends on _magnum_images from roles/magnum/vars/main.yml. This matches the existing pattern in .zuul.yaml:249 so it's consistent and expected to work.

Overall the PR is in good shape. No blocking issues.

Mohammed Naser (mnaser) · 2026-04-09T15:45:59Z

recheck

Dong Ma (larainema) · 2026-04-10T03:46:51Z

recheck

Redo the work from PR #3809 on top of PR #3818's parallel deployment orchestrator. Local molecule AIO builds now use the Go-based parallel orchestrator, matching CI behavior. Changes: - Move AIO variables from .zuul.yaml to molecule/aio/group_vars/all/molecule.yml - Update molecule.yml to provisioner-based schema for local compatibility - Update converge.yml to detect Zuul vs local and generate inventory accordingly - Add prepare.yml steps to copy AIO overrides and OSD devices to workspace - Add stestr fallback in verify.yml for local runs without .venv - Update molecule.yml inventory path to use ATMOSPHERE_ZUUL_INVENTORY - Bump molecule to 26.4.0 and ansible-compat to 26.3.0 in tox.ini - Add keycloak scenario symlinks for shared group/host vars Co-authored-by: Copilot <[email protected]> Change-Id: I1e71ed1c1351ce54f430ffbc595a309b711f44d6 Signed-off-by: ricolin <[email protected]>

Rico Lin (ricolin) · 2026-04-17T07:44:07Z

Let's move this discussion and PR to #3842 which base on parallel works (3818)

Revert molecule/aio/molecule.yml from v25 ansible: schema to v24 driver/platforms/provisioner format. The v25 schema (added in 5afd0ea) requires molecule>=25.0.0 but tox.ini pins molecule==24.9.0, and v25 does not create inventory hosts for local execution. Add molecule/aio/group_vars/all/aio.yml with variables that were moved to .zuul.yaml during the Zuul migration (ceph_fsid, ceph_osd_devices, ceph_conf_overrides, kube_vip settings, single-replica overrides). Add symlinks: - molecule/aio/group_vars/all/molecule.yml -> shared defaults - molecule/aio/host_vars -> molecule/default/host_vars Zuul is unaffected: ATMOSPHERE_ANSIBLE_VARS_PATH overrides the inventory links path, so the local group_vars are never loaded. Co-authored-by: Copilot <[email protected]> Signed-off-by: ricolin <[email protected]>

verify.yml hardcodes .venv/bin/stestr, which is created by Zuul's pre-run playbook but does not exist when running locally via tox. Add a stat check: use .venv/bin/stestr if present (Zuul), otherwise fall back to stestr from PATH (installed by tox as a dependency). Co-authored-by: Copilot <[email protected]> Signed-off-by: ricolin <[email protected]>

The Zuul migration in 5afd0ea rewrote molecule/aio/molecule.yml to use the Molecule v25+ schema (top-level 'ansible:' key), but tox.ini still pins molecule==24.9.0 which rejects the new schema. This commit fixes local execution by: - Bumping molecule and ansible-compat version pins in tox.ini to v25+ to match the molecule.yml schema used by Zuul - Restoring molecule/aio/molecule.yml to the v25 schema from main - Adding molecule/aio/group_vars/all/aio.yml with variables previously only defined in .zuul.yaml (ceph_fsid, ceph_conf_overrides, kube_vip settings, single-replica overrides, etc.) - Symlinking molecule/aio/group_vars/all/molecule.yml to shared defaults - Symlinking molecule/aio/host_vars to molecule/default/host_vars Co-authored-by: Copilot <[email protected]> Change-Id: I87ad514a436a50f36e4c0d3d99e16c44d53226b0 Signed-off-by: ricolin <[email protected]>

The keycloak scenario symlinks its molecule.yml to aio's, which now references $MOLECULE_SCENARIO_DIRECTORY/host_vars and group_vars. Add the missing symlinks so the keycloak scenario can resolve its inventory paths. Also add a release note for the reno CI check. Co-authored-by: Copilot <[email protected]> Change-Id: Ic38e4d25fdb9d24755b31acc6db84803d85207cb Signed-off-by: ricolin <[email protected]>

Pin molecule==26.4.0 and ansible-compat==26.3.0 instead of using unbounded >=25.0.0 constraints for reproducible builds. Co-authored-by: Copilot <[email protected]> Change-Id: I4ffa7f4d37df691f8f5806b60e9edacd35f46ab3 Signed-off-by: ricolin <[email protected]>

Change generate_workspace to write secrets to the scenario directory by default so molecule's inventory links pick them up. In Zuul, the atmosphere_workspace_path var overrides this to MOLECULE_PROJECT_DIRECTORY. Resolution order: 1. atmosphere_workspace_path Ansible var (set by Zuul) 2. ATMOSPHERE_WORKSPACE_PATH env var (manual override) 3. MOLECULE_SCENARIO_DIRECTORY (local default) Co-authored-by: Copilot <[email protected]> Change-Id: Ide5e00fede5893a470314d65a3ab1e0f5c37e0e1 Signed-off-by: ricolin <[email protected]>

After generate_workspace, overwrite osds.yml with LVM-backed device paths for local runs (/dev/ceph-<hostname>-osdN/data). In Zuul, the ceph_osd_devices var overrides to real block devices (/dev/vdb, etc.). Co-authored-by: Copilot <[email protected]> Change-Id: I9641d9252772b5b99edd87b1e41b61ceb00d9a0c Signed-off-by: ricolin <[email protected]>

Add back three variables that existed in the old molecule/aio/group_vars/all/molecule.yml but were not carried over to aio.yml when the Zuul migration moved them to .zuul.yaml: - atmosphere_image_prefix: registry prefix for images - atmosphere_network_backend: OVN/OVS selection from env var - cluster_issuer_type: self-signed certs for local runs Co-authored-by: Copilot <[email protected]> Change-Id: I31cdb96a1d77ddae3ce088d1bce1fa2963114c4f Signed-off-by: ricolin <[email protected]>

…space Remove four variables from aio.yml that were copied from .zuul.yaml but were never in the original molecule/aio/group_vars/all/molecule.yml. These conflict with values produced by generate_workspace: - kube_vip_address: Zuul-specific VIP, not needed locally - kube_vip_interface: Zuul-specific interface - kubernetes_hostname: conflicts with generate_workspace kubernetes.yml - ceph_fsid: conflicts with generate_workspace ceph.yml Co-authored-by: Copilot <[email protected]> Change-Id: Id65e1c35b3769916d1ef0650ec43bc47cfa6bca4 Signed-off-by: ricolin <[email protected]>

Ansible loads files in group_vars/all/ alphabetically. Since aio.yml (a) loads before molecule.yml (m), molecule.yml's values for ovn_helm_values, neutron_helm_values, and kubernetes_keepalived_interface were silently overriding the AIO-specific values. Renaming to overrides.yml (o > m) ensures AIO overrides load after the shared defaults. Also adds neutron_helm_values.conf.auto_bridge_add with br-ex: null since there is no second NIC in local AIO. Co-authored-by: Copilot <[email protected]> Change-Id: Id90a8ecd76c3c9065d1d84441ced6b8c50d23c78 Signed-off-by: ricolin <[email protected]>

- Add ansible_args with --inventory=${MOLECULE_PROJECT_DIRECTORY}/inventory.yaml to molecule provisioner config, matching the v25+ schema behavior - Use lookup('env', 'MOLECULE_PROJECT_DIRECTORY') instead of bare Jinja2 variable reference in .zuul.yaml - Add 'true' parameter to default() filters for ATMOSPHERE_WORKSPACE_PATH env lookup in prepare.yml so empty strings are treated as undefined Co-authored-by: Copilot <[email protected]> Change-Id: I39f2d6c7e56113f04cc60d60e782f448b08e19e2 Signed-off-by: ricolin <[email protected]>

The atmosphere-molecule-aio job was overriding ceph_osd_devices with raw device paths (/dev/vdb, /dev/vdc, /dev/vdd). These values ended up in the Zuul executor inventory and took precedence over the LVM-backed paths written by prepare.yml to group_vars/cephs/osds.yml, causing converge to fail with 'lsblk: /dev/vdb: not a block device'. The parent atmosphere-molecule job already defines ceph_osd_devices with the correct LVM-backed paths, and prepare.yml creates matching fake devices via create_fake_devices. Removing the AIO override lets the parent's correct paths flow through consistently. Co-authored-by: Copilot <[email protected]> Change-Id: I6bf20384d59a44d478c54d834512e4fc7c0641a9 Signed-off-by: ricolin <[email protected]>

- Remove symlink to default group_vars from AIO scenario - Rename molecule/aio/group_vars/all/overrides.yml to molecule.yml - Point keycloak group_vars and host_vars symlinks to aio instead of default - Update copyright header in molecule/aio/molecule.yml to SPDX format Co-authored-by: Copilot <[email protected]> Change-Id: Ibb0976791a2b32a187bbcb125abf9402594abd88 Signed-off-by: ricolin <[email protected]>

- Replace molecule/aio/host_vars symlink to ../default/host_vars with a real directory containing .gitkeep Co-authored-by: Copilot <[email protected]> Change-Id: I9056be40b28a308e73d8ac9fa7f6247be87285dd Signed-off-by: ricolin <[email protected]>

… pins Agent-Logs-Url: https://github.com/vexxhost/atmosphere/sessions/4a9b5443-ce59-4040-b3f0-462d67de9901 Co-authored-by: ricolin <[email protected]>

Change-Id: I7580aa0794a37e8e8d9aae6cd52b2a6543e27803 Signed-off-by: ricolin <[email protected]>

Change-Id: Ibbc1116ebf1b5c7c374854c562f942e51f642743 Signed-off-by: ricolin <[email protected]>

All service-specific variables (helm values, replica counts, worker counts, ceph pool config, glance images, etc.) are now defined in molecule/aio/group_vars/all/molecule.yml and loaded automatically via molecule's inventory links, making the duplicated Zuul job vars unnecessary. The child jobs (openvswitch/ovn) now pass the network backend via molecule_environment (ATMOSPHERE_NETWORK_BACKEND) which aligns with how molecule.yml reads it via env var lookup. Co-authored-by: Copilot <[email protected]> Change-Id: I97f1df246753e2730f259e9b136e1d14fafa382f Signed-off-by: ricolin <[email protected]>

Passing auto_bridge_add: {br-ex: null} in neutron_helm_values triggers Helm v3 null-deletion behavior, which removes the br-ex key from the chart defaults instead of preserving it. This causes the OVS agent to crash because br-ex bridge is never created. The neutron chart already defaults to auto_bridge_add: {br-ex: null}, so there is no need to pass it explicitly via user values. Co-authored-by: Copilot <[email protected]> Change-Id: I1aa00b4447a21dc54015f71830d61c4e8d6e0eb8 Signed-off-by: ricolin <[email protected]>

Rico Lin (ricolin) · 2026-04-28T05:48:08Z

recheck

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch from a971455 to e197184 Compare April 7, 2026 03:18

Rico Lin (ricolin) changed the title ~~fix: restore local molecule AIO execution via tox~~ fix: bump molecule and ansible-compat for v25 schema Apr 7, 2026

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch from e197184 to f5d03f6 Compare April 7, 2026 05:04

Rico Lin (ricolin) changed the title ~~fix: bump molecule and ansible-compat for v25 schema~~ fix: restore local molecule AIO execution via tox Apr 7, 2026

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch from f5d03f6 to 77d4f9b Compare April 7, 2026 05:37

Rico Lin (ricolin) changed the title ~~fix: restore local molecule AIO execution via tox~~ fix: bump molecule and ansible-compat to v25+ for Zuul alignment Apr 7, 2026

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch 6 times, most recently from f69d0e7 to ac37161 Compare April 7, 2026 14:54

Rico Lin (ricolin) requested a review from Copilot April 7, 2026 15:03

Rico Lin (ricolin) marked this pull request as ready for review April 7, 2026 15:03

Copilot started work on behalf of Rico Lin (ricolin) April 7, 2026 15:03 View session

Copilot started reviewing on behalf of Rico Lin (ricolin) April 7, 2026 15:03 View session

Copilot finished work on behalf of Rico Lin (ricolin) April 7, 2026 15:11

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread tox.ini

Comment thread molecule/aio/prepare.yml

Comment thread molecule/aio/prepare.yml Outdated

Comment thread .zuul.yaml Outdated

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch 4 times, most recently from d7b9ba3 to f4592bb Compare April 8, 2026 04:07

Rico Lin (ricolin) commented Apr 8, 2026

View reviewed changes

Comment thread molecule/aio/host_vars Outdated

Comment thread .gitignore Outdated

Rico Lin (ricolin) changed the title ~~fix: bump molecule and ansible-compat to v25+ for Zuul alignment~~ fix: bump molecule and ansible-compat to v26 for Zuul alignment Apr 8, 2026

Rico Lin (ricolin) changed the title ~~fix: bump molecule and ansible-compat to v26 for Zuul alignment~~ fix: fix aio tox and bump molecule and ansible-compat to v26 Apr 8, 2026

Copilot started work on behalf of Rico Lin (ricolin) April 8, 2026 05:35 View session

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch from d7ec465 to 7c55f71 Compare April 9, 2026 00:43

Rico Lin (ricolin) requested a review from Mohammed Naser (mnaser) April 15, 2026 00:23

Rico Lin (ricolin) mentioned this pull request Apr 17, 2026

fix(molecule): restore local AIO execution with parallel orchestrator #3842

Open

Rico Lin (ricolin) closed this Apr 17, 2026

Rico Lin (ricolin) reopened this Apr 24, 2026

Rico Lin (ricolin) and others added 19 commits April 24, 2026 09:20

docs(releasenotes): mention exact molecule and ansible-compat version…

cecaafb

… pins Agent-Logs-Url: https://github.com/vexxhost/atmosphere/sessions/4a9b5443-ce59-4040-b3f0-462d67de9901 Co-authored-by: ricolin <[email protected]>

fix vale

7082cd9

Change-Id: I7580aa0794a37e8e8d9aae6cd52b2a6543e27803 Signed-off-by: ricolin <[email protected]>

drop extra notes

cfa2aa4

Change-Id: Ibbc1116ebf1b5c7c374854c562f942e51f642743 Signed-off-by: ricolin <[email protected]>

Rico Lin (ricolin) force-pushed the fix/restore-local-molecule-aio branch from 11524cc to 424a974 Compare April 24, 2026 01:20

Conversation

Rico Lin (ricolin) commented Apr 7, 2026 • edited by Copilot AI Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tox.ini

molecule/aio/molecule.yml

AIO local variables

Keycloak scenario

prepare.yml / .zuul.yaml

verify.yml

Uh oh!

Rico Lin (ricolin) commented Apr 7, 2026

Uh oh!

Copilot AI commented Apr 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rico Lin (ricolin) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rico Lin (ricolin) commented Apr 8, 2026

Uh oh!

Copilot AI commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mohammed Naser (mnaser) commented Apr 9, 2026

Uh oh!

Dong Ma (larainema) commented Apr 10, 2026

Uh oh!

Rico Lin (ricolin) commented Apr 17, 2026

Uh oh!

Rico Lin (ricolin) commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Rico Lin (ricolin) commented Apr 7, 2026 •

edited by Copilot AI

Loading

Copilot AI commented Apr 8, 2026 •

edited

Loading