Coreos release model

[hanthor]

🚀 Bootc Image Promotion Pipeline

1. Goal and Objectives

1.1. Goal

To implement an automated, multi-stage continuous integration and continuous deployment (CI/CD) pipeline that promotes immutable bootc operating system images from unstable next to production-ready stable channels, ensuring quality gates are met through increasingly rigorous testing stages (Unit, QEMU, openQA).

1.2. Objectives

• Ensure immutability of the core OCI image throughout promotion.
• Automate the build and basic testing of the next image on every commit.
• Implement a QEMU-based integration test to validate successful boot and service initialization.
• Integrate with openQA for rigorous, end-to-end (E2E) system validation before tagging as stable.
• Maintain clear channel tags (:next, :testing, :stable) for consumers.

2. Architecture and Flow

The pipeline uses GitHub Actions as the orchestrator and a Container Registry (e.g., Quay.io) as the single source of truth for the OCI images.

2.1. Workflow Diagram

2.2. Promotion Policy

Channel	Quality Level	Trigger	Artifact State	Testing	Tagging
Next	Unstable/Bleeding Edge	Push to main branch.	OCI Image built from source.	Unit Tests, Linting.	:, :next
Testing	QA Ready/Internal Use	Manual Push of vX.Y.Z-test Git Tag.	OCI Image promoted from :next.	QEMU Boot/Service Checks.	:, :testing
Stable	Production/Approved	Manual Push of vX.Y.Z Git Tag.	OCI Image promoted from :testing.	openQA Full E2E Suite.	:, :vX.Y.Z, :stable

3. Implementation Details by Stage

3.1. Stage 1: Next (Fast CI)

• Trigger: on: push: branches: [main]
• Workflow: build-next.yml
• Steps:
  • Build OCI Image: Use podman build or docker build based on the provided Containerfile.
  • Run Unit Tests: Execute application/OS configuration unit tests.
  • Push Image: Use skopeo or a registry login action to push the image to the registry.
  • Tagging: Apply both the Git SHA and the :next tag.

3.2. Stage 2: Testing (QEMU Integration)

• Trigger: on: push: tags: ['v*-test']
• Workflow: test-qemu.yml
• Requirements: This stage requires a self-hosted GitHub Runner with KVM/virtualization support enabled to achieve fast and reliable QEMU execution.
• Steps:
  • Pull :next Image: Fetch the image tagged :next.
  • Generate Disk Image: Run the bootc-image-builder container to convert the OCI image into a QCOW2 disk image artifact.
  • QEMU Boot & Test:
    • Use QEMU to boot the QCOW2 image.
    • Implement a CI script that waits for SSH/network access.
    • Run basic integration tests via SSH (e.g., check kernel version, verify critical services like sshd.service are running).
  • Promotion (Retag): If QEMU tests pass, use skopeo copy to retag the original SHA-tagged image with the :testing tag.
    3.3. Stage 3: Stable (openQA E2E)
• Trigger: on: push: tags: ['v*..'] (Semantic Version tags)
• Workflow: release-stable.yml
• Requirements: Access to a separate, persistent openQA testing infrastructure.
• Steps:
  • Pull Disk Image: Pull the QCOW2 image artifact generated in the Testing stage.
  • Upload to Test Location: Upload the QCOW2 image to an accessible location (e.g., S3 bucket or internal HTTP server) that the openQA workers can reach.
  • Submit openQA Job: Use the openQA Client CLI within the GitHub Action to submit the job, referencing the uploaded image URL and specifying the "Full Regression Suite."
  • Poll for Result: The workflow polls the openQA instance for the job status.
  • Gating and Promotion: If the openQA job completes with Passed:
    • Use skopeo copy to retag the immutable image with the final :stable and the : tags (e.g., :v1.2.0).

4. Tools and Dependencies

Tool	Purpose	Location
GitHub Actions	CI/CD Orchestration.	Repository Workflows.
Podman/Docker	OCI Image Build.	Runner/Local.
bootc-image-builder	Converts OCI image to bootable disk formats (QCOW2, etc.).	Runner/Container.
QEMU/KVM	Virtualization for fast integration testing.	Self-hosted Runner.
openQA	Rigorous, full system, end-to-end testing platform.	Dedicated External Infrastructure.
skopeo	Image copying and retagging (promotion) without rebuilding.	Runner/Container.
Container Registry	Stores and serves all OCI images (e.g., Quay.io, GitHub Container Registry).	External Service.

5. Security and Governance

• Immutability: Promotion steps must only use skopeo copy to retag images based on their immutable SHA, ensuring no tampering between stages.
• Access Control: Registry secrets for pushing and retagging images must be stored securely in GitHub Secrets. The stable promotion requires a manual, reviewed Git tag push, acting as an implicit human approval gate.
• Artifact Retention: Implement a cleanup policy on the container registry to prune old Git SHA and :next images, but retain all versioned and :stable images indefinitely.