OpenStack source + images architecture

[Mohammed Naser (mnaser)]

One of the longstanding things that I'm not happy with in Atmosphere where I feel that we can "do better" is the infrastructure around managing sources of the things we ship and how we build images for them, more specifically OpenStack and the components around it.

At the moment, we have an intertwined set of Docker images which are built with Docker Bake (allowing us to reference each other). Those images check out code from OpenDev, apply patches to it (potentially) which have not landed upstream yet, and publish all that into a virtual environment based on Ubuntu. There are several approaches that we can take here if we want to change things (including the current option).

Virtual environment in images

• Fetch sources from OpenDev
• Apply patches in the Dockerfile
• Build virtual environment with multi-stage builds

Pros

• We already have this, nothing to write.
• Simple and anyone can build a new image.

Cons

• Managing patches is a headache, no proper "patch management"
• No unit (or any) testing of our patches, resulted in regressions in the past

External Packages (either RDO or UCA)

• Install packages into a Docker image

Pros

• Simple, nothing to maintain at all.

Cons

• Results in large images in my experience (package have a lot of dependencies)
• No packages for unmerged changes (i.e. not able to ship fixes that haven't landed yet)
• No ability to easy patch downstream version

Internal packages (RPM or DEB)

• Build internal packages and ship them into repository

Pros

• Decouple "build" packages from "build images"
• Allows us to manage our own "downstream" patches

Cons

• Additional work that mostly exists upstream
• Potential regressions for things that upstream worked out already

Hybrid approach (RPM or DEB)

• Only build packages that we want to patch

Pros

• Same pros as internal packages
• Build tools typically will run unit tests so it will make sure that our "package builds" are somewhat validated
• We can make efforts to add CI to build images using newly generated images before merging to make sure they work 100%

Cons

• Not knowing what is "forked" and what isn't
• Need to keep "in sync" with upstream a fair bit

Alpine packages using Wolfi

• Same as RPM or DEB packages, but using https://github.com/wolfi-dev

Pros

• Ultra small footprint images

Cons

• Need to build everything from scratch, a lot of stuff missing.

Nix packages

• Same as other package solutions, but using Nix

Pros

• Reproducible builds at any time
• Ability to patch things as needed

Cons

• Still a lot of OpenStack dependencieis to package
• Burden of maintaining all packages is all on us

---

Personally, I am torn on what we can do here, I do feel like we should split "package build" and "image build" from each other. I am leaning towards the hybrid approach of building images from packages, however, maintaining forks of packages and managing our own patches for them so we have Atmosphere specific builds.

[Oleksandr K. (okozachenko1203)]

I believe we've been implicitly discussing this topic for quite some time—through changes in how we build images and manage patches—just without explicitly framing it as a unified discussion. Over time, we've explored nearly all available approaches to managing patches, packages, and images individually, but we haven’t fully consolidated these processes into a streamlined, end-to-end workflow. So thank you for raising this topic in a structured and high-level context.

---

I'd like to propose another approach—not entirely new, but one that hasn’t been explicitly discussed here yet.

Prior to our current workflow, we used to maintain forked versions of source repositories on GitHub to manage downstream patches. However, those forks lacked CI pipelines for patch validation, and we didn’t have our own package build process. Instead, patches were referenced directly during the image build.

We didn’t stick with that model for long. At the time, the number of patches was small, and maintaining forks for just a few changes felt more like overhead than value. Without CI to validate those forks, the effort invested in maintaining them didn’t pay off.

---

What I’m proposing now is a middle-ground approach between our current process and a full internal packages strategy:

• Maintain forked source code repositories for downstream patches
• Implement CI pipelines to validate those patches
• Skip internal package builds—instead, continue installing software directly during the image build, just as we currently do with upstream patching

This gives us the structure and testability benefits of keeping forks and CI, without the added complexity of building and maintaining a separate packaging process.

---

I'm suggesting this mainly to add it to the list of considered approaches in the discussion. I'm not advocating for it at this approach. :)

[Mohammed Naser (mnaser)]

Oleksandr K. (@okozachenko1203) thanks for the feedback, if we do indeed do take this route, this means we have to maintain forks and keep them in sync.

The challenge with this is that when upstream creates a new release, then we have to cherry-pick all of the patches for the new branch afterwards (which would be highly manual and error prone in my opinion). We will also need to align with upstream to when they cut branches and cut at the same time, which can become a challenge across different releases.

The patches approach maintains the patches outside in the packaging and if we bump the version, the patches are very clearly indicated and would either apply/conflict/etc.

In addition, I think we'd end up with a lot of extra downstream CI.. but I don't think that's bad if we get it done right.

[Dong Ma (larainema)]

From my perspective, I vote following two approaches:

1. Hybrid Package Approach, for this approach, following things we should consider:

• Workflow Gap – Need to establish a package build pipeline, I think we didn't have any package build workflow

• Patch Discipline – Must enforce: Changelog updates per patch; Metadata tracking; Automated patch applicability checks

• Upstream Sync – Requires scheduled automation to detect new upstream releases and validate patches against updates

---

2. Forked source code repositories for downstream patches

Pros

• Direct Control – Patches live in version-controlled forks.

• CI Integration – Can enforce tests pre-merge.

Cons

• Manual efforts of branch synchronization

[Adrian Reza (adrianreza)]

From my understanding I also vote like this, using hybrid internal packages because we can selectively patching component thet we need to patch or customize while relying upstream for the rest. And from my two cents opinion we can combine with multi stage docker image build to keep the images lean.

Using nix or wolfi-image can be good considerations for the future but I am not sure about the maturity to handle complex system like openstack.

[Mohammed Naser (mnaser)]

I think finally we're ending up at the KISS method:

https://github.com/vexxhost/atmosphere/pull/2681/files

In this, we have a simple Docker image that takes care of patching files, then passing that context into another image. We should be able to leverage this image in other components so that "prepping" the source code is just an extra target which gets called using the one that builds it. That way, we can reduce the amount of copy-pasta that happens.

We can eventually transition the other projects to the same, and maintain all of our patches in the patches folder, which also means that it would be good to do the same for OpenStack Helm as well eventually.