AP-554: build a UCBLIT-specific Airflow image by anarchivist · Pull Request #1 · BerkeleyLibrary/mokelumne

anarchivist · 2026-02-03T01:54:57Z

I believe all Airflow config options can be passed in as env variables per the configuration documentation.
This includes a single test called from the build pipeline to ensure there are no import errors in the Airflow dags.

* I believe all Airflow config options can be passed in as env variables per the [configuration documentation](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html). * This includes a single test called from the build pipeline to ensure there are no import errors in the Airflow dags.

awilfox

Looking pretty good. A few things to address.

example.env

dags/sample_dag.py

.github/workflows/build.yml

awilfox

Looks great to me. r+ after Docker Compose is running properly in CI.

anarchivist · 2026-02-03T22:17:25Z

I have the build passing. One thing to note - in GHA, the container is running as root, which I don't love, but that seems to be the first thing that fixed the Error: EACCES: permission denied when it runs actions/checkout@v4 in the test job. Feedback/suggestions here are welcome.

awilfox

Looks good overall. A few questions to think over, then r+.

.github/workflows/build.yml

Dockerfile

anarchivist · 2026-02-04T06:17:55Z

Okay - v2 separates the docker startup checks from the dag import checks, which don't require a full Airflow deployment to run. Accordingly, it uploads two test result artifacts - one for Docker/Compose-related logs and one for the Python unittest output.

awilfox

~~There's a fix needed for the test runner~~ (edit: there is no fix needed for the test runner, I miscounted the number of parent calls), and a question on the build workflow regarding the version of Airflow. Otherwise, lgtm.

.github/workflows/build.yml

we can successfully get unittest output written in the container and copy it out now

all outstanding issues have been fixed

danschmidt5189

Nothing major jumped out, but I am very curious about the intention behind running the test stage in a container while also using docker-compose commands. I have to be missing something there!

danschmidt5189 · 2026-02-05T16:54:59Z

Dockerfile

+
+USER airflow
+WORKDIR /home/airflow
+COPY --chown=airflow:0 requirements.txt .


Why use the GID "0" rather than "root", as we do everywhere else? We only use IDs when necessary, e.g. at runtime when the given named user doesn't exist in the container.

This is relates to an Airflow-on-OpenShiftism that I'm trying to keep consistent with upstream. I'm happy to change it if we don't care:

Airflow image is Open-Shift compatible, which means that you can start it with random user ID and the
group id 0 (root). If you want to run the image with user different than Airflow, you MUST set
GID of the user to 0. In case you try to use different group, the entrypoint exits with error.

OpenShift randomly assigns UID when it starts the container, but you can utilise this flexible UID
also in case of running the image manually. This might be useful for example in case you want to
mount dag and logs folders from host system on Linux, in which case the UID should be set
the same ID as your host user.

Relatedly, the overly-complex command for airflow-init is pulled mostly from the upstream compose file.

README.md

.github/workflows/build.yml

also, clean up the airflow-init container; minor docs fix

anarchivist requested review from awilfox and danschmidt5189 February 3, 2026 01:54

anarchivist marked this pull request as draft February 3, 2026 01:57

awilfox requested changes Feb 3, 2026

View reviewed changes

example.env Outdated Show resolved Hide resolved

dags/sample_dag.py Show resolved Hide resolved

.github/workflows/build.yml Show resolved Hide resolved

anarchivist added 2 commits February 2, 2026 18:36

address PR feedback; add missing CI compose file

3f8a483

properly pull in the built container for tests

0134a53

anarchivist force-pushed the AP-554-build-image branch from ff078ce to 0134a53 Compare February 3, 2026 02:43

awilfox approved these changes Feb 3, 2026

View reviewed changes

anarchivist force-pushed the AP-554-build-image branch 6 times, most recently from 5ea163a to 2a88a9c Compare February 3, 2026 22:13

anarchivist requested a review from awilfox February 3, 2026 22:14

anarchivist marked this pull request as ready for review February 3, 2026 22:14

awilfox approved these changes Feb 4, 2026

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

Dockerfile Show resolved Hide resolved

anarchivist force-pushed the AP-554-build-image branch 11 times, most recently from de7661a to 1d776e9 Compare February 4, 2026 04:35

anarchivist force-pushed the AP-554-build-image branch 2 times, most recently from 5c17118 to 35bbee5 Compare February 4, 2026 04:50

Separate dag import/container tests; clean up

dbd2827

anarchivist force-pushed the AP-554-build-image branch from 35bbee5 to dbd2827 Compare February 4, 2026 05:15

split into two artifacts

4397dff

anarchivist force-pushed the AP-554-build-image branch from 99d1074 to 4397dff Compare February 4, 2026 06:15

anarchivist force-pushed the AP-554-build-image branch from 7c068cb to 32fc810 Compare February 4, 2026 17:21

awilfox previously requested changes Feb 4, 2026

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

anarchivist force-pushed the AP-554-build-image branch from 32fc810 to a59f272 Compare February 4, 2026 18:11

final cleanup/permissions checks

0339cb6

anarchivist force-pushed the AP-554-build-image branch from a59f272 to 0339cb6 Compare February 4, 2026 18:27

add build args for airflow version and image name

cfbbf7b

anarchivist force-pushed the AP-554-build-image branch 2 times, most recently from 02ff98c to 139cbe5 Compare February 4, 2026 19:32

anarchivist added 2 commits February 4, 2026 11:52

merge tests back into one job

f92ba85

we can successfully get unittest output written in the container and copy it out now

remove setting AIRFLOW_VERSION in build config

2bce6db

anarchivist force-pushed the AP-554-build-image branch 2 times, most recently from cacd207 to 7177b75 Compare February 4, 2026 20:08

improve documentation

731c8a6

anarchivist force-pushed the AP-554-build-image branch from 7177b75 to 731c8a6 Compare February 4, 2026 20:09

anarchivist requested a review from awilfox February 4, 2026 20:12

danschmidt5189 reviewed Feb 5, 2026

View reviewed changes

compose and tests should run on the runner 🤦🏽‍♀️

4cb9ffb

also, clean up the airflow-init container; minor docs fix

anarchivist merged commit e346a13 into main Feb 6, 2026
5 checks passed

anarchivist deleted the AP-554-build-image branch February 6, 2026 21:37

Conversation

anarchivist commented Feb 3, 2026

Uh oh!

awilfox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awilfox left a comment

Choose a reason for hiding this comment

Uh oh!

anarchivist commented Feb 3, 2026

Uh oh!

awilfox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anarchivist commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

awilfox left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danschmidt5189 left a comment

Choose a reason for hiding this comment

Uh oh!

danschmidt5189 Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

anarchivist Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

anarchivist Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anarchivist commented Feb 4, 2026 •

edited

Loading

awilfox left a comment •

edited

Loading