chore: flip filtering label to mark examples with tests #15002 #15014

gianarb · 2025-11-10T09:09:14Z

Reference: #15002

Motivation

We want to increase the test coverage for our examples since they are important
to guide users and for documentations we want them to run correctly.

Modifications

Thi PR flips how we mark example with validation tests. Previously we were
marking only the examples with a test, it makes this process easy to kind of
chill out and avoid writing a validation test. We decided to run example tests
by default for all examples that do not explicitly exclude. This should help us
to writing those tests.

Verification

As part of exeuction of the tests I wrote the path of the examples that ran as
test in a file prev and after the change to validate that the same number of
examples got executed

Documentation

I could not find documentation mentioning the old label used so I didn't add
it. If you can point me to where I should add it I am happy to write something
about it.

This commit removes the label `workflows.argoproj.io/test` used to mark examples as testable because now we want as contributors to be more explicit marking examples that DO NOT have tests, this should help maintainers to ask for those tests as part of code review. This commit adds `workflows.argoproj.io/no-test-broken` for genuinely broken examples which can be fixed or that do not have test yet. `workflows.argoproj.io/no-test-environment` for examples which need to run in a different environment, e.g. need access to artifactory but this commit does not mark those. Related argoproj#15002 Signed-off-by: Gianluca Arbezzano <[email protected]>

Copilot

Pull Request Overview

This PR implements a strategy change for example test coverage by flipping from an opt-in to an opt-out approach. Instead of marking examples with workflows.argoproj.io/test: "true" to include them in testing, examples now run by default unless explicitly excluded with workflows.argoproj.io/no-test-broken or workflows.argoproj.io/no-test-environment labels. This change encourages better test coverage for documentation examples.

Inverted test filtering logic in test/e2e/examples_test.go to skip only labeled examples
Extended the allowed image list in test/e2e/fixtures/given.go to support more example scenarios
Updated approximately 100+ example YAML files to use the new labeling scheme and standardize container images

Reviewed Changes

Copilot reviewed 170 out of 175 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`test/e2e/examples_test.go`	Changed test logic from opt-in to opt-out filtering using new label names
`test/e2e/fixtures/given.go`	Extended allowed images list for examples to include more common base images
`test/e2e/testdata/cluster-workflow-template-ref.yaml`	Added no-test-broken label
`examples/*/.yaml` (100+ files)	Removed old test labels, added new exclusion labels, and standardized images to versions in allowlist
`docs/fields.md`	Updated example references to reflect the new labeling scheme
`.github/workflows/ci-build.yaml`	Added comment explaining example tests are non-retryable without controller restart

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-18T16:59:25Z

.github/workflows/ci-build.yaml

            profile: minimal
            use-api: false
            retries: 0
+          # Example tests are not retryable withou a controller restart


Spelling error in comment: "withou" should be "without".

Copilot · 2025-11-18T16:59:26Z

examples/expression-reusing-verbose-snippets.yaml

+        image: busybox
        command: [bash]


The image "debian:9.4" has been changed to "busybox", but the command remains [bash]. The busybox image typically uses sh instead of bash, which may cause this script to fail. Consider changing the command to [sh] or using an image that includes bash.

Copilot · 2025-11-18T16:59:26Z

examples/expression-destructure-json-complex.yaml

+        image: busybox 
        command: [bash]
        source: |


The image has been changed from "debian:9.4" to "busybox" with a trailing space. This trailing space in "busybox " may cause the container to fail to pull the image. Additionally, the command [bash] is incompatible with busybox, which typically uses sh. Consider removing the trailing space and changing the command to [sh].

Copilot · 2025-11-18T16:59:26Z

examples/influxdb-ci.yaml

          path: /var/lib/influxdb/data
    container:
-      image: debian:9.4
+      image: busybox 


The image has been changed from "debian:9.4" to "busybox" with a trailing space. This trailing space in "busybox " may cause the container to fail to pull the image. Please remove the trailing space.

Copilot · 2025-11-18T16:59:27Z

examples/expression-destructure-json.yaml

+        image: busybox 
        command: [bash]


The image has been changed from "debian:9.4" to "busybox" with a trailing space. This trailing space in "busybox " may cause the container to fail to pull the image. Additionally, the command [bash] is incompatible with busybox, which typically uses sh. Consider removing the trailing space and changing the command to [sh].

Copilot · 2025-11-18T16:59:27Z

examples/volumes-emptydir.yaml

  - name: volumes-emptydir-example
    container:
-      image: debian:latest
+      image: aline:3.6


Typo in image name: "aline:3.6" should be "alpine:3.6".

Copilot · 2025-11-18T16:59:27Z

.github/workflows/ci-build.yaml

            profile: minimal
            use-api: false
            retries: 0
+          # Example tests are not retryable withou a controller restart


Spelling error in comment: "exeuction" should be "execution".

…5002 Since we have changed how we identify if an example runs correctly as test this commit applies to the new protocol in the test runner. Signed-off-by: Gianluca Arbezzano <[email protected]>

Joibel

I think we need some more flags, and to really consider if any are broken. I've only reviewed down to http-success-condition, and will return to it later.

Joibel · 2025-11-25T13:06:52Z

examples/timeouts-workflow.yaml

 metadata:
  generateName: timeouts-workflow-
+  labels:
+    workflows.argoproj.io/no-test-environment: "true"


Another duration

Joibel · 2025-11-25T13:07:28Z

examples/timeouts-step.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


I presume this one "fails" but runs. It feels like maybe we need a separate label for no-test-expectedfailure or something?

yes makes sense to me. Should I proceed with that?

Joibel · 2025-11-25T13:08:19Z

docs/fields.md

This is mismatching on standard labels and misleading. Could you raise an issue to fix it please?

Joibel · 2025-11-25T13:11:20Z

examples/configmaps/simple-parameters-configmap.yaml

  labels:
    # Note that this label is required for the informer to detect this ConfigMap.
    workflows.argoproj.io/configmap-type: Parameter
+    workflows.argoproj.io/no-test-environment: "true"


It would be cleaner if we only attempted files which contained kind: Workflow.

Joibel · 2025-11-25T13:11:34Z

examples/workflow-event-binding/event-consumer-workfloweventbinding.yaml

 apiVersion: argoproj.io/v1alpha1
 kind: WorkflowEventBinding
 metadata:
+  labels:


Another untestable kind

Joibel · 2025-11-25T13:41:40Z

examples/fibonacci-seq-conditional-param.yaml

 metadata:
  generateName: fibonacci-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Broken, or just takes too long?

This is the error I see at the moment:

└ ✔ [2] StepGroup 0s 20:45:19 [14/1953] └ ○ fibonacci-helper Skipped 0s when '2 != 1 && 2 != 2' evaluated false └ ✔ fib-sub-1 Steps 0s └ ✔ [3] StepGroup 0s └ ✖ add Pod 3s main: Error (exit code 1) └ ✖ [4] StepGroup 5s child 'fibonacci-cfsxr-3090884672' failed └ ✔ fib-sub-2 Steps 0s examples_test.go:75: condition never and cannot be met because the workflow is done

Logs from the failing pod:

time=2025-11-25T19:45:15.042Z level=INFO msg="capturing logs" argo=true time=2025-11-25T19:45:15.042Z level=INFO msg="waiting for signals" argo=true signalPath=/var/run/argo/ctr/main/signal File "/argo/staging/script", line 1 print(2 + {{steps.fib-sub-2.outputs.parameters.fib}}) ^ SyntaxError: invalid syntax time=2025-11-25T19:45:15.088Z level=DEBUG msg="ignore signal" argo=true signal="child exited" time=2025-11-25T19:45:16.043Z level=INFO msg="file signal handler exiting due to context cancellation" argo=true time=2025-11-25T19:45:16.043Z level=INFO msg="sub-process exited" argo=true error="exit status 1" Error: exit status 1

Joibel · 2025-11-25T13:42:21Z

examples/forever.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Not really broken, but untestable

Joibel · 2025-11-25T13:42:57Z

examples/hdfs-artifact.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


environment surely, as no HDFS?

Joibel · 2025-11-25T13:43:32Z

examples/hello-world.yaml

 metadata:
  generateName: hello-world-
  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Really need this one to be working, so tested, what's wrong with it?

Joibel · 2025-11-25T13:44:04Z

examples/http-success-condition.yaml

 metadata:
  generateName: http-template-condition-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Another can we fix it one?

fixed the url used to retrieve status code did not work anymore

This commits adds two new labels to communicate why an example can't run as part of the validation test suite: 1. `duration` when an example is expected to run for more than the expected test timeout. 2. `expected-failure` when an example is expected to fail. Signed-off-by: Gianluca Arbezzano <[email protected]>

Joibel

Can we make sure that only tests that are really properly broken are marked as broken, because we should fix those. If they're otherwise untestable they should get a different flag.

Joibel · 2025-11-26T09:30:25Z

examples/cluster-workflow-template/cluster-wftmpl-dag.yaml

 metadata:
  generateName: workflow-template-dag-diamond-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


I'm not sure how you'd intend to implement this, but could we just do two passes of all the yaml?

First pass installs anything it finds that's not a workflow (e.g. the WFT and ClusterWFTs, ConfigMap) and then second pass is the test. This means its easy to extend, and doesn't need anything hard coded. Failure to install the other types would be a worthwhile check as well, and any that we can't install could be labelled with no-test also

Joibel · 2025-11-26T10:12:49Z

examples/dag-continue-on-fail.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Ok, the second half is designed to fail, so this one should be an expected failure

Joibel · 2025-11-26T10:15:55Z

examples/init-container.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


This one should work, why is it broken?

Joibel · 2025-11-26T10:16:38Z

examples/input-artifact-http.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-environment: "true"


This should be updated to something that works

Joibel · 2025-11-26T10:17:33Z

examples/intermediate-parameters.yaml

 kind: Workflow
 metadata:
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


This is one that needs interaction, so not broken, needs a different label

Joibel · 2025-11-26T11:17:15Z

examples/sidecar-dind.yaml

 metadata:
  generateName: sidecar-dind-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Environment surely

Joibel · 2025-11-26T11:17:29Z

examples/secrets.yaml

 metadata:
  generateName: secrets-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


environment?

Joibel · 2025-11-26T11:17:59Z

examples/retry-on-error.yaml

 metadata:
  generateName: retry-on-error-
+  labels:
+    workflows.argoproj.io/no-test-environment: "true"


untestable I suspect due to randomness

Joibel · 2025-11-26T11:18:38Z

examples/parallelism-template-limit.yaml

 metadata:
  generateName: parallelism-template-limit-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"


Joibel · 2025-11-26T11:19:12Z

examples/output-parameter.yaml

  generateName: output-parameter-
+  labels:
+    workflows.argoproj.io/no-test-broken: "true"
 spec:


Really needs to work

Many examples could pass tests but they were marked as no-test for various reasons. I analyzed all the tests and I marked the one that did not pass as `no-test-broken` or `-no-test-environment` according to why they were not passing. The most common issues for failing example were the image used as part of the workflow since we have a list of allowed images that can run as tests I expanded the list accordingly and I moved some images that were doing easy tasks for already allowed images. For example we had some debian images that were running `sleep` or `cat`, such commands can run from a busybox and it was already allowed so I moved the workflow to use it. As follow up I will update images to their up to date version. Signed-off-by: Gianluca Arbezzano <[email protected]>

gianarb force-pushed the chore/new-example-testing-labeling branch 7 times, most recently from 639e09f to 87317dd Compare November 13, 2025 13:26

gianarb force-pushed the chore/new-example-testing-labeling branch 17 times, most recently from 295a2ba to 9af4b12 Compare November 17, 2025 19:55

gianarb marked this pull request as ready for review November 18, 2025 09:45

Joibel requested a review from Copilot November 18, 2025 16:49

Copilot started reviewing on behalf of Joibel November 18, 2025 16:49 View session

Copilot finished reviewing on behalf of Joibel November 18, 2025 16:51

Copilot AI reviewed Nov 18, 2025

View reviewed changes

gianarb mentioned this pull request Nov 20, 2025

chore: move example suite from testify to go subtest #15002 #15063

Open

gianarb force-pushed the chore/new-example-testing-labeling branch 5 times, most recently from 236ed66 to 46cbf4f Compare November 24, 2025 11:49

chore: improve example test runner to check for new labels argoproj#1…

0b3bff0

…5002 Since we have changed how we identify if an example runs correctly as test this commit applies to the new protocol in the test runner. Signed-off-by: Gianluca Arbezzano <[email protected]>

gianarb force-pushed the chore/new-example-testing-labeling branch 3 times, most recently from 2eeab30 to 6b35e1c Compare November 24, 2025 21:18

Joibel requested changes Nov 25, 2025

View reviewed changes

gianarb force-pushed the chore/new-example-testing-labeling branch 2 times, most recently from 50f93bd to 74e63ed Compare November 25, 2025 20:08

Joibel requested changes Nov 26, 2025

View reviewed changes

gianarb force-pushed the chore/new-example-testing-labeling branch from 74e63ed to 4deedf7 Compare November 27, 2025 22:24

gianarb force-pushed the chore/new-example-testing-labeling branch from 4deedf7 to 2daf284 Compare November 27, 2025 22:26

Joibel mentioned this pull request Dec 3, 2025

[Review] chore: move example suite from testify to go subtest #15002 Joibel/argo-workflows#34

Open

chore: flip filtering label to mark examples with tests #15002 #15014

Are you sure you want to change the base?

chore: flip filtering label to mark examples with tests #15002 #15014

Uh oh!

Conversation

gianarb commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verification

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Joibel left a comment

Choose a reason for hiding this comment

Uh oh!

Joibel Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Joibel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianarb commented Nov 10, 2025 •

edited

Loading

Joibel Nov 25, 2025 •

edited

Loading