Aggregating dynamic step output parameters, including some failed steps

[abuob]

Hello everyone,

My example might be a bit contrived and I could imagine that I'm "holding the tool wrong", so any input is welcome.

My current scenario is a CI-pipeline on a monorepo:

• Detect changes; generate a list of necessary tests
• Dynamically run those tests on separate steps using withParams
• Last step: Collect the results (which failed, which succeeded) and then act based on those (trigger downstream workflows, send notifications, etc.)

The current setup looks somewhat similar to this:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-pipeline
spec:
  entrypoint: my-pipeline
  templates:
    - name: my-pipeline
      steps:
        - - name: detect-changes
            template: detect-changes-template
        - - name: stage
            template: stage-template
            arguments:
              parameters:
                - name: stage-name
                  value: '{{item.stageName}}'
            withParam: '{{steps.detect-changes.outputs.parameters.stages-to-be-run}}'
            continueOn:
              failed: true
        - - name: handle-results
            template: handle-results-template
            arguments:
              parameters:
                - name: aggregated-results
                  value: '{{steps.stage.outputs.parameters.stage-result}}'
    - name: stage-template
      inputs:
        parameters:
          - name: stage-name
      outputs:
        parameters:
          - name: stage-name
            value: '{{inputs.parameters.stage-name}}'
          - name: stage-result
            valueFrom:
              path: /stage_result.json
      container:
        image: 'some-image'
        command: [sh, -c]
        args: [
            # run-stage will run the tests, create a "stage_result.json" containing results and metadata, then exit non-zero if the tests were not successful
            'run-stage {{inputs.parameters.stage-name}}'
          ]
    - name: detect-changes-template
      outputs:
        parameters:
          - name: stages-to-be-run
            valueFrom:
              path: /stages_to_be_run.json
      container:
        image: 'some-image'
        command: [sh, -c]
        args: [
            '
            detect-changes > /stages_to_be_run.json
            ',
          ]
    - name: handle-results-template
      inputs:
        parameters:
          - name: aggregated-results
      container:
        image: 'some-image'
        command: [sh, -c]
        args: [
            '
            result-handling {{inputs.parameters.aggregated-results}}
            ',
          ]

It might be a bit too much pseudo-code but I hope it's understandable what I'm trying to achieve:

• The tests can fail, but regardless, a stage_result.json should be generated
• I want to collect all the results in a handle-results-step and then act based upon the results.

Currently, I seem to only receive the output-parameters of the succeeded steps - not the ones that failed. Is there any way to get access to outputs of failed steps?

The motivation: We would like a step that fails to appear red in the UI, hence we exist non-zero. However, we would still like to retrieve the "stage-results" in the result-handling, as it contains valuable metadata - is there a way for this?

Many thanks in advance! 🙂

[jswxstw]

processAggregateNodeOutputs is designed to ignore outputs from failed nodes, but I think your request is reasonable; maybe you can submit a feature request for it.

argo-workflows/workflow/controller/operator.go

Lines 3428 to 3430 in 5d75b62

	if node.Outputs == nil || node.Phase != wfv1.NodeSucceeded {
	continue
	}