PRs can still be merged with failing workflows

[iainlane]

We had this failing run but we still merged #147 and it broke users.

It's because this check isn't required. Conditional checks can't simply be made required, or else they will block pull requests which don't change the files they trigger on.

Possible fixes:

1. Make checks unconditional, just run them always
2. Move the conditionality to runtime using a "changed files" type of action
3. Use our deployment of https://github.com/palantir/policy-bot and make these checks conditional if triggered by writing a .policy.yml containing rules that replicate the file triggers, or using a generator (we have one internally in deployment_tools - we could open source that by moving it to this repo).

[Elfo404]

Option 3 should not be viable due to #497 (comment)

[dblinkhorn]

What about if we implement an "orchestrator" workflow that analyzes the changed files and decides which test workflows need to be run? This workflow can be set as a required check, but if none of the tests are triggered it will just return success.

name: Test Workflow Orchestrator

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  pull-requests: read 

jobs:
# check the changed files and set outputs used to trigger individual test jobs below
  changes:
    runs-on: ubuntu-latest
    outputs:
      vault: ${{ steps.filter.outputs.vault }}
      gar: ${{ steps.filter.outputs.gar }}
      # etc...
    steps:
      - id: filter
        uses: dorny/paths-filter@de90cc6d5d3da4d2a4f21f4e0f6c1e73948b9a6
        with:
          filters: |
            vault:
              - 'actions/get-vault-secrets/**'
              - '.github/workflows/test-get-vault-secrets.yaml'
            gar:
              - 'actions/login-to-gar/**'
              - '.github/workflows/test-login-to-gar.yaml'

# invoke the tests which are needed based on changed files in diff

  vault-test:
    needs: changes
    if: needs.changes.outputs.vault == 'true'
    uses: ./.github/workflows/test-get-vault-secrets.yaml
    permissions:           # forward permissions to called workflow if needed
      contents: read
      id-token: write
    secrets: inherit       # forward secrets

  gar-test:
    needs: changes
    if: needs.changes.outputs.gar == 'true'
    uses: ./.github/workflows/test-login-to-gar.yaml

# etc...

[zerok]

That's pretty much what policy-bot would do. The question is, if we are really at the point where we need that here given that it requires a service running somewhere.

[iainlane]

That's pretty much what policy-bot would do. The question is, if we are really at the point where we need that here given that it requires a service running somewhere.

We do have the service running for grafana - it just can't be used due to the issue @Elfo404 mentioned 😢