Kustodian prometheus

[clarkezone]

Add a Prometheus listener and register metrics tracking

1. Number of instances of Kustodian pods running
2. Number of successful script runs
   TODO what else?

Get basic metrics going

• Build a test mop script that waits 5 seconds and progresses
• Verify test script in test cluster built via capz
• Delete previous mop run
• Deploy prometheus / grafana to test cluster
• Add prometheus metric exporter infra with metric
• Redeploy Kustodian with prom exporting
• verify metric

Demo on my machine:

• keep monitoring up as demo progresses

  • mop only on user nodes, monitoring on system

• Demo blocker 2: fix metric logic in kustodian
  put in more logging

Demo is repeatable
PR for metrics changes to kustodian
Move kustodian service and servicemonitor into helm chart
Install dashboard for kustodian
Add monitoring and logging stack
Add service / service monitoring for kustodian
Scripts for port forward

Add kustodian app annotation

create an issue on capz for hello world not working with KIND

[clarkezone]

dd