[EPIC] Monitoring & alerting framework

[richardtief]

Context

Define which functionalities rely on labels and which labels these are. This should be done for the entire stack (e.g. metrics, logs, alerts, playbooks, Plutono oder Perses, UIs [Supernova, ...] ...)
This also includes the identification of mandatory labels (e.g. owner-info etc.) that are used for e.g alert routing.

The goal of this EPIC is to define a framework including a common set of labels that can be used in all components of the monitoring and alerting stack. These labels should then also be enforced, preset and integrated into Greenhouse.

Acceptance criteria

1. Define Prometheus external labels (e.g. organization, geographic region, cluster name, cluster type usage)
2. Include something like support group and service labels to properly route alerts
3. Make Slack notification template, Greenhouse Supernove UI aware of those labels
4. Preset alert routing to those labels
5. Documentation
6. Add to ADR feat: init kube monitoring labels adr documentation#13

[github-actions]

This issue is stale because it has been open for 90 days with no activity.