fix: Enforce cluster information on monitoring alerts #3929
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Rael Garcia <[email protected]>
Signed-off-by: Rael Garcia <[email protected]>
Signed-off-by: Rael Garcia <[email protected]>
raelga
changed the title
Collaborator
|
/test e2e-parallel |
Signed-off-by: Rael Garcia <[email protected]>
Collaborator
Author
|
@sclarkso There was a legit error parsing the |
Collaborator
|
/test e2e-parallel |
Contributor
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: raelga, stevekuznetsov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AROSLSRE-91
https://redhat-external.slack.com/archives/C075PHEFZKQ/p1769620814361439
This pull request updates Prometheus alerting rules and their tests to add cluster-awareness to all relevant alerts. The changes ensure that alerts are correctly grouped and fired on a per-cluster basis, improving accuracy and scalability in multi-cluster environments. Several PromQL expressions are updated to use
group by (cluster)and to join service health checks with cluster membership. Corresponding test files are also updated to reflect these changes, ensuring correct alert firing and label expectations.Prometheus Alert Rule Updates for Cluster Awareness:
group by (cluster)and/orunless on(cluster)to ensure alerts are evaluated and fired per cluster. This affects both the main rules and the generated Bicep templates. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]Test Updates for Cluster Labeling and Cluster-Aware Logic:
clusterlabel in input series and expected alert labels, ensuring that alerts are validated for correct cluster-specific behavior. [1] [2] [3] [4] [5] [6] [7] [8] [9]Improvements to Service Discovery and Alert Coverage:
group by (cluster) (up{job="kube-state-metrics", cluster=~".*-svc-\d+"}) unless on(cluster) .... Test coverage is expanded for various scenarios (metric missing, metric present, metric goes down). [1] [2]These changes collectively make alerting more robust and accurate in multi-cluster Kubernetes environments, and the updated tests ensure the new logic is thoroughly validated.