Skip to content

feat(monitoring): add VM packets drop alert#3889

Open
Yaguang Tang (yaguangtang) wants to merge 1 commit intomainfrom
add-vm-alert
Open

feat(monitoring): add VM packets drop alert#3889
Yaguang Tang (yaguangtang) wants to merge 1 commit intomainfrom
add-vm-alert

Conversation

@yaguangtang
Copy link
Copy Markdown
Member

No description provided.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces Prometheus alerting for libvirt virtual machine interface packet drops, wiring the new alert rules into the kube-prometheus-stack jsonnet mixin system and documenting the new alerts.

Changes:

  • Add two new alerts for VM interface receive/transmit packet drops (P3) under a new libvirt alert group.
  • Add promtool rule tests covering negative/positive cases for both alerts.
  • Add operator documentation + a release note describing the new alerts.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
roles/kube_prometheus_stack/files/jsonnet/tests.yml Adds promtool test cases for the new libvirt drop alerts.
roles/kube_prometheus_stack/files/jsonnet/mixins.libsonnet Registers the new libvirt jsonnet mixin so its alert group is rendered.
roles/kube_prometheus_stack/files/jsonnet/libvirt.libsonnet Defines LibvirtDomainInterfaceReceiveDrops and LibvirtDomainInterfaceTransmitDrops alerts.
releasenotes/notes/add-libvirt-interface-drop-alerts-5a7e1d9552a1a38e.yaml Adds a feature release note for the new alerts.
doc/source/admin/monitoring.rst Adds runbook-style documentation entries for both new alerts.

Comment thread roles/kube_prometheus_stack/files/jsonnet/libvirt.libsonnet
Comment thread roles/kube_prometheus_stack/files/jsonnet/libvirt.libsonnet
Comment thread doc/source/admin/monitoring.rst
Comment thread doc/source/admin/monitoring.rst
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comment on lines +1 to +26
{
prometheusAlerts+: {
groups+: [
{
name: 'libvirt',
rules:
local dropAlert(direction) =
local capitalized = std.asciiUpper(std.substr(direction, 0, 1)) + std.substr(direction, 1, std.length(direction) - 1);
local alertName = 'LibvirtDomainInterface' + capitalized + 'Drops';
{
alert: alertName,
expr: 'increase(libvirt_domain_interface_stats_' + direction + '_drops_total[5m]) > 0',
'for': '5m',
labels: {
severity: 'P3',
},
annotations: {
summary: 'Libvirt: virtual machine network interface dropping ' + direction + ' packets',
description: 'Domain {{ $labels.domain }} interface {{ $labels.target_device }} has dropped {{ $value }} ' + direction + ' packets in the last 5 minutes. Sustained packet drops can degrade tenant connectivity. Normal behavior is zero drops.',
runbook_url: 'https://vexxhost.github.io/atmosphere/admin/monitoring.html#' + std.asciiLower(alertName),
},
};
[
dropAlert('receive'),
dropAlert('transmit'),
],
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with the other custom alert files, consider defining helper locals at file scope and keeping rules as a plain array literal. For example, coredns.libsonnet and nginx.libsonnet define locals at the top of the file and use rules: [ ... ], which is easier to scan and keeps alert rule objects more uniform.

Suggested change
{
prometheusAlerts+: {
groups+: [
{
name: 'libvirt',
rules:
local dropAlert(direction) =
local capitalized = std.asciiUpper(std.substr(direction, 0, 1)) + std.substr(direction, 1, std.length(direction) - 1);
local alertName = 'LibvirtDomainInterface' + capitalized + 'Drops';
{
alert: alertName,
expr: 'increase(libvirt_domain_interface_stats_' + direction + '_drops_total[5m]) > 0',
'for': '5m',
labels: {
severity: 'P3',
},
annotations: {
summary: 'Libvirt: virtual machine network interface dropping ' + direction + ' packets',
description: 'Domain {{ $labels.domain }} interface {{ $labels.target_device }} has dropped {{ $value }} ' + direction + ' packets in the last 5 minutes. Sustained packet drops can degrade tenant connectivity. Normal behavior is zero drops.',
runbook_url: 'https://vexxhost.github.io/atmosphere/admin/monitoring.html#' + std.asciiLower(alertName),
},
};
[
dropAlert('receive'),
dropAlert('transmit'),
],
local dropAlert(direction) =
local capitalized = std.asciiUpper(std.substr(direction, 0, 1)) + std.substr(direction, 1, std.length(direction) - 1);
local alertName = 'LibvirtDomainInterface' + capitalized + 'Drops';
{
alert: alertName,
expr: 'increase(libvirt_domain_interface_stats_' + direction + '_drops_total[5m]) > 0',
'for': '5m',
labels: {
severity: 'P3',
},
annotations: {
summary: 'Libvirt: virtual machine network interface dropping ' + direction + ' packets',
description: 'Domain {{ $labels.domain }} interface {{ $labels.target_device }} has dropped {{ $value }} ' + direction + ' packets in the last 5 minutes. Sustained packet drops can degrade tenant connectivity. Normal behavior is zero drops.',
runbook_url: 'https://vexxhost.github.io/atmosphere/admin/monitoring.html#' + std.asciiLower(alertName),
},
};
{
prometheusAlerts+: {
groups+: [
{
name: 'libvirt',
rules: [
dropAlert('receive'),
dropAlert('transmit'),
],

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants