💡 [REQUEST] - Prometheus SLO Recording Rule examples

[skinlayers]

Summary

Please provide examples of prometheus recording rules that could be used with the sample prometheus SLOs.

Basic Example

A Prometheus (Mimir) latency SLO is taking several minutes to return all data points when calculating the 28d window in one of our production environment. We'd like to leverage prometheus recording rules to reduce the load the SLO causes. However, SLO Generator's [window] placeholder isn't compatible with recording rules that already include a time range e.g. prometheus_http_request_duration_seconds_count{}[5m].

So far, the only potential workarounds I can think of are using a recording rule without a time range to just reduce cardinality, or to create a separate error_budget_policies in config.yaml with subqueries for the window values, instead of the number of seconds.

Screenshots

No response

Drawbacks

I don't see any drawbacks.

Unresolved questions

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[lvaylet]

Hi @skinlayers, thanks for reporting this behavior.

I am not a Prometheus expert so I will probably need to do a bit of research.

Could you provide more context regarding this statement?

SLO Generator's [window] placeholder isn't compatible with recording rules that already include a time range e.g. prometheus_http_request_duration_seconds_count{}[5m].

If this (lack of) compatibility is the issue here, and you already know how to write recording rules that pre-compute and cache the 28d SLOs, another workaround may be to work on the queries directly.