Monitor storagebox usage

[mweinelt]

Is your feature request related to a problem? Please describe.

We started sharing a dedicated storagebox among multiple machines. We should monitor its fullness so we know when we need to investigate space usage.

Describe the solution you'd like

https://github.com/fleaz/prometheus-storagebox-exporter

Describe alternatives you've considered

Managing quotas for all servers.

Additional context

[Erethon]

Describe the solution you'd like
https://github.com/fleaz/prometheus-storagebox-exporter

I see this isn't packaged in nixpkgs yet.

Since I was thinking of doing something similar for my personal infra, if no one else has started work on this yet, I can happily take it, package the exporter in nixpkgs and open a PR in infra to have it running.

[jfly]

Please go for it, @Erethon! Thanks for the help =) I could use this as well.

[jfly]

On a related note, @mweinelt, have we set up (or considered setting up) some sort of "deadman's switch" for our backups? While it's nice to monitor all the individual things that could go wrong (such as running out of disk space), it's also nice to know that if things somehow start silently failing, we'll still find out about it.

I do this at home with a prometheus alert, but it's a bit awkward to generate the "when the backup last succeeded" metric. I've also used https://healthchecks.io/, which is simple and works great.

[Erethon]

Please go for it, @Erethon! Thanks for the help =) I could use this as well.

Ok, will do so in the next few days. Feel free to assign this to me, since I don't have the permissions to do this myself.

As for your question about identifying if a job failed, depending on the size of the backups, we might be able to figure this out using anomaly detection. I.e. if the average in 30 days is for backups to grow > X bytes and they haven't, we can alert on this. This won't work however if everything is always deduplicated and the size of the backup can sometimes stay the same.

Alternatively, if the jobs run through a systemd timer, we can use the systemd exporter to figure out if a backup operation succeeded or failed.