Silence notifications

[mattbostock]

If a silence expires and the root cause that triggered the alert is not yet resolved, it can generate confusion when the alert starts firing. This can be particularly frustrating if an alert becomes unsilenced over a weekend or if the person that originally silenced the alert is on vacation.

I would find it useful if Alertmanager could send a notification when:

a) silences are created (e.g. so they show up in Slack/HipChat)
b) silences have just expired (which helps explain why the alerts start firing again)

You could take this one step further by sending 'silence reminders' to the silence author, for example on a Friday before the weekend if the silence would expire over the weekend.

[brian-brazil]

I've seen features like this in the past, and they're pretty much pure spam.

Including in the notification if it's due to an expired silence is useful.

[mattbostock]

I like the idea of including details of an expired (or recently expired) silence in the alert notification.

What about this feature made it spammy? I can guess what you mean, but if you have any examples that'd be really helpful (particularly regarding reminding people about silences expiring over weekends).

[brian-brazil]

What about this feature made it spammy? I can guess what you mean, but if you have any examples that'd be really helpful (particularly regarding reminding people about silences expiring over weekends).

It was often directed towards a mailing list, and it turns out between automation and general operator activity that there are many many silences created, deleted and expired in a week.

In addition, emailing me to tell me that I create or deleted a silence or that it's going to expire is useless - I already know that.

In my opinion such notifications should be treated in the same fashion as email notifications generally, and eliminated with extreme prejudice.

[solongtony]

There is another request to address this situation in a different way, using badges instead of alerts. #1497
That provides the same information but avoids the "spam" message factor.

[anarcat]

i would find having silence notifications useful. i've been drilling through the code to figure out whether there would be some webhook we could use, but couldn't find anything, and was actually surprised.

just like alerts, yes, this can be noisy, but we don't use silences lightly, and it's useful to communicate to teammates stuff like "okay, anarcat is on top of this one".

this feels especially important since we don't show notifications when alerts are resolved but silenced, see #226... because right now, if i look at a stream of notification, it can end with an alert marked as "firing" and never resolved, and i will never figure out why unless i actually know there was a silence posted.

at least not just from the notification stream. of course, i can just go look at active alerts and silences and figure it out...

i will also politely note that, if you weed out receiver-specific issues (pagerduty and slack are prominent there), this issue and #226 are the most voted (:+1:) issues in the issue queue right now, so somehow someone somewhere thinks this is a good idea. ;)

[LeoQuote]

I implemented a gadget to scan silences constantly, hope this helps

https://github.com/LeoQuote/silence-scanner

[copolycube]

I implemented a gadget to scan silences constantly, hope this helps

can it be plugged back into alertmanager so that it sends specific alerts (aka. reminder of silences) on a regular basis ?

[LeoQuote]

Not supported currently, I think the silence and alertgroup have different data structure, it could be hard to transform them. You can try it if you like.

[anarcat]

one thing that could help in implementing this on our own (without patching alertmanager to implement direct notifications) would be to log those silences and, lo and behold, there's been a PR open about this for .. herm... 3 years now:

#2536

alternatively, i've been thinking we could just hijack the /silence POST endpoint and log (or relay) that at the webserver level somehow. not sure you can proxy to two backends simultaneously in nginx though...