alertmanager/notify
TJ Hoplock f00025d037
feat: add counter to track alerts dropped outside of time_intervals (#3565)
* feat: add counter to track alerts dropped outside of time_intervals

Addresses: #3512

This adds a new counter metric `alertmanager_alerts_supressed_total`
that is incremented by `len(alerts)` when an alert is suppressed for
being outside of a time_interval, ie inside of a mute_time_intervals or
outside of an active_time_intervals.

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* test: add time interval suppression metric checks for notify

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* test: fix failure message log values in notifier

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* ref: address PR feedback for #3565

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* fix: track suppressed notifications metric for inhibit/silence

Based on PR feedback:

https://github.com/prometheus/alertmanager/pull/3565/files#r1393068026

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* fix: broken notifier tests

- fixed metric count check to properly check the diff between
  input/output notifications from the suppression to compare to suppression
metric, was previously inverted to compare to how many notifications it
suppressed.
- stopped using `Reset()` to compare collection counts between the
  multiple stages that are executed in `TestMuteStageWithSilences()`.
the intent was to compare a clean metric collection after each stage
execution, but the final stage where all silences are lifted results in
no metric being created in the test, causing `prom_testutil.ToFloat64()`
to panic. changed to separate vars to check counts between each stage,
with care to consider prior counts.

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* rename metric and add constants

Signed-off-by: gotjosh <josue.abreu@gmail.com>

---------

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
Signed-off-by: gotjosh <josue.abreu@gmail.com>
Co-authored-by: gotjosh <josue.abreu@gmail.com>
2024-02-13 11:17:24 +00:00
..
discord feat: implement webhook_url_file for discord and msteams (#3555) 2024-02-07 10:33:46 +00:00
email Newlines in smtp password file are now ignored 2024-01-22 21:26:18 +01:00
msteams feat: implement webhook_url_file for discord and msteams (#3555) 2024-02-07 10:33:46 +00:00
opsgenie golangci-lint: enable testifylint linter 2023-12-10 08:50:03 +00:00
pagerduty use Go standard errors 2023-12-08 16:44:13 +01:00
pushover Add missing ttl in pushhover (#3474) 2023-08-31 17:35:04 +02:00
slack golangci-lint: enable testifylint linter 2023-12-10 08:50:03 +00:00
sns golangci-lint: enable testifylint linter 2023-12-10 08:50:03 +00:00
telegram Support loading Telegram bot token from file 2023-01-22 16:35:01 +03:00
test feat: add template.FromGlobsWithAdditionalFuncs (#3174) 2022-12-16 10:13:13 -04:00
victorops use Go standard errors 2023-12-08 16:44:13 +01:00
webex Remove the dedicated field to APIURL in favour of the configuration one 2022-12-15 11:51:45 -04:00
webhook golangci-lint: enable testifylint linter 2023-12-10 08:50:03 +00:00
wechat use Go standard errors 2023-12-08 16:44:13 +01:00
notify.go feat: add counter to track alerts dropped outside of time_intervals (#3565) 2024-02-13 11:17:24 +00:00
notify_test.go feat: add counter to track alerts dropped outside of time_intervals (#3565) 2024-02-13 11:17:24 +00:00
util.go use Go standard errors 2023-12-08 16:44:13 +01:00
util_test.go Use the new truncation in bytes functions to ensure strings are not butchered 2022-11-11 15:15:09 +00:00