Commit Graph

8 Commits

Author SHA1 Message Date
stuart nelson
80f2eeb2ca
Fix resolved alerts still inhibiting (#1331)
* inhibit: update inhibition cache when alerts resolve

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* inhibit: remove unnecessary fmt.Sprintf

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* inhibit: add unit tests

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* inhibit: use NopLogger in tests

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Update old alert with result of merge with new

On ingest, alerts with matching fingerprints are
merged if the new alert's start and end times
overlap with the old alert's.

The merge creates a new alert, which is then
updated in the internal alert store.

The original alert is not updated (because merge
creates a copy), so it is never marked as resolved
in the inhibitor's reference to it.

The code within the inhibitor relies on skipping
over resolved alerts, but because the old alert is
never updated it is never marked as resolved. Thus
it continues to inhibit other alerts until it is
cleaned up by the internal GC.

This commit updates the struct of the old alert
with the result of the merge with the new alert.

An alternative would be to always update the
inhibitor's internal cache of alerts regardless of
an alert's resolve status.

Signed-off-by: stuart nelson <stuartnelson3@gmail.com>

* Update inhibitor cache even if alert is resolved

This seems like a better choice than the previous
commit. I think it is more sane to have the
inhibitor update its own cache, rather than having
one of its pointers updated externally.

Signed-off-by: stuart nelson <stuartnelson3@gmail.com>
2018-04-18 16:26:04 +02:00
Frederic Branczyk
f4c226c7d5
inhibit: Fix race in stopping inhibitor 2017-11-24 12:05:52 +01:00
Alin Sinpalean
dc3c78e6ae Prevent alerts from inhibiting themselves (#1017)
* Prevent alerts from inhibiting themselves.

* Don't inhibit alerts that match the source filter.

* No-op, nudge CircleCI.
2017-11-07 11:30:14 +01:00
Julius Volz
947970af44 Convert Alertmanager to use non-global go-kit loggers
Fixes https://github.com/prometheus/alertmanager/issues/1040
2017-10-22 00:20:40 -07:00
Frederic Branczyk
de24c5b7ec Fix inhibit race (#1032)
* inhibit: restructure stop logic, to prevent race condition

* vendor oklog/oklog/pkg/group
2017-10-07 13:01:37 +02:00
stuart nelson
6a909abf17 Add processing status field to alert 2017-04-27 14:18:52 +02:00
Carl Henrik Lunde
2935ff84e7 inhibitrule: defer Unlock to fix race 2016-10-08 22:07:29 +02:00
Fabian Reinartz
3931d4e64b *: restructure package tree
This commit packages up individual modules and removes the top-level
main package.
2016-08-09 14:24:52 +02:00