The expression alertmanager_cluster_members{job="alertmanager"}[5m]) is assumed to return
one series for each alertmanager instance in the cluster. When running inside Kubernetes,
alertmanager pods can get evicted and rescheduled. This can change the instance label and
produce a new series for that alertmanager instance.
When the same pod gets evicted several times in a row, there will be a short interval in which
Prometheus will return values from both the new series and the old series.
As a result, counting the number of series for the alertmanager_cluster_members metric
will overestimate the number of instances in the given cluster.
This commit modifies the the AlertmanagerMembersInconsistent alert to increase the for clause to 15m
in order to reduce the probability of a false positive.
Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>
* Implements a Grafana dashboard to the mixin.
The dashboard aims to show an overview of the overall health of Alertmanager.
Signed-off-by: ArthurSens <arthursens2005@gmail.com>
While the documentation for the matchers themselves was updated, we
missed the examples.
I propose to merge this into the release-0.22 branch so that it gets
included in the ongoing release.
Signed-off-by: beorn7 <beorn@grafana.com>
* Drop /vendor.
* Update Go to 1.16.
* Update djfarrelly/maildev to 1.1.0.
* Update protoc to 3.15.8.
* Update mixin test for Go 1.16.
* Bump Go modules.
Signed-off-by: SuperQ <superq@gmail.com>
The alert was just looking at the minimum across integrations. So a
complete failure of one integration would be masked by a still worknig
other integration. With this fix, the `integration` label is retained
(as it was already expected by the `description`), and thus any
failing integration will trigger the alert.
In addition, an `alertmanagerCriticalIntegrationsRegEx` is provided
that allows to mark integrations as critical. Integrations that are
not used to deliver critical alerts, or those that are just there for
auditing and logging purposes can now be configured to only trigger a
warning alert if they fail.
Signed-off-by: beorn7 <beorn@grafana.com>
To aggregate by all possible labels use '...' as the sole label name.
This effectively disables aggregation entirely, passing through all
alerts as-is. This is unlikely to be what you want, unless you have
a very low alert volume or your upstream notification system performs
its own grouping. Example: group_by: [...]
Signed-off-by: Kyryl Sablin <kyryl.sablin@schibsted.com>
* Update Architecture diagram
Update diagram from sketch to vector.
Add draw.io XML source file.
Update README.md to display master doc/arch.jpg
Signed-off-by: Silvio Gissi <silvio@gissilabs.com>
* Updated README.md with relative link to architecture doc.
* Updated Architecture document from JPG to SVG
Signed-off-by: Silvio Gissi <silvio@gissilabs.com>
* Small fix in graph.
* Updated font to align with Prometheus architecture.
Signed-off-by: Silvio Gissi <silvio@gissilabs.com>
* Embedded images at arch.svg
* Removed images from SVG, update source XML
Simon Pasquier
clyang82
fpetkovski
Arthur Silva Sens
beorn7
SuperQ
Bor Grošelj Simić
Tom Wilkie
Julien Pivotto
Dominik-K
Max Leonard Inden
kirillsablin
Martin Chodur
Silvio Gissi
Tom Paine
Jose Donizetti
Max Chadwick
Fabian Reinartz
louis
beorn7