RepoMirrors
/
prometheus
mirror of https://github.com/prometheus/prometheus
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improve comments around resending resolved alerts (#13990) 

Signed-off-by: George Robinson <george.robinson@grafana.com>
This commit is contained in:
George Robinson 2024-04-25 13:18:50 +01:00 committed by GitHub
parent 4ac78063ee
commit dde2e5eb73
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 11 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
rules/alerting.go
View File

@ -457,8 +457,17 @@ func (r *AlertingRule) Eval(ctx context.Context, ts time.Time, query QueryFunc,
} }
} }
// If the alert was previously firing, keep it around for a given // If the alert is resolved (was firing but is now inactive) keep it for
// retention time so it is reported as resolved to the AlertManager. // at least the retention period. This is important for a number of reasons:
//
// 1. It allows for Prometheus to be more resilient to network issues that
// would otherwise prevent a resolved alert from being reported as resolved
// to Alertmanager.
//
// 2. It helps reduce the chance of resolved notifications being lost if
// Alertmanager crashes or restarts between receiving the resolved alert
// from Prometheus and sending the resolved notification. This tends to
// occur for routes with large Group intervals.
if a.State == StatePending || (!a.ResolvedAt.IsZero() && ts.Sub(a.ResolvedAt) > resolvedRetention) { if a.State == StatePending || (!a.ResolvedAt.IsZero() && ts.Sub(a.ResolvedAt) > resolvedRetention) {
delete(r.active, fp) delete(r.active, fp)
} }