From 5027863c40d318ae68894c0908aacb2a476ac338 Mon Sep 17 00:00:00 2001 From: Jennifer Villa Date: Sat, 30 Sep 2023 10:23:54 -0400 Subject: [PATCH 1/4] Clarify what happens when a rule group takes too long to execute Namely, call out that all subsequent evaluations will be skipped until the initial evaluation completes. Signed-off-by: Jennifer Villa --- docs/configuration/recording_rules.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/configuration/recording_rules.md b/docs/configuration/recording_rules.md index eda0214b3..6ccde040d 100644 --- a/docs/configuration/recording_rules.md +++ b/docs/configuration/recording_rules.md @@ -147,3 +147,7 @@ by the rule are discarded, and if it's an alerting rule, _all_ alerts for the rule, active, pending, or inactive, are cleared as well. The event will be recorded as an error in the evaluation, and as such no stale markers are written. + +# Failed rule evaluations due to slow evaluation + +If a rule group hasn't finished evaluating before the next evaluation is supposed to start, the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration. From 601e5b902844f9fb02c422ec516dee642277c6a0 Mon Sep 17 00:00:00 2001 From: Jennifer Villa Date: Sat, 30 Sep 2023 10:26:29 -0400 Subject: [PATCH 2/4] Update recording_rules.md specified the evaluation interval defines when the next evaluation should start. Signed-off-by: Jennifer Villa --- docs/configuration/recording_rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration/recording_rules.md b/docs/configuration/recording_rules.md index 6ccde040d..0c8aef637 100644 --- a/docs/configuration/recording_rules.md +++ b/docs/configuration/recording_rules.md @@ -150,4 +150,4 @@ written. # Failed rule evaluations due to slow evaluation -If a rule group hasn't finished evaluating before the next evaluation is supposed to start, the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration. +If a rule group hasn't finished evaluating before its next evaluation is supposed to start (as defined by the `evaluation_interval`), the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration. From 65ffa0c511f3eef8c813680b7e43740bb8a1705d Mon Sep 17 00:00:00 2001 From: Jennifer Villa Date: Mon, 2 Oct 2023 21:37:29 -0400 Subject: [PATCH 3/4] Update recording_rules.md adding Josh's suggestion Signed-off-by: Jennifer Villa --- docs/configuration/recording_rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration/recording_rules.md b/docs/configuration/recording_rules.md index 0c8aef637..90bd3df73 100644 --- a/docs/configuration/recording_rules.md +++ b/docs/configuration/recording_rules.md @@ -150,4 +150,4 @@ written. # Failed rule evaluations due to slow evaluation -If a rule group hasn't finished evaluating before its next evaluation is supposed to start (as defined by the `evaluation_interval`), the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration. +If a rule group hasn't finished evaluating before its next evaluation is supposed to start (as defined by the `evaluation_interval`), the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration of the rule group. From ea7bec4cc7dd8984d86996e7e711e6884c38425b Mon Sep 17 00:00:00 2001 From: Jennifer Villa Date: Mon, 2 Oct 2023 21:44:54 -0400 Subject: [PATCH 4/4] Update recording_rules.md updated language to be a bit more clear Signed-off-by: Jennifer Villa --- docs/configuration/recording_rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration/recording_rules.md b/docs/configuration/recording_rules.md index 90bd3df73..48ab951f9 100644 --- a/docs/configuration/recording_rules.md +++ b/docs/configuration/recording_rules.md @@ -150,4 +150,4 @@ written. # Failed rule evaluations due to slow evaluation -If a rule group hasn't finished evaluating before its next evaluation is supposed to start (as defined by the `evaluation_interval`), the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration of the rule group. +If a rule group hasn't finished evaluating before its next evaluation is supposed to start (as defined by the `evaluation_interval`), the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the metric produced by the recording rule. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration of the rule group.