This commit removes the metrics from the compat package
in favour of the existing logging and the additional tools
at hand, such as amtool, to validate Alertmanager configurations.
Due to the global nature of the compat package, a consequence
of config.Load, these metrics have proven to be less useful
in practice than expected, both in Alertmanager and other projects
such as Mimir.
There are a number of reasons for this:
1. Because the compat package is global, these metrics cannot be
reset each time config.Load is called, as in multi-tenant
projects like Mimir loading a config for one tenant would reset
the metrics for all tenants. This is also the reason the metrics
are counters and not gauges.
2. Since the metrics are counters, it is difficult to create
meaningful dashboards for Alertmanager as, unlike in Mimir,
configurations are not reloaded at fixed intervals, and as such,
operators cannot use rate to track configuration changes
over time.
In Alertmanager, there are much better tools available to validate
that an Alertmanager configuration is compatible with the UTF-8
parser, including both the existing logging from Alertmanager
server and amtool check-config.
In other projects like Mimir, we can track configurations for
individual tenants using log aggregation and storage systems
such as Loki. This gives operators far more information than
what is possible with the metrics, including the timestamp,
input and ID of tenant configurations that are incompatible
or have disagreement.
Signed-off-by: George Robinson <george.robinson@grafana.com>
This commit changes the metrics in the compat package from gauges
to counters. The reason for this is that in some cases the gauge
should behave like a gauge (i.e. loading configurations) but in
other cases should behave like a counter (i.e. HTTP requests).
Second, because the compat package is a global package
(due to how config.Load works), in tenanted systems like Cortex
and Mimir it was non-trivial to reset the gauges per tenant
each time their configuration was reloaded.
Instead, it's easier to compute the rate of increase as 0 instead
of check that the gauge is 0 to know if UTF-8 strict mode can be
enabled.
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Add metrics to matchers compat package
This commit adds the following metrics to the compat package:
alertmanager_matchers_parse
alertmanager_matchers_disagree
alertmanager_matchers_incompatible
alertmanager_matchers_invalid
With a label called origin to differentiate the different sources
of inputs: the configuration file, the API, and amtool.
The disagree_total metric is incremented when an input is invalid
in both parsers, but results in different parsed representations,
then there is disagreement. This should not happen, and suggests
their is either a bug in one of the parsers or a mistake in the
backwards compatible guarantees of the matchers/parse parser.
The incompatible_total metric is incremented when an input is valid
in pkg/labels, but not the UTF-8 parser in matchers/parse. In such
case, the matcher should be updated to be compatible. This often
means adding double quotes around the right hand side of the matcher.
For example, foo="bar".
The invalid_total metric is incremented when an input is invalid
in both parsers. This was never a valid input.
The tests have been updated to check the metrics are incremented
as expected.
Signed-off-by: George Robinson <george.robinson@grafana.com>
---------
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Make functions in compat package public
This commit makes functions in the compat package public. These
functions are useful for software that builds on top of the
Alertmanager that also need to migrate from classic mode to UTF-8.
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Fix lint
Signed-off-by: George Robinson <george.robinson@grafana.com>
---------
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Fix UTF-8 not supported in group_by
This commit fixes missing UTF-8 support in the group_by for routes.
Signed-off-by: George Robinson <george.robinson@grafana.com>
---------
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Do not allow unquoted escape sequences
This commit updates the matchers parser to reject unquoted
openmetrics escape sequences. As an example, foo=bar\n
will no longer parse, and must instead be written as
foo="bar\n". This avoids an issue where the input is valid
in both the matchers and classic parsers, but results
in two different parsings.
---------
Signed-off-by: George Robinson <george.robinson@grafana.com>
* Add adapter package for parser feature flag
This commit adds the compat package allowing users to switch
between the new matchers/parse parser and the old pkg/labels parser.
The new matchers/parse parser uses a fallback mechanism where if
the input cannot be parsed in the new parser it then attempts to
use the old parser. If an input is parsed in the old parser but
not the new parser, then a warning log is emitted.
---------
Signed-off-by: George Robinson <george.robinson@grafana.com>
George Robinson
Matthieu MOREL