b50f9c1c84
* scrape: add label limits per scrape Add three new limits to the scrape configuration to provide some mechanism to defend against unbound number of labels and excessive label lengths. If any of these limits are broken by a sample from a scrape, the whole scrape will fail. For all of these configuration options, a zero value means no limit. The `label_limit` configuration will provide a mechanism to bound the number of labels per-scrape of a certain sample to a user defined limit. This limit will be tested against the sample labels plus the discovery labels, but it will exclude the __name__ from the count since it is a mandatory Prometheus label to which applying constraints isn't meaningful. The `label_name_length_limit` and `label_value_length_limit` will prevent having labels of excessive lengths. These limits also skip the __name__ label for the same reasons as the `label_limit` option and will also make the scrape fail if any sample has a label name/value length that exceed the predefined limits. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * scrape: add metrics and alert to label limits Add three gauge, one for each label limit to easily access the limit set by a certain scrape target. Also add a counter to count the number of targets that exceeded the label limits and thus were dropped. This is useful for the `PrometheusLabelLimitHit` alert that will notify the users that scraping some targets failed because they had samples exceeding the label limits defined in the scrape configuration. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * scrape: apply label limits to __name__ label Apply limits to the __name__ label that was previously skipped and truncate the label names and values in the error messages as they can be very very long. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> * scrape: remove label limits gauges and refactor Remove `prometheus_target_scrape_pool_label_limit`, `prometheus_target_scrape_pool_label_name_length_limit`, and `prometheus_target_scrape_pool_label_value_length_limit` as they are not really useful since we don't have the information on the labels in it. Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com> |
||
|---|---|---|
| .. | ||
| .gitignore | ||
| Makefile | ||
| README.md | ||
| alerts.jsonnet | ||
| alerts.libsonnet | ||
| config.libsonnet | ||
| dashboards.jsonnet | ||
| dashboards.libsonnet | ||
| go.mod | ||
| go.sum | ||
| jsonnetfile.json | ||
| mixin.libsonnet | ||
| tools.go | ||
README.md
Prometheus Mixin
This is work in progress. We aim for it to become a good role model for alerts and dashboards eventually, but it is not quite there yet.
The Prometheus Mixin is a set of configurable, reusable, and extensible alerts and dashboards for Prometheus.
To use them, you need to have jsonnet (v0.13+) and jb installed. If you
have a working Go development environment, it's easiest to run the following:
$ go get github.com/google/go-jsonnet/cmd/jsonnet
$ go get github.com/google/go-jsonnet/cmd/jsonnetfmt
$ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
Note: The make targets lint and fmt need the jsonnetfmt binary, which is
available from v.0.16.0 in the Go implementation of jsonnet. If your jsonnet version is older than 0.16.0 you have to either upgrade or install the C++ version of
jsonnetfmt if you want to use make lint
or make fmt.
Next, install the dependencies by running the following command in this directory:
$ jb install
You can then build a prometheus_alerts.yaml with the alerts and a directory
dashboards_out with the Grafana dashboard JSON files:
$ make prometheus_alerts.yaml
$ make dashboards_out
For more advanced uses of mixins, see https://github.com/monitoring-mixins/docs.