mirror of
https://github.com/ceph/ceph
synced 2024-12-22 11:31:55 +00:00
98236e3a1d
Mixin is a way to bundle dashboards, prometheus rules and alerts into jsonnet package. Shifting to mixin will allow easier integration with monitoring automation that some users may use. This commit moves `/monitoring/grafana/dashboards` and `/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts was also converted to Jsonnet using an automated way (from yaml to json to jsonnet). This commit minimises any change made to the generated files and should not change neithers the dashboards nor the Prometheus alerts. In the future some configuration will also be added to jsonnet to add more functionalities to the dashboards or alerts (i.e.: multi cluster). Fixes: https://tracker.ceph.com/issues/53374 Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
5.3 KiB
5.3 KiB
Alert Rule Standards
The alert rules should adhere to the following principles
- each alert must have a unique name
- each alert should define a common structure
- labels : must contain severity and type
- annotations : must provide description
- expr : must define the promql expression
- alert : defines the alert name
- alerts that have a corresponding section within docs.ceph.com must include a documentation field in the annotations section
- critical alerts should declare an oid in the labels section
- critical alerts should have a corresponding entry in the Ceph MIB
Testing Prometheus Rules
Once you have updated the ceph_default_alerts.yml file, you should use the
validate_rules.py script directly, or via tox to ensure the format of any update
or change aligns to our rule structure guidelines. The validate_rules.py script will
process the rules and look for any configuration anomalies and output a report if
problems are detected.
Here's an example run, to illustrate the format and the kinds of issues detected.
[paul@myhost tests]$ ./validate_rules.py
Checking rule groups
cluster health : ..
mon : E.W..
osd : E...W......W.E..
mds : WW
mgr : WW
pgs : ..WWWW..
nodes : .EEEE
pools : EEEW.
healthchecks : .
cephadm : WW.
prometheus : W
rados : W
Summary
Rule file : ../alerts/ceph_default_alerts.yml
Unit Test file : test_alerts.yml
Rule groups processed : 12
Rules processed : 51
Rule errors : 10
Rule warnings : 16
Rule name duplicates : 0
Unit tests missing : 4
Problem Report
Group Severity Alert Name Problem Description
----- -------- ---------- -------------------
cephadm Warning Cluster upgrade has failed critical level alert is missing an SNMP oid entry
cephadm Warning A daemon managed by cephadm is down critical level alert is missing an SNMP oid entry
mds Warning Ceph Filesystem damage detected critical level alert is missing an SNMP oid entry
mds Warning Ceph Filesystem switched to READ ONLY critical level alert is missing an SNMP oid entry
mgr Warning mgr module failure critical level alert is missing an SNMP oid entry
mgr Warning mgr prometheus module is not active critical level alert is missing an SNMP oid entry
mon Error Monitor down, quorum is at risk documentation link error: #mon-downwah not found on the page
mon Warning Ceph mon disk space critically low critical level alert is missing an SNMP oid entry
nodes Error network packets dropped invalid alert structure. Missing field: for
nodes Error network packet errors invalid alert structure. Missing field: for
nodes Error storage filling up invalid alert structure. Missing field: for
nodes Error MTU Mismatch invalid alert structure. Missing field: for
osd Error 10% OSDs down invalid alert structure. Missing field: for
osd Error Flapping OSD invalid alert structure. Missing field: for
osd Warning OSD Full critical level alert is missing an SNMP oid entry
osd Warning Too many devices predicted to fail critical level alert is missing an SNMP oid entry
pgs Warning Placement Group (PG) damaged critical level alert is missing an SNMP oid entry
pgs Warning Recovery at risk, cluster too full critical level alert is missing an SNMP oid entry
pgs Warning I/O blocked to some data critical level alert is missing an SNMP oid entry
pgs Warning Cluster too full, automatic data recovery impaired critical level alert is missing an SNMP oid entry
pools Error pool full invalid alert structure. Missing field: for
pools Error pool filling up (growth forecast) invalid alert structure. Missing field: for
pools Error Ceph pool is too full for recovery/rebalance invalid alert structure. Missing field: for
pools Warning Ceph pool is full - writes blocked critical level alert is missing an SNMP oid entry
prometheus Warning Scrape job is missing critical level alert is missing an SNMP oid entry
rados Warning Data not found/missing critical level alert is missing an SNMP oid entry
Unit tests are incomplete. Tests missing for the following alerts;
- Placement Group (PG) damaged
- OSD Full
- storage filling up
- pool filling up (growth forecast)