mirror of
https://github.com/ceph/ceph
synced 2024-12-22 11:31:55 +00:00
98236e3a1d
Mixin is a way to bundle dashboards, prometheus rules and alerts into jsonnet package. Shifting to mixin will allow easier integration with monitoring automation that some users may use. This commit moves `/monitoring/grafana/dashboards` and `/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts was also converted to Jsonnet using an automated way (from yaml to json to jsonnet). This commit minimises any change made to the generated files and should not change neithers the dashboards nor the Prometheus alerts. In the future some configuration will also be added to jsonnet to add more functionalities to the dashboards or alerts (i.e.: multi cluster). Fixes: https://tracker.ceph.com/issues/53374 Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
93 lines
5.3 KiB
Markdown
93 lines
5.3 KiB
Markdown
|
|
## Alert Rule Standards
|
|
|
|
The alert rules should adhere to the following principles
|
|
- each alert must have a unique name
|
|
- each alert should define a common structure
|
|
- labels : must contain severity and type
|
|
- annotations : must provide description
|
|
- expr : must define the promql expression
|
|
- alert : defines the alert name
|
|
- alerts that have a corresponding section within docs.ceph.com must include a
|
|
documentation field in the annotations section
|
|
- critical alerts should declare an oid in the labels section
|
|
- critical alerts should have a corresponding entry in the Ceph MIB
|
|
|
|
|
|
## Testing Prometheus Rules
|
|
Once you have updated the `ceph_default_alerts.yml` file, you should use the
|
|
`validate_rules.py` script directly, or via `tox` to ensure the format of any update
|
|
or change aligns to our rule structure guidelines. The validate_rules.py script will
|
|
process the rules and look for any configuration anomalies and output a report if
|
|
problems are detected.
|
|
|
|
Here's an example run, to illustrate the format and the kinds of issues detected.
|
|
|
|
```
|
|
[paul@myhost tests]$ ./validate_rules.py
|
|
|
|
Checking rule groups
|
|
cluster health : ..
|
|
mon : E.W..
|
|
osd : E...W......W.E..
|
|
mds : WW
|
|
mgr : WW
|
|
pgs : ..WWWW..
|
|
nodes : .EEEE
|
|
pools : EEEW.
|
|
healthchecks : .
|
|
cephadm : WW.
|
|
prometheus : W
|
|
rados : W
|
|
|
|
Summary
|
|
|
|
Rule file : ../alerts/ceph_default_alerts.yml
|
|
Unit Test file : test_alerts.yml
|
|
|
|
Rule groups processed : 12
|
|
Rules processed : 51
|
|
Rule errors : 10
|
|
Rule warnings : 16
|
|
Rule name duplicates : 0
|
|
Unit tests missing : 4
|
|
|
|
Problem Report
|
|
|
|
Group Severity Alert Name Problem Description
|
|
----- -------- ---------- -------------------
|
|
cephadm Warning Cluster upgrade has failed critical level alert is missing an SNMP oid entry
|
|
cephadm Warning A daemon managed by cephadm is down critical level alert is missing an SNMP oid entry
|
|
mds Warning Ceph Filesystem damage detected critical level alert is missing an SNMP oid entry
|
|
mds Warning Ceph Filesystem switched to READ ONLY critical level alert is missing an SNMP oid entry
|
|
mgr Warning mgr module failure critical level alert is missing an SNMP oid entry
|
|
mgr Warning mgr prometheus module is not active critical level alert is missing an SNMP oid entry
|
|
mon Error Monitor down, quorum is at risk documentation link error: #mon-downwah not found on the page
|
|
mon Warning Ceph mon disk space critically low critical level alert is missing an SNMP oid entry
|
|
nodes Error network packets dropped invalid alert structure. Missing field: for
|
|
nodes Error network packet errors invalid alert structure. Missing field: for
|
|
nodes Error storage filling up invalid alert structure. Missing field: for
|
|
nodes Error MTU Mismatch invalid alert structure. Missing field: for
|
|
osd Error 10% OSDs down invalid alert structure. Missing field: for
|
|
osd Error Flapping OSD invalid alert structure. Missing field: for
|
|
osd Warning OSD Full critical level alert is missing an SNMP oid entry
|
|
osd Warning Too many devices predicted to fail critical level alert is missing an SNMP oid entry
|
|
pgs Warning Placement Group (PG) damaged critical level alert is missing an SNMP oid entry
|
|
pgs Warning Recovery at risk, cluster too full critical level alert is missing an SNMP oid entry
|
|
pgs Warning I/O blocked to some data critical level alert is missing an SNMP oid entry
|
|
pgs Warning Cluster too full, automatic data recovery impaired critical level alert is missing an SNMP oid entry
|
|
pools Error pool full invalid alert structure. Missing field: for
|
|
pools Error pool filling up (growth forecast) invalid alert structure. Missing field: for
|
|
pools Error Ceph pool is too full for recovery/rebalance invalid alert structure. Missing field: for
|
|
pools Warning Ceph pool is full - writes blocked critical level alert is missing an SNMP oid entry
|
|
prometheus Warning Scrape job is missing critical level alert is missing an SNMP oid entry
|
|
rados Warning Data not found/missing critical level alert is missing an SNMP oid entry
|
|
|
|
Unit tests are incomplete. Tests missing for the following alerts;
|
|
- Placement Group (PG) damaged
|
|
- OSD Full
|
|
- storage filling up
|
|
- pool filling up (growth forecast)
|
|
|
|
```
|