From bc0bff22fdfd575617f57b48d6a7cbee818a34b8 Mon Sep 17 00:00:00 2001 From: Johannes Ziemke Date: Mon, 25 Jul 2022 16:26:50 +0200 Subject: [PATCH] Add sysctl docs to README.md Signed-off-by: Johannes Ziemke --- README.md | 103 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 77 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index 434d5ef9..d83a003a 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,6 @@ selinux | Exposes SELinux statistics. | Linux sockstat | Exposes various statistics from `/proc/net/sockstat`. | Linux softnet | Exposes statistics from `/proc/net/softnet_stat`. | Linux stat | Exposes various statistics from `/proc/stat`. This includes boot time, forks and interrupts. | Linux -sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux tapestats | Exposes statistics from `/sys/class/scsi_tape`. | Linux textfile | Exposes statistics read from local disk. The `--collector.textfile.directory` flag must be set. | _any_ thermal | Exposes thermal statistics like `pmset -g therm`. | Darwin @@ -156,6 +155,35 @@ and does not time out. In addition, monitor the `scrape_samples_post_metric_relabeling` metric to see the changes in cardinality. +Name | Description | OS +---------|-------------|---- +buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux +cgroups | A summary of the number of active and enabled cgroups | Linux +devstat | Exposes device statistics | Dragonfly, FreeBSD +drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux +ethtool | Exposes network interface information and network driver statistics equivalent to `ethtool`, `ethtool -S`, and `ethtool -i`. | Linux +interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD +ksmd | Exposes kernel and system statistics from `/sys/kernel/mm/ksm`. | Linux +lnstat | Exposes stats from `/proc/net/stat/`. | Linux +logind | Exposes session counts from [logind](http://www.freedesktop.org/wiki/Software/systemd/logind/). | Linux +meminfo\_numa | Exposes memory statistics from `/proc/meminfo_numa`. | Linux +mountstats | Exposes filesystem statistics from `/proc/self/mountstats`. Exposes detailed NFS client statistics. | Linux +network_route | Exposes the routing table as metrics | Linux +ntp | Exposes local NTP daemon health to check [time](./docs/TIME.md) | _any_ +perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux +processes | Exposes aggregate process statistics from `/proc`. | Linux +qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux +runit | Exposes service status from [runit](http://smarden.org/runit/). | _any_ +slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux +supervisord | Exposes service status from [supervisord](http://supervisord.org/). | _any_ +sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux +systemd | Exposes service and system status from [systemd](http://www.freedesktop.org/wiki/Software/systemd/). | Linux +tcpstat | Exposes TCP connection status information from `/proc/net/tcp` and `/proc/net/tcp6`. (Warning: the current version has potential performance issues in high load situations.) | Linux +wifi | Exposes WiFi device and station statistics. | Linux +zoneinfo | Exposes NUMA memory zone metrics. | Linux + +### Perf Collector + The `perf` collector may not work out of the box on some Linux systems due to kernel configuration and security settings. To allow access, set the following `sysctl` parameter: @@ -191,33 +219,56 @@ found using [`perf list`](http://man7.org/linux/man-pages/man1/perf.1.html) or from debugfs. And example usage of this would be `--collector.perf.tracepoint="sched:sched_process_exec"`. +### Sysctl Collector -Name | Description | OS ----------|-------------|---- -buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux -cgroups | A summary of the number of active and enabled cgroups | Linux -devstat | Exposes device statistics | Dragonfly, FreeBSD -drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux -ethtool | Exposes network interface information and network driver statistics equivalent to `ethtool`, `ethtool -S`, and `ethtool -i`. | Linux -interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD -ksmd | Exposes kernel and system statistics from `/sys/kernel/mm/ksm`. | Linux -lnstat | Exposes stats from `/proc/net/stat/`. | Linux -logind | Exposes session counts from [logind](http://www.freedesktop.org/wiki/Software/systemd/logind/). | Linux -meminfo\_numa | Exposes memory statistics from `/proc/meminfo_numa`. | Linux -mountstats | Exposes filesystem statistics from `/proc/self/mountstats`. Exposes detailed NFS client statistics. | Linux -network_route | Exposes the routing table as metrics | Linux -ntp | Exposes local NTP daemon health to check [time](./docs/TIME.md) | _any_ -perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux -processes | Exposes aggregate process statistics from `/proc`. | Linux -qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux -runit | Exposes service status from [runit](http://smarden.org/runit/). | _any_ -slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux -supervisord | Exposes service status from [supervisord](http://supervisord.org/). | _any_ -systemd | Exposes service and system status from [systemd](http://www.freedesktop.org/wiki/Software/systemd/). | Linux -tcpstat | Exposes TCP connection status information from `/proc/net/tcp` and `/proc/net/tcp6`. (Warning: the current version has potential performance issues in high load situations.) | Linux -wifi | Exposes WiFi device and station statistics. | Linux -zoneinfo | Exposes NUMA memory zone metrics. | Linux +The `sysctl` collector can be enabled with `--collector.sysctl`. It supports exposing numeric sysctl values +as metrics using the `--collector.sysctl.include` flag and string values as info metrics by using the +`--collector.sysctl.include-info` flag. The flags can be repeated. For sysctl with multiple numeric values, +an optional mapping can be given to expose each value as its own metric. Otherwise an `index` label is used +to identify the different fields. +#### Examples +##### Numeric values +###### Single values +Using `--collector.sysctl.include=vm.user_reserve_kbytes`: +`vm.user_reserve_kbytes = 131072` -> `node_sysctl_vm_user_reserve_kbytes 131072` + +###### Multiple values +A sysctl can contain multiple values, for example: +``` +net.ipv4.tcp_rmem = 4096 131072 6291456 +``` +Using `--collector.sysctl.include=net.ipv4.tcp_rmem` the collector will expose: +``` +node_sysctl_net_ipv4_tcp_rmem{index="0"} 4096 +node_sysctl_net_ipv4_tcp_rmem{index="1"} 131072 +node_sysctl_net_ipv4_tcp_rmem{index="2"} 6291456 +``` +If the indexes have defined meaning like in this case, the values can be mapped to multiple metrics by appending the mapping to the --collector.sysctl.include flag: +Using `--collector.sysctl.include=net.ipv4.tcp_rmem:min,default,max` the collector will expose: +``` +node_sysctl_net_ipv4_tcp_rmem_min 4096 +node_sysctl_net_ipv4_tcp_rmem_default 131072 +node_sysctl_net_ipv4_tcp_rmem_max 6291456 +``` + +##### String values +String values need to be exposed as info metric. The user selects them by using the `--collector.sysctl.include-info` flag. + +###### Single values +`kernel.core_pattern = core` -> `node_sysctl_info{key="kernel.core_pattern_info", value="core"} 1` + +###### Multiple values +Given the following sysctl: +``` +kernel.seccomp.actions_avail = kill_process kill_thread trap errno trace log allow +``` +Setting `--collector.sysctl.include-info=kernel.seccomp.actions_avail` will yield: +``` +node_sysctl_info{key="kernel.seccomp.actions_avail", index="0", value="kill_process"} 1 +node_sysctl_info{key="kernel.seccomp.actions_avail", index="1", value="kill_thread"} 1 +... +``` ### Textfile Collector