Alertmanager

Prometheus Alertmanager

WARNING: The Alertmanager is still very experimental and early in its development and design. More than any other Prometheus component, it will still undergo frequent breaking changes, including ones that will affect its architecture as a whole. While we do plan on making it mature and stable eventually, use it at your own risk for now.

Alertmanager receives alerts generated by Prometheus and takes care of the following aspects:

• manual silencing of specific alerts
• inhibiting alerts based on alert dependencies
• aggregating alerts by labelset
• handling notification repeats
• sending alert notifications via external services (currently email, generic web hook, PagerDuty, HipChat, Slack, Pushover, or Flowdock)

See config/fixtures/sample.conf.input for an example config. The full configuration schema including a documentation for all possible options can be found in config/config.proto. Alertmanager automatically reloads the configuration when it changes, so restarts are not required for configuration updates.

Building and running

make
./alertmanager -config.file=/path/to/alertmanager.conf

Configuring Prometheus to send alerts

To make Prometheus send alerts to your Alertmanager, set the alertmanager.url command-line flag on the server:

./prometheus -alertmanager.url=http://<alertmanager-host>:<port> <...other flags...>

Prometheus only pushes firing alerts to Alertmanager. Alertmanager expects to receive regular pushes of firing alerts from Prometheus. Alerts which are not refreshed for a period of -alerts.min-refresh-period (5 minutes by default) are expired.

Alertmanager only shows alerts which are currently firing and pushed to Alertmanager.

Running tests

make test

Caveats and roadmap

Alertmanager is still in an experimental state. Some of the known caveats which are going to be addressed in the future:

• Alertmanager is run as a single instance and does not provide high availability yet. We plan on clustering multiple replicated Alertmanager instances to ensure reliability in the future.
• Relatedly, silence information is currently only persisted locally in a file and lost if you lose the machine your Alertmanager is running on.
• Alert aggregation needs to become more flexible. Currently alerts are aggregated based on their full labelsets. In the future, we want to allow grouping alerts based on a subset thereof (for example, grouping all alerts with one alert name and from the same job together).
• For alert dependencies, we want to support time delays: if alert A inhibits alert B due to a dependency and B begins firing before A, wait for a configurable amount of time for A to start firing as well before sending notifications for B. This is not yet supported.
• Alertmanager has not been tested or optimized for high alert loads yet.