Alertmanager

This is the development version of the Alertmanager. It is a rewrite and is incompatible to the present version 0.0.4. The only backport was the API endpoint used by Prometheus to push new alerts.

Installation

The current version has to be run from the repository folder as UI assets and notification templates are not yet statically compiled into the binary.

You can either go get it:

$ GO15VENDOREXPERIMENT=1 go get github.com/prometheus/alertmanager
# cd $GOPATH/src/github.com/prometheus/alertmanager
$ alertmanager -config.file=<your_file>

Or checkout the source code and build manually:

$ mkdir -p $GOPATH/src/github.com/prometheus
$ cd $GOPATH/src/github.com/prometheus
$ git clone https://github.com/prometheus/alertmanager.git
$ cd alertmanager
$ make
$ ./alertmanager -config.file=<your_file>

Status

This version was written from scratch. Core features enabled by this is are more advanced alert routing configurations and grouping/batching of alerts. Thus, squashing expression results through aggregation in alerting rules is no longer required to avoid noisyness.

The concepts of alert routing were outlined in this document.

The version implements full persistence of alerts, silences, and notification state. On restart it picks up right where it left off.

Known issues

This development version still has an extensive list of improvements and changes. This is an incomplete list of things that are still missing or need to be improved.

This will happen based on priority and demand. Feel free to ping fabxc about it

• On deleting silences it may take up to one group_wait cycle for a notification of a previously silenced alert to be sent.
• Limiting inhibition rules to routing subtrees to avoid accidental interference
• Definition of a minimum data set provided to notification templates
• Best practices around notification templating
• Various common command line flags like path-prefix
• Compiling templates and UI assets into the binary

Example

This is an example configuration that should cover most relevant aspects of the new YAML configuration format. Authoritative source for now is the code.

global:
  # The smarthost and SMTP sender used for mail notifications.
  smarthost: 'localhost:25'
  smtp_sender: 'alertmanager@example.org'

# The directory from which notification templates are read.
templates: 
- 'template/*.tmpl'

# The root route on which each incoming alert enters.
route:
  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  group_by: ['alertname', 'cluster']
  
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first 
  # notification.
  group_wait: 30s

  # When the first notification was sent, wait 'group_interval' to send a betch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 3h 

  # If 'continue' is false, the first sub-route that matches this alert will
  # terminate the search and the alert will be inserted at that routing node.
  # If true, the alert is inserted to sibling nodes as well if there is a
  # match.
  # This allows to do first-match semantics (=false) in smaller scopes (e.g. team-level),
  # while avoiding accidental shadowing (=true) at alerts at larger scopes (e.g. company-level)
  continue: true

  # All the above attributes are inherited by all child routes and can 
  # overwritten on each.

  # The child route trees.
  routes:
  # This routes performs a regular expression match on alert labels to
  # catch alerts that are related to a list of services.
  - match_re:
      service: ^(foo1|foo2|baz)$
    send_to: team-X-mails

    # The service has a sub-route for critical alerts, any alerts
    # that do not match, i.e. severity != critical, fall-back to the
    # parent node and are sent to 'team-X-mails'
    routes:
    - match:
        severity: critical
      send_to: team-X-pager

  - match:
      service: files
    send_to: team-Y-mails

    routes:
    - match:
        severity: critical
      send_to: team-Y-pager

  # This route handles all alerts coming from a database service. If there's
  # no team to handle it, it defaults to the DB team.
  - match:
      service: database

    send_to: team-DB-pager
    # Also group alerts by affected database.
    group_by: [alertname, cluster, database]
    continue: false

    routes:
    - match:
        owner: team-X
      send_to: team-X-pager

    - match:
        owner: team-Y
      send_to: team-Y-pager
        

# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is 
# already critical.
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  # Apply inhibition if the alertname is the same.
  equal: ['alertname']


notification_configs:
- name: 'team-X-mails'
  email_configs:
  - email: 'team-X+alerts@example.org'

- name: 'team-X-pager'
  email_configs:
  - email: 'team-X+alerts-critical@example.org'
  pagerduty_configs:
  - service_key: <team-X-key>

- name: 'team-Y-mails'
  email_configs:
  - email: 'team-Y+alerts@example.org'

- name: 'team-Y-pager'
  pagerduty_configs:
  - service_key: <team-Y-key>

- name: 'team-DB-pager'
  pagerduty_configs:
  - service_key: <team-DB-key>

Testing

If you want to test the new Alertmanager while running the current version, you can mirror traffic to the new one with a simple nginx configuration similar to this:

server {
  server_name <your_current_alertmanager>;
  location / {
    proxy_pass  http://localhost:9093;
    post_action @forward;
  }
  location @forward {
    proxy_pass http://<your_new_alertmanager>:9093; 
  }
}

Architecture