prometheus/docs/migration.md

# Prometheus 2.0 migration guide

In line with our [stability promise](https://prometheus.io/blog/2016/07/18/prometheus-1-0-released/#fine-print),
the Prometheus 2.0 release contains a number of backwards incompatible changes.
This document offers guidance on migrating from Prometheus 1.8 to Prometheus 2.0.

## Flags

The format of the Prometheus command line flags have changed.  Instead of a
single dash, all flags now use a double dash. Common flags (`--config.file`,
`--web.listen-address` and `--web.external-url`) are still the same but beyond
that, almost all the storage-related flags have been removed.

Some notable flags which have been removed:
- `-alertmanager.url` In Prometheus 2.0, the command line flags for configuring
  a static Alertmanager URL have been removed.  Alertmanager must now be
  discovered via service discovery, see [Alertmanager service discovery](#amsd).

- `-log.format` In Prometheus 2.0 logs can only be streamed to standard error.

- `-query.staleness-delta` has been renamed to `--query.lookback-delta`; Prometheus
  2.0 introduces a new mechanism for handling staleness, see [staleness](querying/basics.md#staleness).

- `-storage.local.*` Prometheus 2.0 introduces a new storage engine, as such all
  flags relating to the old engine have been removed.  For information on the
  new engine, see [Storage](#storage).

- `-storage.remote.*` Prometheus 2.0 has removed the already deprecated remote
  storage flags, and will fail to start if they are supplied.  To write to
  InfluxDB, Graphite, or OpenTSDB use the relevant storage adapter.

## Alertmanager service discovery

Alertmanager service discovery was introduced in Prometheus 1.4, allowing Prometheus
to dynamically discover Alertmanager replicas using the same mechanism as scrape
targets.  In Prometheus 2.0, the command line flags for static Alertmanager config
have been removed, so the following command line flag:

```
./prometheus -alertmanager.url=http://alertmanager:9093/
```

Would be replaced with the following in the `prometheus.yml` config file:

```yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager:9093
```

You can also use all the usual Prothetheus service discovery integrations and
relabeling in your Alertmanager configuration.  This snippet instructs
Prometheus to search for Kubernetes pods, in the `default` namespace, with the
label `name: alertmanager` and with a non-empty port.

```yml
alerting:
  alertmanagers:
  - kubernetes_sd_configs:
      - role: pod
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_name]
      regex: alertmanager
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      regex: default
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      regex:
      action: drop
```

## Recording rules and alerts

The format for configuring alerting and recording rules has been changed to YAML.
An example of a recording rule and alert in the old format:

```
job:request_duration_seconds:histogram_quantile99 =
  histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m])) by (le, job))

ALERT FrontendRequestLatency
  IF job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1
  FOR 5m
  ANNOTATIONS {
    summary = "High frontend request latency",
  }
```

Would look like this:

```yml
groups:
- name: example.rules
  rules:
  - record: job:request_duration_seconds:99percentile
    expr: histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m]))
      BY (le, job))
  - alert: FrontendRequestLatency
    expr: job:request_duration_seconds:99percentile{job="frontend"} > 0.1
    for: 5m
    annotations:
      summary: High frontend request latency
```

To help with the change, the `promtool` tool has a mode to automate the rules conversion.  Given a `.rules` file, it will output a `.rules.yml` file in the
new format. For example:

```
$ promtool update rules example.rules
```

## Storage

The data format in Prometheus 2.0 has completely changed and is not backwards
compatible with 1.8. To retain access to your historic monitoring data we recommend
you run a non-scraping Prometheus 1.8.1 instance in parallel with your Prometheus 2.0
instance, and have the new server read existing data from the old one via the
remote write protocol.

Your Prometheus 1.8 instance should be started with the following flags and an
empty config file (`empty.yml`):

```
$ ./prometheus-1.8.1.linux-amd64/prometheus -web.listen-address ":9094" -config.file empty.yml
```

NOTE: **NOTE** If you used external labels in your Prometheus 2.0 config, they need to be
preserved in your Prometheus 1.8 config.
Prometheus 2.0 can then be started (on the same machine) with the following flags:

```
$ ./prometheus-2.0.0.linux-amd64/prometheus --config.file prometheus.yml
```

Where `prometheus.yml` contains the stanza:

```
remote_read:
  - url: "http://localhost:9094/api/v1/read"
```

## PromQL

The follow features have been removed from PromQL:

- `drop_common_labels` function - the `without` aggregation modifier should be used
  instead.
- `keep_common` aggregation modifier - the `by` modifier should be used instead.
- `count_scalar` function - use cases are better handled by `absent()` or correct
  propagation of labels in operations.

See [issue #3060](https://github.com/prometheus/prometheus/issues/3060) for more
details.

## Miscellaneous

### Prometheus non-root user

The Prometheus Docker image is now built to [run Prometheus
as a non-root user](https://github.com/prometheus/prometheus/pull/2859).  If you
want the Prometheus UI/API to listen on a low port number (say, port 80), you'll
need to override it.  For Kubernetes, you would use the following YAML:

```yml
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo-2
spec:
  securityContext:
    runAsUser: 0
...
```

See [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](Configure a Security Context for a Pod or Container)
for more details.

If you're using Docker, then the follow snippet would be used:

```
docker run -u root -p 80:80 prom/prometheus:v2.0.0-rc.2  --web.listen-address :80
```

## Prometheus lifecycle

If you use the Prometheus `/-/reload` HTTP endpoint to [automatically reload your
Prometheus config when it changes](configuration/configuration.md),
these endpoints are disabled by default for security reasons in Prometheus 2.0.
To enable them, set the `--web.enable-lifecycle` flag.