196 lines
6.5 KiB
Markdown
196 lines
6.5 KiB
Markdown
# Prometheus 2.0 migration guide
|
|
|
|
In line with our [stability promise](https://prometheus.io/blog/2016/07/18/prometheus-1-0-released/#fine-print),
|
|
the Prometheus 2.0 release contains a number of backwards incompatible changes.
|
|
This document offers guidance on migrating from Prometheus 1.8 to Prometheus 2.0.
|
|
|
|
## Flags
|
|
|
|
The format of the Prometheus command line flags have changed. Instead of a
|
|
single dash, all flags now use a double dash. Common flags (`--config.file`,
|
|
`--web.listen-address` and `--web.external-url`) are still the same but beyond
|
|
that, almost all the storage-related flags have been removed.
|
|
|
|
Some notable flags which have been removed:
|
|
- `-alertmanager.url` In Prometheus 2.0, the command line flags for configuring
|
|
a static Alertmanager URL have been removed. Alertmanager must now be
|
|
discovered via service discovery, see [Alertmanager service discovery](#amsd).
|
|
|
|
- `-log.format` In Prometheus 2.0 logs can only be streamed to standard error.
|
|
|
|
- `-query.staleness-delta` has been renamed to `--query.lookback-delta`; Prometheus
|
|
2.0 introduces a new mechanism for handling staleness, see [staleness](querying/basics.md#staleness).
|
|
|
|
- `-storage.local.*` Prometheus 2.0 introduces a new storage engine, as such all
|
|
flags relating to the old engine have been removed. For information on the
|
|
new engine, see [Storage](#storage).
|
|
|
|
- `-storage.remote.*` Prometheus 2.0 has removed the already deprecated remote
|
|
storage flags, and will fail to start if they are supplied. To write to
|
|
InfluxDB, Graphite, or OpenTSDB use the relevant storage adapter.
|
|
|
|
## Alertmanager service discovery
|
|
|
|
Alertmanager service discovery was introduced in Prometheus 1.4, allowing Prometheus
|
|
to dynamically discover Alertmanager replicas using the same mechanism as scrape
|
|
targets. In Prometheus 2.0, the command line flags for static Alertmanager config
|
|
have been removed, so the following command line flag:
|
|
|
|
```
|
|
./prometheus -alertmanager.url=http://alertmanager:9093/
|
|
```
|
|
|
|
Would be replaced with the following in the `prometheus.yml` config file:
|
|
|
|
```yml
|
|
alerting:
|
|
alertmanagers:
|
|
- static_configs:
|
|
- targets:
|
|
- alertmanager:9093
|
|
```
|
|
|
|
You can also use all the usual Prothetheus service discovery integrations and
|
|
relabeling in your Alertmanager configuration. This snippet instructs
|
|
Prometheus to search for Kubernetes pods, in the `default` namespace, with the
|
|
label `name: alertmanager` and with a non-empty port.
|
|
|
|
```yml
|
|
alerting:
|
|
alertmanagers:
|
|
- kubernetes_sd_configs:
|
|
- role: pod
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_pod_label_name]
|
|
regex: alertmanager
|
|
action: keep
|
|
- source_labels: [__meta_kubernetes_namespace]
|
|
regex: default
|
|
action: keep
|
|
- source_labels: [__meta_kubernetes_pod_container_port_number]
|
|
regex:
|
|
action: drop
|
|
```
|
|
|
|
## Recording rules and alerts
|
|
|
|
The format for configuring alerting and recording rules has been changed to YAML.
|
|
An example of a recording rule and alert in the old format:
|
|
|
|
```
|
|
job:request_duration_seconds:histogram_quantile99 =
|
|
histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m])) by (le, job))
|
|
|
|
ALERT FrontendRequestLatency
|
|
IF job:request_duration_seconds:histogram_quantile99{job="frontend"} > 0.1
|
|
FOR 5m
|
|
ANNOTATIONS {
|
|
summary = "High frontend request latency",
|
|
}
|
|
```
|
|
|
|
Would look like this:
|
|
|
|
```yml
|
|
groups:
|
|
- name: example.rules
|
|
rules:
|
|
- record: job:request_duration_seconds:99percentile
|
|
expr: histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[1m]))
|
|
BY (le, job))
|
|
- alert: FrontendRequestLatency
|
|
expr: job:request_duration_seconds:99percentile{job="frontend"} > 0.1
|
|
for: 5m
|
|
annotations:
|
|
summary: High frontend request latency
|
|
```
|
|
|
|
To help with the change, the `promtool` tool has a mode to automate the rules conversion. Given a `.rules` file, it will output a `.rules.yml` file in the
|
|
new format. For example:
|
|
|
|
```
|
|
$ promtool update rules example.rules
|
|
```
|
|
|
|
## Storage
|
|
|
|
The data format in Prometheus 2.0 has completely changed and is not backwards
|
|
compatible with 1.8. To retain access to your historic monitoring data we recommend
|
|
you run a non-scraping Prometheus 1.8.1 instance in parallel with your Prometheus 2.0
|
|
instance, and have the new server read existing data from the old one via the
|
|
remote write protocol.
|
|
|
|
Your Prometheus 1.8 instance should be started with the following flags and an
|
|
empty config file (`empty.yml`):
|
|
|
|
```
|
|
$ ./prometheus-1.8.1.linux-amd64/prometheus -web.listen-address ":9094" -config.file empty.yml
|
|
```
|
|
|
|
NOTE: **NOTE** If you used external labels in your Prometheus 2.0 config, they need to be
|
|
preserved in your Prometheus 1.8 config.
|
|
Prometheus 2.0 can then be started (on the same machine) with the following flags:
|
|
|
|
```
|
|
$ ./prometheus-2.0.0.linux-amd64/prometheus --config.file prometheus.yml
|
|
```
|
|
|
|
Where `prometheus.yml` contains the stanza:
|
|
|
|
```
|
|
remote_read:
|
|
- url: "http://localhost:9094/api/v1/read"
|
|
```
|
|
|
|
## PromQL
|
|
|
|
The follow features have been removed from PromQL:
|
|
|
|
- `drop_common_labels` function - the `without` aggregation modifier should be used
|
|
instead.
|
|
- `keep_common` aggregation modifier - the `by` modifier should be used instead.
|
|
- `count_scalar` function - use cases are better handled by `absent()` or correct
|
|
propagation of labels in operations.
|
|
|
|
See [issue #3060](https://github.com/prometheus/prometheus/issues/3060) for more
|
|
details.
|
|
|
|
## Miscellaneous
|
|
|
|
### Prometheus non-root user
|
|
|
|
The Prometheus Docker image is now built to [run Prometheus
|
|
as a non-root user](https://github.com/prometheus/prometheus/pull/2859). If you
|
|
want the Prometheus UI/API to listen on a low port number (say, port 80), you'll
|
|
need to override it. For Kubernetes, you would use the following YAML:
|
|
|
|
```yml
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: security-context-demo-2
|
|
spec:
|
|
securityContext:
|
|
runAsUser: 0
|
|
...
|
|
```
|
|
|
|
See [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](Configure a Security Context for a Pod or Container)
|
|
for more details.
|
|
|
|
If you're using Docker, then the follow snippet would be used:
|
|
|
|
```
|
|
docker run -u root -p 80:80 prom/prometheus:v2.0.0-rc.2 --web.listen-address :80
|
|
```
|
|
|
|
## Prometheus lifecycle
|
|
|
|
If you use the Prometheus `/-/reload` HTTP endpoint to [automatically reload your
|
|
Prometheus config when it changes](configuration/configuration.md),
|
|
these endpoints are disabled by default for security reasons in Prometheus 2.0.
|
|
To enable them, set the `--web.enable-lifecycle` flag.
|