6f69e31398
This change switches the remote_write API to use the TSDB WAL. This should reduce memory usage and prevent sample loss when the remote end point is down. We use the new LiveReader from TSDB to tail WAL segments. Logic for finding the tracking segment is included in this PR. The WAL is tailed once for each remote_write endpoint specified. Reading from the segment is based on a ticker rather than relying on fsnotify write events, which were found to be complicated and unreliable in early prototypes. Enqueuing a sample for sending via remote_write can now block, to provide back pressure. Queues are still required to acheive parallelism and batching. We have updated the queue config based on new defaults for queue capacity and pending samples values - much smaller values are now possible. The remote_write resharding code has been updated to prevent deadlocks, and extra tests have been added for these cases. As part of this change, we attempt to guarantee that samples are not lost; however this initial version doesn't guarantee this across Prometheus restarts or non-retryable errors from the remote end (eg 400s). This changes also includes the following optimisations: - only marshal the proto request once, not once per retry - maintain a single copy of the labels for given series to reduce GC pressure Other minor tweaks: - only reshard if we've also successfully sent recently - add pending samples, latest sent timestamp, WAL events processed metrics Co-authored-by: Chris Marchbanks <csmarchbanks.com> (initial prototype) Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> (sharding changes) Signed-off-by: Callum Styan <callumstyan@gmail.com> |
||
|---|---|---|
| .circleci | ||
| .github | ||
| cmd | ||
| config | ||
| console_libraries | ||
| consoles | ||
| discovery | ||
| docs | ||
| documentation | ||
| notifier | ||
| pkg | ||
| prompb | ||
| promql | ||
| relabel | ||
| rules | ||
| scrape | ||
| scripts | ||
| storage | ||
| template | ||
| util | ||
| vendor | ||
| web | ||
| .dockerignore | ||
| .gitignore | ||
| .promu.yml | ||
| .travis.yml | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| MAINTAINERS.md | ||
| Makefile | ||
| Makefile.common | ||
| NOTICE | ||
| README.md | ||
| RELEASE.md | ||
| VERSION | ||
| code-of-conduct.md | ||
| go.mod | ||
| go.sum | ||
README.md
Prometheus
Visit prometheus.io for the full documentation, examples and guides.
Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.
Prometheus' main distinguishing features as compared to other monitoring systems are:
- a multi-dimensional data model (timeseries defined by metric name and set of key/value dimensions)
- a flexible query language to leverage this dimensionality
- no dependency on distributed storage; single server nodes are autonomous
- timeseries collection happens via a pull model over HTTP
- pushing timeseries is supported via an intermediary gateway
- targets are discovered via service discovery or static configuration
- multiple modes of graphing and dashboarding support
- support for hierarchical and horizontal federation
Architecture overview
Install
There are various ways of installing Prometheus.
Precompiled binaries
Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Prometheus. See the Installing chapter in the documentation for all the details.
Debian packages are available.
Docker images
Docker images are available on Quay.io or Docker Hub.
You can launch a Prometheus container for trying it out with
$ docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus
Prometheus will now be reachable at http://localhost:9090/.
Building from source
To build Prometheus from the source code yourself you need to have a working Go environment with version 1.11 or greater installed.
You can directly use the go tool to download and install the prometheus
and promtool binaries into your GOPATH:
$ go get github.com/prometheus/prometheus/cmd/...
$ prometheus --config.file=your_config.yml
You can also clone the repository yourself and build using make:
$ mkdir -p $GOPATH/src/github.com/prometheus
$ cd $GOPATH/src/github.com/prometheus
$ git clone https://github.com/prometheus/prometheus.git
$ cd prometheus
$ make build
$ ./prometheus --config.file=your_config.yml
The Makefile provides several targets:
- build: build the
prometheusandpromtoolbinaries - test: run the tests
- test-short: run the short tests
- format: format the source code
- vet: check the source code for common errors
- assets: rebuild the static assets
- docker: build a docker container for the current
HEAD
More information
- The source code is periodically indexed: Prometheus Core.
- You will find a Travis CI configuration in
.travis.yml. - See the Community page for how to reach the Prometheus developers and users on various communication channels.
Contributing
Refer to CONTRIBUTING.md
License
Apache License 2.0, see LICENSE.