277f04f0c4
* Stop compactions if there's a block to write db.Compact() checks if there's a block to write with HEAD chunks before calling db.compactBlocks(). This is to ensure that if we need to write a block then it happens ASAP, otherwise memory usage might keep growing. But what can also happen is that we don't need to write any block, we start db.compactBlocks(), compaction takes hours, and in the meantime HEAD needs to write out chunks to a block. This can be especially problematic if, for example, you run Thanos sidecar that's uploading block, which requires that compactions are disabled. Then you disable Thanos sidecar and re-enable compactions. When db.compactBlocks() is finally called it might have a huge number of blocks to compact, which might take a very long time, during which HEAD cannot write out chunks to a new block. In such case memory usage will keep growing until either: - compactions are finally finished and HEAD can write a block - we run out of memory and Prometheus gets OOM-killed This change adds a check for pending HEAD block writes inside db.compactBlocks(), so that we bail out early if there are still compactions to run, but we also need to write a new block. Also add a test for compactBlocks. --------- Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com> Signed-off-by: Lukasz Mierzwa <lukasz@cloudflare.com> |
||
|---|---|---|
| .. | ||
| agent | ||
| chunkenc | ||
| chunks | ||
| docs | ||
| encoding | ||
| errors | ||
| fileutil | ||
| goversion | ||
| index | ||
| record | ||
| testdata | ||
| tombstones | ||
| tsdbutil | ||
| wlog | ||
| .gitignore | ||
| CHANGELOG.md | ||
| README.md | ||
| block.go | ||
| block_test.go | ||
| blockwriter.go | ||
| blockwriter_test.go | ||
| compact.go | ||
| compact_test.go | ||
| db.go | ||
| db_test.go | ||
| example_test.go | ||
| exemplar.go | ||
| exemplar_test.go | ||
| head.go | ||
| head_append.go | ||
| head_bench_test.go | ||
| head_read.go | ||
| head_read_test.go | ||
| head_test.go | ||
| head_wal.go | ||
| isolation.go | ||
| isolation_test.go | ||
| mocks_test.go | ||
| ooo_head.go | ||
| ooo_head_read.go | ||
| ooo_head_read_test.go | ||
| ooo_head_test.go | ||
| ooo_isolation.go | ||
| ooo_isolation_test.go | ||
| querier.go | ||
| querier_bench_test.go | ||
| querier_test.go | ||
| repair.go | ||
| repair_test.go | ||
| tsdbblockutil.go | ||
README.md
TSDB
This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.
Documentation
External resources
- A writeup of the original design can be found here.
- Video: Storing 16 Bytes at Scale from PromCon 2017.
- Compression is based on the Gorilla TSDB white paper.
A series of blog posts explaining different components of TSDB: