d4b9fe801f
When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. **Prombench results** _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> |
||
|---|---|---|
| .. | ||
| chunkenc | ||
| chunks | ||
| cmd/tsdb | ||
| docs/format | ||
| encoding | ||
| errors | ||
| fileutil | ||
| goversion | ||
| index | ||
| record | ||
| test | ||
| testdata | ||
| tombstones | ||
| tsdbutil | ||
| wal | ||
| .gitignore | ||
| CHANGELOG.md | ||
| README.md | ||
| block.go | ||
| block_test.go | ||
| compact.go | ||
| compact_test.go | ||
| db.go | ||
| db_test.go | ||
| head.go | ||
| head_bench_test.go | ||
| head_test.go | ||
| isolation.go | ||
| mocks_test.go | ||
| querier.go | ||
| querier_bench_test.go | ||
| querier_test.go | ||
| repair.go | ||
| repair_test.go | ||
| tsdbblockutil.go | ||
| wal.go | ||
| wal_test.go | ||
README.md
TSDB
This repository contains the Prometheus storage layer that is used in its 2.x releases.
A writeup of its design can be found here.
Based on the Gorilla TSDB white papers.
Video: Storing 16 Bytes at Scale from PromCon 2017.
See also the format documentation.