prometheus/tsdb
Justin Lei 8ef7dfdeeb
Add a chunk size limit in bytes (#12054)
Add a chunk size limit in bytes

This creates a hard cap for XOR chunks of 1024 bytes.

The limit for histogram chunk is also 1024 bytes, but it is a soft limit as a histogram has a dynamic size, and even a single one could be larger than 1024 bytes.

This also avoids cutting new histogram chunks if the existing chunk has fewer than 10 histograms yet. In that way, we are accepting "jumbo chunks" in order to have at least 10 histograms in a chunk, allowing compression to kick in.

Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-08-24 15:21:17 +02:00
..
agent Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
chunkenc Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
chunks Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
docs Merge branch 'main' into sparsehistogram 2022-08-10 17:54:37 +02:00
encoding Merge 'main' into sparsehistogram 2022-10-05 22:14:49 +02:00
errors Fix context.Canceled wrapping in compaction 2023-03-23 11:10:00 +01:00
fileutil Update go to 1.19, set min version to 1.18 (#11279) 2022-09-07 11:30:48 +02:00
goversion lint: Adjust to the lint warnings raised by current versions of golint-ci 2023-04-19 17:10:10 +02:00
index Merge pull request #10623 from songjiayang/update-index 2023-07-18 13:47:27 +02:00
record tsdb: Support native histograms in snapshot on shutdown (#12258) 2023-07-05 11:44:13 +02:00
testdata
tombstones Copy tombstone intervals to avoid race (#12245) 2023-05-17 15:15:12 +02:00
tsdbutil Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
wlog Add Zstandard compression option for wlog (#11666) 2023-07-11 14:57:57 +02:00
.gitignore
CHANGELOG.md
README.md Fixed broken link in tsdb README.md 2022-10-07 16:20:20 +00:00
block.go tsdb: sort values for Postings only when required 2023-01-05 14:05:54 +00:00
block_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
blockwriter.go Merge 'main' into sparsehistogram 2022-10-05 22:14:49 +02:00
blockwriter_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
compact.go Replace sort.Slice with faster slices.SortFunc 2023-07-02 22:17:08 +00:00
compact_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
db.go Merge pull request #12664 from prometheus/superq/cleanup_chunk_snapshots 2023-08-08 13:02:39 +02:00
db_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
example_test.go storage: allow re-use of iterators 2022-12-15 18:32:45 +00:00
exemplar.go scrape: Enable ingestion of multiple exemplars per sample 2023-07-13 14:16:10 +02:00
exemplar_test.go tsdb: in tests use labels.FromStrings 2022-09-09 13:34:49 +02:00
head.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
head_append.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
head_bench_test.go Pass ref to SeriesLifecycleCallback.PostDeletion (#12626) 2023-08-03 10:56:27 +02:00
head_read.go Use a linked list for memSeries.headChunk (#11818) 2023-07-31 11:10:24 +02:00
head_read_test.go Use a linked list for memSeries.headChunk (#11818) 2023-07-31 11:10:24 +02:00
head_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
head_wal.go fix: error message typo 2023-08-17 16:34:45 +03:00
isolation.go ci(lint): enable predeclared linter 2023-05-21 07:33:54 +00:00
isolation_test.go tsdb: turn off transaction isolation for head compaction (#11317) 2022-09-27 19:31:23 +05:30
mocks_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
ooo_head.go Name float values as "floats", not as "values" 2023-04-13 19:25:24 +02:00
ooo_head_read.go remove repetitive words (#12556) 2023-07-13 15:53:40 +02:00
ooo_head_read_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
ooo_head_test.go Name float values as "floats", not as "values" 2023-04-13 19:25:24 +02:00
querier.go Hide histogram chunk append and reset header internals (#12352) 2023-07-26 15:08:16 +02:00
querier_bench_test.go Apply relevant label matchers in LabelValues before fetching extra postings (#12274) 2023-07-04 10:37:58 +01:00
querier_test.go Add a chunk size limit in bytes (#12054) 2023-08-24 15:21:17 +02:00
repair.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
repair_test.go Simplify IndexReader.Series interface 2022-12-19 15:22:09 +00:00
tsdbblockutil.go Support FloatHistogram in TSDB (#11522) 2022-12-28 14:25:07 +05:30
wal.go Add Zstandard compression option for wlog (#11666) 2023-07-11 14:57:57 +02:00
wal_test.go Add Zstandard compression option for wlog (#11666) 2023-07-11 14:57:57 +02:00

README.md

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A series of blog posts explaining different components of TSDB: