RepoMirrors
/
prometheus
mirror of https://github.com/prometheus/prometheus
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Jesus Vazquez c1b669bf9b
Add out-of-order sample support to the TSDB (#11075) 
* Introduce out-of-order TSDB support

This implementation is based on this design doc:
https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing

This commit adds support to accept out-of-order ("OOO") sample into the TSDB
up to a configurable time allowance. If OOO is enabled, overlapping querying
are automatically enabled.

Most of the additions have been borrowed from
https://github.com/grafana/mimir-prometheus/
Here is the list ist of the original commits cherry picked
from mimir-prometheus into this branch:
- 4b2198d7ec
- 2836e5513f
- 00b379c3a5
- ff0dc75758
- a632c73352
- c6f3d4ab33
- 5e8406a1d4
- abde1e0ba1
- e70e769889
- df59320886

Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Dieter Plaetinck <dieter@grafana.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* gofumpt files

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Add license header to missing files

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix OOO tests due to existing chunk disk mapper implementation

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix truncate int overflow

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Add Sync method to the WAL and update tests

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* remove useless sync

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Update minOOOTime after truncating Head

* Update minOOOTime after truncating Head

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Add a unit test

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Load OutOfOrderTimeWindow only once per appender

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix OOO Head LabelValues and PostingsForMatchers

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix replay of OOO mmap chunks

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Remove unnecessary err check

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Prevent panic with ApplyConfig

Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Run OOO compaction after restart if there is OOO data from WBL

Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Apply Bartek's suggestions

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Refactor OOO compaction

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Address comments and TODOs

- Added a comment explaining why we need the allow overlapping
  compaction toggle
- Clarified TSDBConfig OutOfOrderTimeWindow doc
- Added an owner to all the TODOs in the code

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Run go format

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix remaining review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Change wbl reference when truncating ooo in TestHeadMinOOOTimeUpdate

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>

* Fix TestWBLAndMmapReplay test failure on windows

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Address most of the feedback

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Refactor the block meta for out of order

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix windows error

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review comments

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Ganesh Vernekar 15064823+codesome@users.noreply.github.com
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Dieter Plaetinck <dieter@grafana.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
 		2022-09-20 22:35:50 +05:30
..
agent Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
chunkenc Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
chunks Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
docs Fix type byte of WAL metadata records in docs (#11035) 2022-07-19 18:22:02 +05:30
encoding chore: remove duplicate word in comments (#11225) 2022-08-27 22:21:41 +02:00
errors Address golint failures from revive 2021-10-23 00:53:11 +02:00
fileutil Update go to 1.19, set min version to 1.18 (#11279) 2022-09-07 11:30:48 +02:00
goversion Add new Go build tags. 2021-08-27 10:24:14 +02:00
index Fix typo in symbol table size exceeded error message (#10746) 2022-05-25 10:40:36 +02:00
record Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
test Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
testdata tsdb: Delete blocks atomically; Remove tmp blocks on start; Added test. (#7772) 2020-08-11 06:56:08 +01:00
tombstones refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
tsdbutil Remove unnecessary tsdb/tsdbutil/buffer.go (#11302) 2022-09-13 19:36:32 +05:30
wal Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
CHANGELOG.md Rename default branch to main 2021-02-22 20:28:02 +01:00
README.md Update godoc links and remove note about TSDB versioning (#10754) 2022-05-26 18:34:43 +10:00
block.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
block_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
blockwriter.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
blockwriter_test.go tsdb: in tests use labels.FromStrings 2022-09-09 13:34:49 +02:00
compact.go CompactBlockMetas should produce correct mint/maxt for overlapping blocks. (#10108) 2022-01-05 15:10:00 +05:30
compact_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
db.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
db_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
example_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
exemplar.go tsdb: use simpler map key to improve exemplar ingest performance (#10111) 2022-01-06 15:58:58 +05:30
exemplar_test.go tsdb: in tests use labels.FromStrings 2022-09-09 13:34:49 +02:00
head.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_append.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_bench_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_read.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_read_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
head_wal.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
isolation.go tsdb: comment reason for isolation tracking reads (#11301) 2022-09-13 11:11:03 +02:00
isolation_test.go Add the ability to disable tsdb isolation (#9270) 2021-11-19 15:41:32 +05:30
mocks_test.go Move packages out of deprecated pkg directory 2021-11-09 08:03:10 +01:00
ooo_head.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
ooo_head_read.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
ooo_head_read_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
ooo_head_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
querier.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
querier_bench_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
querier_test.go Add out-of-order sample support to the TSDB (#11075) 2022-09-20 22:35:50 +05:30
repair.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00
repair_test.go tsdb: in tests use labels.FromStrings 2022-09-09 13:34:49 +02:00
tsdbblockutil.go tsdb: commit data periodically in CreateBlock (#10788) 2022-06-17 11:26:19 +05:30
wal.go tsdb: extract functions to encode and decode labels (#11045) 2022-07-26 20:12:00 +05:30
wal_test.go refactor: move from io/ioutil to io and os packages (#10528) 2022-04-27 11:24:36 +02:00

README.md

TSDB

GoPkg

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A series of blog posts explaining different components of TSDB: