prometheus/tsdb
Kemal Akkoyun 66dfb951c4
*: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251)
* Add errors and Warnings to SeriesSet

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Change Querier interface and refactor accordingly

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactor promql/engine to propagate warnings at eval stage

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Make sure all the series from all Selects are pre-advanced

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Separate merge series sets

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Clean

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactor merge querier failure handling

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Refactored and simplified fanout with improvements from incoming chunk iterator PRs.

* Secondary logic is hidden, instead of weird failed series set logic we had.
* Fanout is well commented
* Fanout closing record all errors
* MergeQuerier improved API (clearer)
* deferredGenericMergeSeriesSet is not needed as we return no samples anyway for failed series sets (next = false).

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix formatting

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix CI issues

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Added final tests for error handling.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

* Moved hints in populate to be allocated only when needed.
* Used sync.Once in secondary Querier to achieve all-or-nothing partial response logic.
* Select after first Next is done will panic.

NOTE: in lazySeriesSet in theory we could just panic, I think however we can
totally just return error, it will panic in expand anyway.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Utilize errWithWarnings

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix recently introduced expansion issue

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Add tests for secondary querier error handling

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Implement lazy merge

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Add name to test cases

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Reorganize

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review comments

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Address review comments

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Remove redundant warnings

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

* Fix rebase mistake

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-06-09 17:57:31 +01:00
..
chunkenc storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005) 2020-03-24 20:15:47 +00:00
chunks Remove time based m-map file creation (#7314) 2020-05-29 20:08:41 +05:30
cmd/tsdb *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
docs/format M-map full chunks of Head from disk (#6679) 2020-05-06 21:00:00 +05:30
encoding Avoid WriteAt for Postings. 2019-12-16 15:30:49 +00:00
errors Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
fileutil Replace fileutil.ReadDir with ioutil.ReadDir (#7029) (#7033) 2020-04-06 19:04:20 +05:30
goversion Fix some typo in comments (#6730) 2020-01-31 12:11:52 +05:30
index Goroutine: Fix ambiguous variable (#7175) 2020-04-28 11:02:26 +01:00
record Add unit test case for func Type in record.go (#7082) 2020-05-27 12:08:33 +05:30
test Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
testdata Handle V1 indexes, some of which have unsorted posting offset tables. (#6564) 2020-01-06 14:06:11 +00:00
tombstones tsdb: add tombstonesHeaderSize constant (#7028) 2020-03-22 12:59:35 +05:30
tsdbutil storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005) 2020-03-24 20:15:47 +00:00
wal Capitalizing first letter of all log lines (#7043) 2020-04-11 09:22:18 +01:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
block_test.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
block.go Revert head posting optimization 2020-03-25 20:19:33 +01:00
CHANGELOG.md Spelling (#6517) 2020-01-02 15:54:09 +01:00
compact_test.go Callbacks for lifecycle of series in TSDB (#7159) 2020-05-20 18:52:08 +05:30
compact.go More explicit chunks and head error handling. (#7277) 2020-05-22 12:03:23 +03:00
db_test.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
db.go Use min and maxt of the range head when creating a block (#7282) 2020-05-22 17:00:06 +05:30
head_bench_test.go Callbacks for lifecycle of series in TSDB (#7159) 2020-05-20 18:52:08 +05:30
head_test.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
head.go Optimise lowWatermark in Isolation (#7332) 2020-06-03 20:09:05 +02:00
isolation_test.go Optimise lowWatermark in Isolation (#7332) 2020-06-03 20:09:05 +02:00
isolation.go Optimise lowWatermark in Isolation (#7332) 2020-06-03 20:09:05 +02:00
mocks_test.go Revert head posting optimization 2020-03-25 20:19:33 +01:00
querier_bench_test.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
querier_test.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
querier.go *: Consistent Error/Warning handling for SeriesSet iterator: Allowing Async Select (#7251) 2020-06-09 17:57:31 +01:00
README.md Edit TSDB README badges 2019-10-24 15:35:47 +05:30
repair_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
repair.go Capitalizing first letter of all log lines (#7043) 2020-04-11 09:22:18 +01:00
tsdbblockutil.go Callbacks for lifecycle of series in TSDB (#7159) 2020-05-20 18:52:08 +05:30
wal_test.go Replace fileutil.ReadDir with ioutil.ReadDir (#7029) (#7033) 2020-04-06 19:04:20 +05:30
wal.go Capitalizing first letter of all log lines (#7043) 2020-04-11 09:22:18 +01:00

TSDB

GoDoc

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.