RepoMirrors
/
prometheus
mirror of https://github.com/prometheus/prometheus
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
975 Commits 245 Branches 436 Tags 657 MiB
Commit Graph

40 Commits

Author SHA1 Message Date
Ganesh Vernekar c59ed492b2 Vertical query merging and compaction (#370) 
* Vertical series iterator

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Select overlapped blocks first in compactor Plan()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Added vertical compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Code cleanup and comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add benchmark for compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Perform vertical compaction only when blocks are overlapping.

Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for vertical compaction

* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Benchmark for query iterator and seek for non overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Vertical query merge only for overlapping blocks

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplify logging in Compact(...)

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG.md

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Calculate overlapping inside populateBlock

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* MinTime and MaxTime for BlockReader.

Using this to find overlapping blocks in populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Sort blocks w.r.t. MinTime in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping in LeveledCompactor.write() instead of returning bool

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Log about overlapping inside LeveledCompactor.populateBlock()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor createBlock to take optional []Series

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* review1

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>

* Updated CHANGELOG and minor nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* nits

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Updated CHANGELOG

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor iterator and seek benchmarks for Querier.

Also has as overlapping blocks.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Additional test case

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Split genSeries into genSeries and populateSeries

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Check error in benchmark

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Warn about overlapping blocks in reload()

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2019-02-14 14:29:41 +01:00
Krasi Georgiev bf79c767f0 new line 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-12 11:08:09 +02:00
Krasi Georgiev beee5c58f3 Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-12 10:56:45 +02:00
Krasi Georgiev 07df4fd383 nits 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-11 11:25:57 +02:00
Krasi Georgiev 0f8f5027ef remove nested for if 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-08 18:09:23 +02:00
Krasi Georgiev da9da9fbee fix the sleep logic 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-08 12:39:25 +02:00
Krasi Georgiev 457534d5c4 simplify nesting. 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-08 11:36:30 +02:00
Krasi Georgiev 752ab86e4e change the test block series for more stable tests 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-02-04 11:14:39 +02:00
Krasi Georgiev 6c34eb8b63 nits 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 12:17:15 +02:00
Krasi Georgiev 1603222bbc small refactor of openTestDB to handle errors properly. 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-30 11:40:12 +02:00
Krasi Georgiev b521559c3b do a proper cleanup for a failed reload after a compaction 
a failed reload immediately after a compaction should delete the
resulting block to avoid creating blocks with the same time range.

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-29 10:31:54 +02:00
Krasi Georgiev fa1c00f9e2 no backgorund 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-24 14:38:45 +02:00
Krasi Georgiev 1b0d85bbf2 use a db copy instead of creating it again. 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-24 14:15:32 +02:00
Krasi Georgiev 8ffd705346 fix the misleading log during compaction cancelation 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-24 13:33:12 +02:00
Krasi Georgiev 9638c13ec8 test compares normal vs canceled compaction times 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-24 12:48:56 +02:00
Krasi Georgiev 9874377ead nits 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-18 19:22:45 +02:00
Krasi Georgiev 921f82cfc1 WIP 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-18 18:58:17 +02:00
Krasi Georgiev 0b31b8af88 Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2019-01-18 16:58:18 +02:00
Krasi Georgiev a2779cc901
fix flaky tests: TestDisableAutoCompactions,TestBlockRanges (#472) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-12 14:49:03 +03:00
Krasi Georgiev db127a60e0 changelog 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-10 12:48:30 +02:00
Krasi Georgiev fced260a24 test 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-10 12:48:30 +02:00
Krasi Georgiev bd5ccee5c1 use context to cancel compactions 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-10 12:48:30 +02:00
Krasi Georgiev bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. (#461) 
closes https://github.com/prometheus/prometheus/issues/4643

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-12-04 13:30:49 +03:00
Krasi Georgiev b75d702ceb
fix flaky compaction test (#458) 
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-21 00:33:14 +02:00
Krasi Georgiev 7f00217d77
Allow manual compaction for tests when compaction is disabled globally. (#412) 
for tests we need to control when a compaction happens so with this
change automated compaction can be disabled, but allow to run it
manually it tests.

fixes failing tests in : https://github.com/prometheus/tsdb/pull/374

Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-11-20 12:34:26 +02:00
Bartek Płotka 047b1b1357 compact: Verify for chunks outside of compacted time range. Added unit test for populateBlocs. (#349) 
* compact: Verify for chunks outside of compacted time range. 
 Unit test for populateBlocs.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
 2018-10-12 12:45:19 +03:00
Bartek Plotka 2aceac0a12 Fixed some comments. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-13 15:30:00 +00:00
Bartek Plotka 328c0ff5b0 Simplified the flow and tests. 
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-13 14:12:32 +00:00
Bartek Plotka 483da43660 compact: Exclude last block with shortest range instead of newest one by ULID. 
Fixes https://github.com/prometheus/prometheus/issues/3943 issue.

Added tests.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
 2018-03-13 12:30:27 +00:00
Krasi Georgiev eb6cdbcb5b some test funcitons didn't cleanup after themselves. 2018-02-23 18:45:59 +02:00
Callum Styan ca4e817372 move test utils to it's own package so we can use it in tests for other packages in tsdb repo 2017-12-06 17:06:14 -08:00
Callum Styan fa26e2b6c3 use test utils in compact_test 2017-12-06 16:30:24 -08:00
Goutham Veeramachaneni 1fc94a02d1
Don't retry failed compactions. 
Fixes prometheus/prometheus#3487

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-11-23 15:16:20 +05:30
Fabian Reinartz 4cc37eecab Refactor and add tests for compactor 2017-09-01 11:50:58 +02:00
Goutham Veeramachaneni 178f840295 Fix panic caused by 0 division 
Introduced in #108

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-08-18 07:45:29 +02:00
Fabian Reinartz 905af27cf9 Refactor compactor 2017-08-09 11:10:29 +02:00
Fabian Reinartz 3951d8cc29 Fix selection of compactable blocks 2017-08-04 12:02:34 +02:00
Fabian Reinartz 5d7b5994d6 Fix compaction range selection 2017-07-13 16:13:59 +02:00
Goutham Veeramachaneni 9a2ab732b5 Choose ranges by retention and min-block-duration 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-12 17:30:34 +02:00
Goutham Veeramachaneni ce23adb579 Initial compaction implementation for block-ranges 
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
 2017-07-07 16:32:18 +02:00