c59ed492b2
Vertical query merging and compaction ( #370 )
...
* Vertical series iterator
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Select overlapped blocks first in compactor Plan()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Added vertical compaction
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Code cleanup and comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix tests
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add benchmark for compaction
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Perform vertical compaction only when blocks are overlapping.
Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Benchmark for vertical compaction
* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Benchmark for query iterator and seek for non overlapping blocks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Vertical query merge only for overlapping blocks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Simplify logging in Compact(...)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Updated CHANGELOG.md
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Calculate overlapping inside populateBlock
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* MinTime and MaxTime for BlockReader.
Using this to find overlapping blocks in populateBlock()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Sort blocks w.r.t. MinTime in reload()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Log about overlapping in LeveledCompactor.write() instead of returning bool
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Log about overlapping inside LeveledCompactor.populateBlock()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Refactor createBlock to take optional []Series
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* review1
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* Updated CHANGELOG and minor nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Updated CHANGELOG
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Refactor iterator and seek benchmarks for Querier.
Also has as overlapping blocks.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Additional test case
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Split genSeries into genSeries and populateSeries
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Check error in benchmark
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Warn about overlapping blocks in reload()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-02-14 14:29:41 +01:00
0b72f9af4c
Merge pull request #270 from codesome/master
...
Head: don't create stones, delete samples directly
2019-02-08 12:35:01 +02:00
5481549324
Merge remote-tracking branch 'upstream/master'
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-02-07 22:15:06 +05:30
95334f13c5
Merge branch 'master' into update-makefile-common
...
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-02-07 12:10:22 +01:00
dac2b97dfd
make createBlock more generic so it can be used in other tests. ( #489 )
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-28 14:24:49 +03:00
4592b77035
Patch by Krasi ( https://github.com/codesome/tsdb/pull/3 )
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-01-11 22:04:09 +05:30
d7e505db34
Dont store stones in head, delete samples directly
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-01-08 22:38:41 +05:30
d5d7a097e1
Update Makefile.common
...
This change also uses the latest staticcheck version which comes with
new verifications, hence some clean up in the code.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-04 16:08:43 +01:00
48c439d26d
fix statick check errors ( #475 )
...
fix the tests for `check_license` and `staticcheck`
the static check also found some actual bugs.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-02 19:48:42 +03:00
520ab7dc53
re-add the missing prometheus_tsdb_wal_corruptions_total ( #473 )
...
closes https://github.com/prometheus/tsdb/issues/471
after implementing the new WAL this metric was missing so adding it again.
Also added it in a test to make sure it works as expected.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-18 13:24:56 +03:00
2962202ed3
fix windows tests ( #469 )
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-13 16:29:29 +03:00
bac9cbed2e
no overlapping on compaction when an existing block is not within default boundaries. ( #461 )
...
closes https://github.com/prometheus/prometheus/issues/4643
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-12-04 13:30:49 +03:00
0493efb7c5
repair wal when the record cannot be decoded ( #453 )
...
* repair wal when the record cannot be decoded
Currently repair is run only when the error happens in the reader.
A corruption can occur after the record is read and when it is decoded.
This change wraps the error at decoding as a CorruptionErr as this error
is expected to trigger a repair.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-30 13:37:04 +02:00
5a9ddeecef
fix lint errors ( #439 )
...
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-14 18:40:01 +02:00
d804a27062
refactor util funcs to allow re-usage. ( #419 )
...
* refactor util funcs to allow reusage.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-10-25 21:06:19 +01:00
a8351dc9d0
Using filepath.Join() instead of strings with slashes ( #428 )
...
fixes: https://github.com/prometheus/tsdb/issues/426
Using `filepath.Join()` instead of strings containing forward slash path delimiters (needed for non-*nix OSes), as suggested by @krasi-georgiev
2018-10-25 10:32:57 +01:00
d05611c027
removed some unused code and moved mockSeriesSet in querier_test ( #394 )
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
2db59a71a6
Fix assert order being backwards in a few places ( #388 )
...
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-21 09:23:01 +03:00
e779603633
Reuse testutil ( #393 )
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-21 11:47:59 +05:30
a8966cb53d
Fix race condition between gc and committing ( #378 )
...
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-09-17 19:58:42 +03:00
ad459ca1f4
re-added TestDeleteUntilCurMax TestDelete_e2e ( #382 )
...
no logic changes just uncommented and fixed the issues.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-17 14:28:55 +03:00
9be883670e
fix the test data file path in benchmarks. ( #345 )
...
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-09-13 19:34:26 +03:00
def912ce0e
Integrate new WAL and checkpoints
...
Remove the old WAL and drop in the new one
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-07-19 07:25:30 -04:00
667e539a7a
Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete
2018-07-06 20:21:32 +08:00
ea607b9fc3
Log series on rollback
...
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-06-28 09:04:07 -04:00
84a45cb79a
add rwmutex to prevent concurrent map read when delete series
...
Signed-off-by: codwu <wuhan9087@163.com>
2018-06-08 19:52:01 +08:00
64cce683cc
gofmt -s.
...
Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>
2018-05-07 15:39:54 +03:00
b7173eb0e5
fix some comments typos ( #315 )
2018-04-08 10:28:30 +01:00
79defa54df
Fix crash when a series has no block
2018-02-21 16:45:06 +01:00
adb6be73dd
move label read function int labels pkg
2017-12-21 11:55:58 +01:00
67f0ca8f0e
Move index and chunk encoders to own packages
2017-12-21 11:27:54 +01:00
e6e3fa566a
Remove verbose log line that is causing travis issues
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-12-21 12:16:11 +05:30
9f2935c5cf
Merge pull request #224 from Gouthamve/fix-214
...
Make sure gc'ed chunks are handled properly
2017-12-21 12:10:24 +05:30
bcde61b237
we shouldn't assume we'll receive the labels in sorted order
2017-12-18 15:32:39 -08:00
8326e410d0
decouple tsdb main.go (and tests) from prometheus/pkg
2017-12-18 15:26:16 -08:00
05d62ca842
Make sure gc'ed chunks are handled properly
...
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2017-12-14 19:48:39 +00:00
df7cc4dff5
remove Nil/NotNil functions
2017-12-08 13:42:08 -08:00
ca4e817372
move test utils to it's own package so we can use it in tests for other packages in tsdb repo
2017-12-06 17:06:14 -08:00
5aa2af255b
use test utils in head_test
2017-12-06 16:00:14 -08:00
3ef4326114
Refactor tombstone reader types
2017-11-13 13:38:07 +01:00
e5ce2bef43
Add explicit error to Querier.Select
...
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
2017-11-13 12:16:58 +01:00
1dad3370fd
Close WAL when closing the DB
...
Also, the `wal` field of the `DB` was not used anywhere, so this removes
it.
2017-11-11 14:56:23 +01:00
82796db37b
Ensure near-empty chunks end at correct boundary
...
We were determining a chunk's end time once it was one quarter full to
compute it so all chunks have uniform number of samples.
This accidentally skipped the case where series started near the end of
a chunk range/block and never reached that threshold. As a result they
got persisted but were continued across the range.
This resulted in corrupted persisted data.
2017-10-25 09:51:55 +02:00
c3e502b194
Merge pull request #168 from prometheus/fasterwal
...
wal: decode and process in separate threads.
2017-10-10 18:11:44 +02:00
fb9da52b11
Add more verbose error handling for closing, reduce locking
...
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.
Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
2017-10-10 12:13:37 +02:00
d3682d701c
wal: decode and process in separate threads.
2017-10-06 14:46:52 +02:00
203012169a
snapshot: Remove truncation check to restore func.
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-10-04 16:58:07 +05:30
162a48e4f2
Create series with ID recorded in WAL when reading it back
2017-09-19 11:31:16 +02:00
7ada9cd805
Simplify series create logic in head
2017-09-18 12:38:39 +02:00
c36d574290
Replace single head lock with granular locks
...
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
Ganesh Vernekar
Krasi Georgiev
Simon Pasquier
Ben
Chris Marchbanks
Alexey Palazhchenko
Fabian Reinartz
codwu
Mario Trangoni
Fabian Reinartz
Goutham Veeramachaneni
Callum Styan
Goutham Veeramachaneni
Julius Volz