c59ed492b2
Vertical query merging and compaction ( #370 )
...
* Vertical series iterator
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Select overlapped blocks first in compactor Plan()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Added vertical compaction
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Code cleanup and comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix tests
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add benchmark for compaction
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Perform vertical compaction only when blocks are overlapping.
Actions for vertical compaction:
* Sorting chunk metas
* Calling chunks.MergeOverlappingChunks on the chunks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Benchmark for vertical compaction
* BenchmarkNormalCompaction => BenchmarkCompaction
* Moved the benchmark from db_test.go to compact_test.go
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Benchmark for query iterator and seek for non overlapping blocks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Vertical query merge only for overlapping blocks
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Simplify logging in Compact(...)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Updated CHANGELOG.md
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Calculate overlapping inside populateBlock
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* MinTime and MaxTime for BlockReader.
Using this to find overlapping blocks in populateBlock()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Sort blocks w.r.t. MinTime in reload()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Log about overlapping in LeveledCompactor.write() instead of returning bool
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Log about overlapping inside LeveledCompactor.populateBlock()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Refactor createBlock to take optional []Series
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* review1
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* Updated CHANGELOG and minor nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Updated CHANGELOG
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Refactor iterator and seek benchmarks for Querier.
Also has as overlapping blocks.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Additional test case
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Split genSeries into genSeries and populateSeries
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Check error in benchmark
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Warn about overlapping blocks in reload()
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-02-14 14:29:41 +01:00
bf2239079d
refactor multi errors
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-02-11 12:28:46 +02:00
d48606827c
simplify closers
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-02-08 13:35:32 +02:00
315de4c782
fix windows tests
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-30 11:40:40 +02:00
53c18e7a41
use a global indexFilename constant
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-29 10:32:32 +02:00
1a9d08adc5
Don't write empty blocks ( #374 )
...
* Dont write empty blocks when a compaction results in a block with no samples.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-01-18 11:35:16 +03:00
ebf5d74325
Added storage size based retention method and new metrics ( #343 )
...
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
2019-01-16 13:03:52 +03:00
c93e261466
Reduce memory taken up by posting/symbol tables.
...
Reuse the string already allocated for symbols
in the posting tables.
Use a slice for symbols in v2 format.
Move symbol size logic into the index code.
Avoid duplication of lookupSymbol logic.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-11-22 09:36:39 +00:00
5a9ddeecef
fix lint errors ( #439 )
...
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-14 18:40:01 +02:00
3a08a71d86
LabelNames() method to get all unique label names ( #369 )
...
* LabelNames() method to get all unique label names
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-07 17:52:41 +02:00
6e712963e2
Fix updating of NumTombstones in block.Delete(..) ( #385 )
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-27 14:43:22 +03:00
d05611c027
removed some unused code and moved mockSeriesSet in querier_test ( #394 )
...
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-09-21 11:07:35 +03:00
5ae6c60d39
Handle a bunch of unchecked errors ( #365 )
...
As discovered by "gosec".
Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-09-20 11:33:52 +03:00
77d2c7cf3d
Added metric for symbol table size
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-09-08 23:58:36 +05:30
667e539a7a
Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete
2018-07-06 20:21:32 +08:00
1e1b2e163d
Make interval overlap comparisons more explicit
...
Blocks are half-open intervals [a, b), while all other intervals
(chunks, head, ...) are closed intervals [a, b].
Make that distinction explicit by defining `OverlapsClosedInterval()`
methods for blocks and chunks, and using them in place of the more
generic `intervalOverlap()` function.
This change also fixes `db.Querier()` and `db.Delete()`, which could
previously return one extraneous block at the end of the specified
interval.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
2018-07-02 10:35:08 +02:00
d907928e57
Clarify docs, error on unexpected meta read errors
...
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-06-27 12:05:21 -04:00
af9003dcef
Add resilience to crashes during deletion
...
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-06-27 09:50:31 -04:00
cd145c90d5
remove `put` function and use RLock in `Iter` function
...
Signed-off-by: codwu <wuhan9087@163.com>
2018-06-25 21:52:11 +08:00
84a45cb79a
add rwmutex to prevent concurrent map read when delete series
...
Signed-off-by: codwu <wuhan9087@163.com>
2018-06-08 19:52:01 +08:00
528439aa93
Cleanup new blocks on 'CleanTombstones' faliure.
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-06-01 15:48:38 -04:00
f82a1fe4f2
Simplify stones counting.
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2018-04-03 18:17:02 +05:30
8a301b126a
Merge pull request #307 from mjtrangoni/fixes
...
Fix some megacheck and unconvert issues
2018-04-03 12:27:27 +05:30
c2182820ed
fix megacheck issue: should omit values from range
2018-03-21 22:39:15 +01:00
79defa54df
Fix crash when a series has no block
2018-02-21 16:45:06 +01:00
d09283f60a
Add repair of bad index version
2018-02-09 13:11:03 +01:00
2c3400ab4e
Add block ULID to index reader errors
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2018-01-16 23:48:58 +05:30
7b27dc2109
address review comments, fix index tests
2018-01-11 13:20:42 +05:30
44052bc937
Make version for index format configurable
2018-01-11 00:49:16 +05:30
129773b41a
merge BlockMeta and blockMeta struct, implement v1 and v2 index readers
2018-01-08 23:03:35 +05:30
1e55b7987f
Improve comments, handle allPostingsKey properly
2017-12-22 09:43:34 +01:00
67f0ca8f0e
Move index and chunk encoders to own packages
2017-12-21 11:27:54 +01:00
3158b03e6c
Select series with label unset for != and !~
...
Fixes https://github.com/prometheus/prometheus/issues/3575
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2017-12-19 02:47:31 +00:00
8a5ea9db74
Merge branch 'master' into tomb-clean
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-30 13:15:23 +05:30
24ff293dc9
Merge pull request #207 from Gouthamve/compact-fail
...
Don't retry failed compactions.
2017-11-30 12:58:19 +05:30
e7445d00b0
Add a function to cleanup tombstones.
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-27 20:03:37 +05:30
1fc94a02d1
Don't retry failed compactions.
...
Fixes prometheus/prometheus#3487
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-23 15:16:20 +05:30
f1512a368a
Expose ChunkSeriesSet and lookups methods.
2017-11-13 14:02:32 +01:00
3ef4326114
Refactor tombstone reader types
2017-11-13 13:38:07 +01:00
e5ce2bef43
Add explicit error to Querier.Select
...
This has been a frequent source of debugging pain since errors are
potentially delayed to a much later point. They bubble up in an
unrelated execution path.
2017-11-13 12:16:58 +01:00
d578b10d5c
chunk: make reader accept abstract ByteSlice
2017-11-10 10:38:22 +00:00
b7c3cfecbf
index: abstract ByteSlice and adjust indexReader
...
This replaces the builtin byte slice with an interface for the index
reader. This allows the complex decoding of the index file format
to be used against more generalized implementations.
2017-11-09 17:38:32 +00:00
4fdf9b195c
Validate index TOC checksum on read
2017-10-25 18:12:13 -04:00
fb9da52b11
Add more verbose error handling for closing, reduce locking
...
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.
Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
2017-10-10 12:13:37 +02:00
78df406dac
Allocate and cache strings for persisted blocks
...
This change loads the full symbol table when we open a persisted block
and allocates a string for each. This ensures that strings retrieved
through the index can be used after the block was closed.
Before we backed the strings by the mmap'd byte regions which would
segfault in this case.
Also remove an inconsistency in the disk format and move both offset
tables to the end (breaking change).
2017-10-02 15:56:57 +02:00
8919baef03
Expose NewIndexReader() and cleanups
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-13 13:47:20 +05:30
1ddedf2b30
Change series ID from uint32 to uint64
2017-09-04 16:08:38 +02:00
4cc37eecab
Refactor and add tests for compactor
2017-09-01 11:50:58 +02:00
5cf2662074
Refactor WAL into Head and misc improvements
2017-09-01 11:50:58 +02:00
3901b6e70b
Remove multiple heads
...
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
Ganesh Vernekar
Krasi Georgiev
mknapphrt
Brian Brazil
Krasi Georgiev
Julius Volz
codwu
Benoît Knecht
Fabian Reinartz
Goutham Veeramachaneni
Mario Trangoni
Simon Pasquier
Fabian Reinartz
Shubheksha Jalan
Sunny Klair