328c0ff5b0
Simplified the flow and tests.
...
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2018-03-13 14:12:32 +00:00
483da43660
compact: Exclude last block with shortest range instead of newest one by ULID.
...
Fixes https://github.com/prometheus/prometheus/issues/3943 issue.
Added tests.
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2018-03-13 12:30:27 +00:00
8cd32d8fd4
Improve usability of snapshots
2018-02-28 12:07:34 +01:00
d09283f60a
Add repair of bad index version
2018-02-09 13:11:03 +01:00
6c62437101
Fix metric unit ( #275 )
...
Fixes #274
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2018-02-05 08:36:18 +00:00
5d47710f8a
Fix panic on nil index writer
2018-01-24 15:55:04 +01:00
44052bc937
Make version for index format configurable
2018-01-11 00:49:16 +05:30
1e55b7987f
Improve comments, handle allPostingsKey properly
2017-12-22 09:43:34 +01:00
7087f68647
Return new block ULID in compactor
2017-12-21 12:02:05 +01:00
67f0ca8f0e
Move index and chunk encoders to own packages
2017-12-21 11:27:54 +01:00
f35e9c7fec
Merge branch 'master' into fix-dels
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-30 13:30:13 +05:30
24ff293dc9
Merge pull request #207 from Gouthamve/compact-fail
...
Don't retry failed compactions.
2017-11-30 12:58:19 +05:30
cf7f6108fd
Order postings lists in index file by key
...
Aligning postings list for similar keys close to each other improves
page cache hit rates in typical queries that select postings for
multiple label pairs with the same name.
2017-11-29 08:28:55 +01:00
1fc94a02d1
Don't retry failed compactions.
...
Fixes prometheus/prometheus#3487
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-23 15:16:20 +05:30
1627a47640
A copy is being assigned newChunk not the original
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-11-22 17:58:06 +05:30
a031cf7424
Return ULID in Compactor
2017-11-14 15:25:30 +01:00
f1512a368a
Expose ChunkSeriesSet and lookups methods.
2017-11-13 14:02:32 +01:00
3ef4326114
Refactor tombstone reader types
2017-11-13 13:38:07 +01:00
a27cf34a36
fix bugs on platform windows to pass all test case. ( #192 )
...
* fix bugs on platform windows to pass all test case.
* fix bugs on platform windows to pass all test case
* clean up codes
2017-10-31 15:37:41 +01:00
d17104f1f0
Prefix all metrics with `prometheus_*`
2017-10-20 12:32:32 +02:00
fb9da52b11
Add more verbose error handling for closing, reduce locking
...
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.
Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
2017-10-10 12:13:37 +02:00
da565f975e
Merge pull request #161 from prometheus/fileutil
...
Remove dependency on etcd/pkg/fileutil
2017-10-04 17:08:54 +05:30
f04ec031eb
compact: sync temporary directory
2017-10-04 12:22:09 +02:00
bbe72dccb9
Remove dependency on etcd/pkg/fileutil
2017-10-04 10:23:41 +02:00
c35d3a65bd
Add levels to all log lines.
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-09-28 12:49:34 +05:30
6ee254e353
Ensure postings are always sorted
...
IDs for new series are handed out before the postings are locked. Thus
series are not indexed in order of their IDs, which could result in only
partially sorted postings list.
Iterating over those silently skipped elements as the sort invariant was
violated.
2017-09-21 09:38:18 +02:00
f904cd385f
Do not build a superflous 'all' postings
2017-09-08 18:41:43 +02:00
c36d574290
Replace single head lock with granular locks
...
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
2017-09-05 14:41:39 +02:00
1ddedf2b30
Change series ID from uint32 to uint64
2017-09-04 16:08:38 +02:00
9f41d9fd3c
Instrument chunks on level 1 compactions
2017-09-01 16:10:10 +02:00
4cc37eecab
Refactor and add tests for compactor
2017-09-01 11:50:58 +02:00
3901b6e70b
Remove multiple heads
...
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
2017-09-01 11:50:58 +02:00
7438ed7035
Expose Intervals type for use by TombstoneReader.
...
TombstoneReader is exposed but Intervals is not.
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-25 16:06:36 +05:30
178f840295
Fix panic caused by 0 division
...
Introduced in #108
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-08-18 07:45:29 +02:00
905af27cf9
Refactor compactor
2017-08-09 11:10:29 +02:00
66ff7b12e9
Pool Chunk objects during compaction
2017-08-08 17:35:34 +02:00
2644c8665c
Don't allocate ChunkMetas, reuse postings slices
2017-08-06 20:41:24 +02:00
96d7f540d4
Persist series without allocating the full set
...
Change index persistence for series to not be accumulated in memory
before being written as one large batch. `Labels` and `ChunkMeta`
objects are reused.
This cuts down memory spikes during compaction of multiple blocks
significantly.
As part of the the Index{Reader,Writer} now have an explicit notion of
symbols and series must be inserted in order.
2017-08-06 12:06:41 +02:00
3951d8cc29
Fix selection of compactable blocks
2017-08-04 12:02:34 +02:00
5d7b5994d6
Fix compaction range selection
2017-07-13 16:13:59 +02:00
401bdfd4a6
Make Chunks return the number of samples inside
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:45:12 +02:00
1263a68875
Recompact blocks with large number of tombstones
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 18:27:31 +02:00
9a2ab732b5
Choose ranges by retention and min-block-duration
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-12 17:30:34 +02:00
ce23adb579
Initial compaction implementation for block-ranges
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-07 16:32:18 +02:00
03f95ee096
Fix directory file descriptor leak.
2017-06-12 00:05:04 +02:00
05e411a8eb
Improve heuristic to spread chunks across block
2017-06-08 11:30:32 +02:00
f006e2d1ab
Merge pull request #92 from prometheus/sources
...
Add list of source head blocks to meta.json
2017-06-07 11:50:44 +02:00
683b4c8c2a
Add list of source head blocks to meta.json
2017-06-07 11:06:45 +02:00
a1c8425357
Initial implementation of HeadBlock Snapshots
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-05 13:48:31 +05:30
3a5ae6b1a4
Merge pull request #82 from Gouthamve/deletes-1
...
Implement deletions
2017-05-27 22:06:45 +05:30
Bartek Plotka
Fabian Reinartz
Goutham Veeramachaneni
Simon Pasquier
Shubheksha Jalan
ranbochen