d3edfb5540
Test multiple full write/restore cylcles in WAL
2017-03-08 21:02:08 +01:00
ca1bc920b7
Test updates
2017-03-08 16:54:13 +01:00
cb4dde7659
Fix WAL log recovery bug
...
This fixes a bug where the last WAL file was closed after consuming it
instead of being left open for further writes.
Reloading of blocks on startup considers loading head blocks now.
2017-03-08 16:53:07 +01:00
8a7addfc44
Split persistence by chunk/index instead of read/write
2017-03-07 12:48:52 +01:00
3910b913bd
Revert interval overlap comparison changes
2017-03-07 12:01:25 +01:00
2841499212
Simplify and split up iterators
...
This simplifies some of the iterators by loading chunks from the
ChunkReader earlier, filtering of chunks vs filtering or series is
split into separate iterators for easier testing
2017-03-07 11:30:35 +01:00
ed63636de4
Add initial index write/read symmetry test.
2017-03-06 17:36:03 +01:00
87805fb83f
Remove Partitioned* code
2017-03-06 17:34:49 +01:00
eedbebe1d7
Misc cleanups
2017-03-06 14:34:49 +01:00
50791a412e
Remove leaky Appender abstraction for hashedAppend
2017-03-06 14:27:33 +01:00
87d270fe11
Fix interval check, make DB shutdown concurrent
2017-03-06 12:13:15 +01:00
04783831e2
Correctly close compacted blocks.
2017-03-06 09:33:55 +01:00
55a9b5428a
Add separate head mutex
...
Introduce a seperate mutex for the head blocks to avoid a race where
a post-compaction reload may run between switching the DB's base mutex
to create a new head block in an appender.
2017-03-04 17:41:01 +01:00
3e569bc964
Optimize memory allocations in writer
2017-03-02 23:35:02 +01:00
cc0a7c8279
Create alloc and inuse space heap profile
2017-03-02 21:54:30 +01:00
ffb24a98f4
Add missing unlock, run debug endpoint in benchmark
2017-03-02 21:54:02 +01:00
21b97d1e04
Fix compaction selection procedure
2017-03-02 15:10:13 +01:00
327e07e8d0
Properly sync moves and removes in compactor
2017-03-02 14:35:06 +01:00
92120448c2
Properly cleanup compacted dirs, fixes, docs
2017-03-02 14:32:09 +01:00
2c3e778d90
Compactor interface
2017-03-02 09:17:30 +01:00
2c3b56350a
Write meta.json file atomically
2017-03-01 17:19:57 +01:00
6c4217276f
Periodically trigger compaction
...
This addresses an issue where the compaction triggered on cutting
a new block doesn't find anything as the writers are still active on the
block that should be ready for compaction.
2017-03-01 16:52:21 +01:00
db5c88ea9a
Misc compaction fixes
2017-02-28 08:40:51 +01:00
b281e4e39b
Accept prometheus.Registerer in constructor
2017-02-28 07:26:24 +01:00
306831f151
Add per-block state ULID
2017-02-27 10:46:15 +01:00
9a5dfadb09
Tweak IndexWriter interface, serialize directly into dir
2017-02-25 07:24:20 +01:00
78780cd2ba
Segment chunk file
...
This adds write path support for segmented chunk data files.
Files of 512MB are pre-allocated and written to. If the file size
is exceeded, the next file is started. On completion, files
are truncated to their final size.
2017-02-23 10:50:22 +01:00
a3b47c4929
Create default logger for DB
2017-02-19 16:04:37 +01:00
f734773214
Replace per-file locking with single PID lock file
...
File locks have a multitude of problems that make them hard to use
correctly. As they are just advisory, they are only meaningful to
prevent accidents like running the same process twice.
A simple PID file lock works reliably in those cases and is simpler.
2017-02-19 13:01:19 +01:00
c808928b90
SeriesWriter -> ChunkWriter
...
This commit simplifies a SeriesWriter into a ChunkWriter and detaches
it entirely from the notion of a series and the series index.
2017-02-19 11:27:31 +01:00
a3d042b54e
Support multiple chunk files in read path
2017-02-18 17:33:20 +01:00
afa084920c
Use Castagnoli polynomial for CRC32s; cache them
2017-02-15 21:13:17 -08:00
9c7a88223e
Add full encode/decode WAL cycle test
2017-02-14 21:55:50 -08:00
2c97428a79
Add basic WAL tests
2017-02-14 16:12:56 -08:00
84e8027a8e
Close consumed WAL segments, verify checksums on read
2017-02-14 00:42:06 -08:00
eeb03e97e2
Add headers and format flag for WAL segments
2017-02-14 00:24:53 -08:00
79944a5912
Break out WAL into segment files
2017-02-13 23:53:19 -08:00
987a90d149
Don't delete files if retention duration is 0
2017-02-11 11:49:17 -08:00
9c76624df2
Add initial retention cutoff
2017-02-09 17:54:26 -08:00
f1435f2e2c
Track appended samples properly in metric
2017-02-08 16:13:16 -08:00
012cf4ef25
Count writer references on head blocks
2017-02-04 11:53:52 +01:00
5a1c8eaa0e
Fix missing appends after reference lookups
2017-02-02 11:10:17 +01:00
f556036037
Close blocks before overwriting with compacted one
...
This prevents the meta.json file of a compacted block
to be overwritten with data of the old block.
2017-02-02 09:32:06 +01:00
ac5229e1b4
Correctly write empty blocks and extend appenders by new blocks
2017-02-02 07:58:54 +01:00
42fa342229
Improve multi-head handling
...
This improves handling of multiple head blocks. Configuration is
simplified to specify the number of concurrently appendable blocks.
2017-02-01 21:35:21 +01:00
30efe4a58c
Support writing to multiple head blocks
...
This is an initial (and hacky) first pass on allowing
appending to multiple blocks simultaniously to avoid
dropping samples right after cutting a new head block.
It's also required for cases like the PGW, where a scrape may
contain varying timestamps.
2017-02-01 15:57:28 +01:00
a61a31a5d7
compaction: add fast-path for compacting mem blocks
2017-01-30 09:49:36 +01:00
c20cc44b06
Add docs, write sequence number to meta.json
2017-01-29 08:11:47 +01:00
30d8866c13
Limit compaction range, make cut and compact concurrent
2017-01-20 07:58:19 +01:00
67d185ceb9
Compact based on compaction generation
2017-01-19 19:45:52 +01:00
Fabian Reinartz