Commit Graph

28 Commits

Author SHA1 Message Date
Fabian Reinartz 79944a5912 Break out WAL into segment files 2017-02-13 23:53:19 -08:00
Fabian Reinartz 9c76624df2 Add initial retention cutoff 2017-02-09 17:54:26 -08:00
Fabian Reinartz 012cf4ef25 Count writer references on head blocks 2017-02-04 11:53:52 +01:00
Fabian Reinartz 5a1c8eaa0e Fix missing appends after reference lookups 2017-02-02 11:10:17 +01:00
Fabian Reinartz 30efe4a58c Support writing to multiple head blocks
This is an initial (and hacky) first pass on allowing
appending to multiple blocks simultaniously to avoid
dropping samples right after cutting a new head block.
It's also required for cases like the PGW, where a scrape may
contain varying timestamps.
2017-02-01 15:57:28 +01:00
Fabian Reinartz c20cc44b06 Add docs, write sequence number to meta.json 2017-01-29 08:11:47 +01:00
Fabian Reinartz 5fb01d41aa Use new Prometheus text format parser 2017-01-16 21:29:53 +01:00
Fabian Reinartz fde69dab49 Use buffer pool for head appenders 2017-01-12 20:03:44 +01:00
Fabian Reinartz a317f252b9 Expose series references to clients
This exposes a reference number of a series represented by a label set
to clients.
Subsequent samples can be directly added via the reference rather than
repeatedly passing in the full labels. This drasitcally speeds up the
append process.

The appender chain uses different sections of the reference number for
assignment to child appenders and invalidating reference numbers as
necessary.

Clients can either pass out reference numbers themselves or have their
own optimized lookup, i.e. by directly associating unparsed metric
descriptors strings with reference numbers.
2017-01-12 20:00:54 +01:00
Fabian Reinartz 80affd98a8 Add barrier to benchmark writer
This adds a barrier to avoid issues with unfair goroutine scheduling
that causes some fake scrapers to run away from the other ones.
2017-01-11 13:01:30 +01:00
Fabian Reinartz c32a94d409 Unexport HeadBlock, export Block interface 2017-01-10 15:41:57 +01:00
Fabian Reinartz d86e8a63c7 Report correct number of appended samples 2017-01-10 11:17:37 +01:00
Fabian Reinartz 300f4e2abf Use separate lock for series creation
This uses the head block's own lock to only lock if new series were
encountered.
In the general append case we just need to hold a
2017-01-06 18:10:50 +01:00
Fabian Reinartz c61b310210 Naive size-based compaction
This adds naive compaction that tries to compact three
blocks of roughly equal size.
It decides based on samples present in a block and has no
safety measures considering the actual file size.
2017-01-06 13:53:05 +01:00
Fabian Reinartz 3ed2c2a14b Rename Partition to regular DB, DB to PartitionedDB 2017-01-06 11:40:09 +01:00
Fabian Reinartz cabd7e4ebc Add small testdata file 2016-12-22 19:15:37 +01:00
Fabian Reinartz ede733ab6c Extract labels package 2016-12-21 09:39:01 +01:00
Fabian Reinartz ee217adc7e Redfine append interface, remove old Prometheus storage from bench 2016-12-21 00:02:37 +01:00
Fabian Reinartz e561c91d53 Implement proper buffered iterator
This adds a proper duration based lookback buffer for series iterators
to allow advancing sequentially while remaining able to calculate time
aggregating functions such as `rate` backwards.

It uses an array ring buffer to minimize heap allocations for
potentially hundreds of thousands of series for a single query.
2016-12-14 21:14:44 +01:00
Fabian Reinartz 3a528c3078 Write plain postings list index 2016-12-10 09:44:00 +01:00
Fabian Reinartz 40a451694f Refactor persistence into interfaces 2016-12-09 20:45:46 +01:00
Fabian Reinartz 74f8dfd95d Persist blocks periodically 2016-12-09 13:41:38 +01:00
Fabian Reinartz 8aa99a3ebd misc 2016-12-09 10:00:14 +01:00
Fabian Reinartz 63b887eb62 Add Makefile 2016-12-08 12:00:05 +01:00
Fabian Reinartz b845f8d3a1 Reduce test data allocations 2016-12-08 11:59:54 +01:00
Fabian Reinartz ce82bdb71a Add write benchmark utility 2016-12-07 17:30:10 +01:00
Fabian Reinartz ff29705571 Delete old benchmarking tool 2016-12-04 13:15:49 +01:00
Fabian Reinartz 40c27a1c2f Add benchmark utility 2016-11-15 10:35:42 +01:00