prometheus/storage/remote
Bryan Boreham 60804c5a09
remote_write: reduce blocking from garbage-collect of series (#9109)
* Refactor: pass segment-reading function as param

To allow a different implementation to be used when garbage-collecting.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* remote_write: reduce blocking from GC of series

Add a method `UpdateSeriesSegment()` which is used together with
`SeriesReset()` to garbage-collect old series. This allows us to
split the lock around queueManager series data and avoid blocking
`Append()` while reading series from the last checkpoint.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Cosmetic: review feedback on comments

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* remote-write benchmark: include GC of series

Reduce the total number of samples per iteration from 5000*5000
(25 million) which is too big for my laptop, to 1*10000.

Extend `createTimeseries()` to add additional labels, so that the
queue manager is doing more realistic work.

Move the Append() call to a background goroutine - this works because
TestWriteClient uses a WaitGroup to signal completion.

Call `StoreSeries()` and `SeriesReset()` while adding samples, to
simulate the garbage-collection that wal.Watcher does.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Change BenchmarkSampleDelivery to call UpdateSeriesSegment

This matches what Watcher.garbageCollectSeries() is doing now

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-07-27 13:21:48 -07:00
..
chunked_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
chunked.go
client_test.go Adds support to configure retry on Rate-Limiting from remote-write config. 2021-02-16 14:52:49 +05:30
client.go Upgrade prometheus/common to v0.21.0 2021-04-21 12:19:16 +02:00
codec_test.go Various enhancements and refactorings for remote write receiver: 2021-02-08 20:41:23 +00:00
codec.go Various enhancements and refactorings for remote write receiver: 2021-02-08 20:41:23 +00:00
ewma.go Replace sync/atomic with uber-go/atomic (#7683) 2020-07-30 13:15:42 +05:30
intern_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
intern.go Move away from testutil, refactor imports (#8087) 2020-10-22 11:00:08 +02:00
max_timestamp.go Remote: Do not collect non-initialized timestamp metrics (#8060) 2020-10-15 23:53:59 +02:00
metadata_watcher_test.go Fix some gofmt warnings (#8743) 2021-04-22 08:43:30 -06:00
metadata_watcher.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
queue_manager_test.go remote_write: reduce blocking from garbage-collect of series (#9109) 2021-07-27 13:21:48 -07:00
queue_manager.go remote_write: reduce blocking from garbage-collect of series (#9109) 2021-07-27 13:21:48 -07:00
read_handler_test.go Move remote read handler to remote package. (#8536) 2021-02-26 16:43:19 +00:00
read_handler.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
read_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
read.go LabelNames API with matchers (#9083) 2021-07-20 18:08:08 +05:30
sigv4_test.go Contribute grafana/agent sigv4 code (#8509) 2021-03-08 12:20:09 -07:00
sigv4.go Contribute grafana/agent sigv4 code (#8509) 2021-03-08 12:20:09 -07:00
storage_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
storage.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
write_handler_test.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
write_handler.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00
write_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
write.go Switched to go-kit/log 2021-06-11 12:28:36 -04:00