prometheus/storage/remote
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
..
chunked_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
chunked.go Fix some warnings: 'redundant type from array, slice, or map composite literal' (#7109) 2020-04-15 11:17:41 +01:00
client_test.go Adds support to configure retry on Rate-Limiting from remote-write config. 2021-02-16 14:52:49 +05:30
client.go Upgrade prometheus/common to v0.21.0 2021-04-21 12:19:16 +02:00
codec_test.go Various enhancements and refactorings for remote write receiver: 2021-02-08 20:41:23 +00:00
codec.go Various enhancements and refactorings for remote write receiver: 2021-02-08 20:41:23 +00:00
ewma.go Replace sync/atomic with uber-go/atomic (#7683) 2020-07-30 13:15:42 +05:30
intern_test.go Testify: move to require (#8122) 2020-10-29 09:43:23 +00:00
intern.go Move away from testutil, refactor imports (#8087) 2020-10-22 11:00:08 +02:00
max_timestamp.go Remote: Do not collect non-initialized timestamp metrics (#8060) 2020-10-15 23:53:59 +02:00
metadata_watcher_test.go Fix some gofmt warnings (#8743) 2021-04-22 08:43:30 -06:00
metadata_watcher.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
queue_manager_test.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
queue_manager.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00
read_handler_test.go Move remote read handler to remote package. (#8536) 2021-02-26 16:43:19 +00:00
read_handler.go Move remote read handler to remote package. (#8536) 2021-02-26 16:43:19 +00:00
read_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
read.go Add matchers to LabelValues() call (#8400) 2021-02-09 23:08:35 +05:30
sigv4_test.go Contribute grafana/agent sigv4 code (#8509) 2021-03-08 12:20:09 -07:00
sigv4.go Contribute grafana/agent sigv4 code (#8509) 2021-03-08 12:20:09 -07:00
storage_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
storage.go Custom headers on remote-read and refactor implementation to roundtripper. 2021-02-26 17:20:29 +05:30
write_handler_test.go Add circular in-memory exemplars storage (#6635) 2021-03-16 15:17:45 +05:30
write_handler.go Move remote read handler to remote package. (#8536) 2021-02-26 16:43:19 +00:00
write_test.go Allow metric metadata to be propagated via Remote Write. (#6815) 2020-11-19 20:53:03 +05:30
write.go Add Exemplar Remote Write support (#8296) 2021-05-06 13:53:52 -07:00