* Refactor: extract function to make scrapeLoop for testing
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Add benchmarks for ScrapeLoopAppend
For Prometheus and OpenMetrics
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Create less garbage when parsing metrics
Exemplar escapes to heap due to being passed through text-parser
interface, but we can reduce the impact by hoisting it out of the loop
and resetting it after every use.
(Note the cost was paid on every line even when exemplars were disabled)
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
* Create less garbage when parsing OpenMetrics
After calling parseLVals() we always append the return value, so pass in
what we want to append it to and save garbage.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
While implementing a different feature, I found that Labels.Get() was
performing a linear search. I wondered whether it would perform any
better with a binary search approach, and wrote a benchmark: the answer
is that it's probably doesn't worth it, so I just decided to leave the
benchmark and the results for the next reader.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use global string map for MatchType.String()
We were unnecessarily creating a new map for each call of String().
This is a 10x improvement in MatchType.String() performance in time,
from 53ns to 4ns on my i7 laptop, and I guess that this method is being
called quite often so why throw out the resources.
I was surprised that benchmark says that there are no allocations made
in the old version.
I also tries using `//go:generate stringer` and the result is even
better, at about 2.8ns, but having to keep the generated code updated
isn't worth the change (at least it's bigger than a small change I was
intended to do)
Benchmark comparison:
name \ time/op old global_map stringer
MatchType_String 53.6ns ± 1% 4.1ns ± 1% 2.8ns ± 1%
name \ alloc/op old global_map stringer
MatchType_String 0.00B 0.00B 0.00B
name \ allocs/op old global_map stringer
MatchType_String 0.00 0.00 0.00
Old benchmark:
goos: darwin
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMatchType_String 21766578 54.36 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21742339 53.28 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21985470 53.37 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 21676282 53.50 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 22075573 53.33 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/prometheus/prometheus/pkg/labels 6.252s
New with global map:
goos: darwin
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMatchType_String 283412692 4.129 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 294859941 4.091 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 295750158 4.113 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 282827982 4.072 ns/op 0 B/op 0 allocs/op
BenchmarkMatchType_String 292942393 4.047 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/prometheus/prometheus/pkg/labels 8.238s
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use array instead of map
Since MatchType is an iota type, we can safely use an array here.
This solution is even better:
name \ time/op old global_map stringer array
MatchType_String 53.6ns ± 1% 4.1ns ± 1% 2.8ns ± 1% 1.0ns ± 1%
name \ alloc/op old global_map stringer array
MatchType_String 0.00B 0.00B 0.00B 0.00B
name \ allocs/op old global_map stringer array
MatchType_String 0.00 0.00 0.00 0.00
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Benchmark all MatchType values
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
* Use constants for limits
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Write exemplars to the WAL and send them over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Update example for exemplars, print data in a more obvious format.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add metrics for remote write of exemplars.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Fix incorrect slices passed to send in remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* We need to unregister the new metrics.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Order of exemplar append vs write exemplar to WAL needs to change.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Condense sample/exemplar delivery tests to parameterized sub-tests
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename test methods for clarity now that they also handle exemplars
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Rename counter variable. Fix instances where metrics were not updated correctly
Signed-off-by: Martin Disibio <mdisibio@gmail.com>
* Add exemplars to LoadWAL benchmark
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* last exemplars timestamp metric needs to convert value to seconds with
ms precision
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Process exemplar records in a separate go routine when loading the WAL.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Regenerate types proto with comments, update protoc version again.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Put remote write of exemplars behind a feature flag.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address some of Ganesh's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Move exemplar remote write feature flag to a config file field.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address Bartek's review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address more reivew comments from Ganesh.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Add exemplar total label length check.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address a few last review comments
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
Currently hashmod hashes all the labels, even if there is a newline. To
ensure it still works in the future, let's add a test.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
This commit adds `@ <timestamp>` modifier as per this design doc: https://docs.google.com/document/d/1uSbD3T2beM-iX4-Hp7V074bzBRiRNlqUdcWP6JTDQSs/edit.
An example query:
```
rate(process_cpu_seconds_total[1m])
and
topk(7, rate(process_cpu_seconds_total[1h] @ 1234))
```
which ranks based on last 1h rate and w.r.t. unix timestamp 1234 but actually plots the 1m rate.
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Testify: move to require
Moving testify to require to fail tests early in case of errors.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* More moves
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Refactor test assertions
This pull request gets rid of assert.True where possible to use
fine-grained assertions.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
This was already fixed by #8013, but add a test case anyway
in case the regexp engine changes in future.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
* cleanup tempfiles for web_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* cleanup tempfiles for api_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* cleanup tempfiles for file_test
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
* Optimized label regex matcher with literal prefix and/or suffix
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Added license
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Added more tests cases with newlines
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Restored deleted test
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Use go1.14 new hash/maphash to hash both RHS and LHS instead of XOR'ing
which has been resulting in hash collisions.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Refactor engine labelset signature generation, just use labels.Labels
instead of hashes.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments; function comments + store result of
lhs.String+rhs.String as key.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Replace all signatureFunc usage with signatureFuncString.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Make optimizations to labels String function and generation of rhs+lhs
as string in resultMetric.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use separate string functions that don't use strconv just for engine
maps.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use a byte invalid separator instead of quoting and have a buffer
attached to EvalNodeHelper instead of using a global pool in the labels
package.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address review comments.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Address more review comments, labels has a function that now builds a
byte slice without turning it into a string.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* Use two different non-ascii hex codes as byte separators between labels
and between sets of labels when building bytes of a Labels struct.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
* We only need the 2nd byte invalid sep. at the beginning of a
labels.Bytes
Signed-off-by: Callum Styan <callumstyan@gmail.com>
time.Unix attaches the local timezone, which can then
leak out (e.g. in the alert json). While this is harmless,
we should be consistent.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
* tsdb: don't allow ingesting empty labelsets
When we ingest an empty labelset in the head, further blocks can not be
compacted, with the error:
```
level=error ts=2020-02-27T21:26:58.379Z caller=db.go:659 component=tsdb
msg="compaction failed" err="persist head block: write compaction:
add series: out-of-order series added with label set \"{}\" / prev:
\"{}\""
```
We should therefore reject those invalid empty labelsets upfront.
This can be reproduced with the following:
```
cat << END > prometheus.yml
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 1s
basic_auth:
username: test
password: test
metric_relabel_configs:
- regex: ".*"
action: labeldrop
static_configs:
- targets:
- 127.0.1.1:9090
END
./prometheus --storage.tsdb.min-block-duration=1m
```
And wait a few minutes.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Howie
Björn Rabenstein
Levi Harrison
Paweł Szulik
George Robinson
Bryan Boreham
DrAuYueng
SuperQ
Oleg Zaytsev
cw9
Michal Wasilewski
Callum Styan
Julien Pivotto
Ganesh Vernekar
Bartlomiej Plotka
Brian Brazil
Li Zhijian
Max Neverov
Guangwen Feng
Marco Pracucci
Mikael Johansson
Chris Marchbanks
Hu Shuai
ZouYu
johncming
Tobias Guggenmos