38d32e0686
Don't sort postings if we only have one block.
...
Sorting the heads postings can be quite slow.
We only need sorted series when merging with another
querier, so only sort then.
This will make big queries that only touch the head faster,
though queries that touch both the head and a block will still
be the same speed. This probably won't help much with graphing
unless the range is under an hour, however it should make most
recording rules faster.
Add gaurantee that remote read streaming produces sorted series.
PromQL benchmarks for histograms show only 2-3% improvement, but
they're only over 1k series.
benchmark old ns/op new ns/op delta
BenchmarkQuerierSelect/Head/1of1000000-4 1375486282 507657736 -63.09%
BenchmarkQuerierSelect/Head/10of1000000-4 1387859004 507769850 -63.41%
BenchmarkQuerierSelect/Head/100of1000000-4 1387087935 506029110 -63.52%
BenchmarkQuerierSelect/Head/1000of1000000-4 1386869064 504521986 -63.62%
BenchmarkQuerierSelect/Head/10000of1000000-4 1386213685 505210422 -63.55%
BenchmarkQuerierSelect/Head/100000of1000000-4 1392754988 529842406 -61.96%
BenchmarkQuerierSelect/Head/1000000of1000000-4 1569414722 725059506 -53.80%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 1381019902 1370495863 -0.76%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 1375696209 1366789468 -0.65%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 1386009422 1364519297 -1.55%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 1377700532 1364486191 -0.96%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 1383539536 1369545314 -1.01%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 1410089163 1394731339 -1.09%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 1634744148 1581554956 -3.25%
BenchmarkQuerierSelect/Block/1of1000000-4 881741242 879839470 -0.22%
BenchmarkQuerierSelect/Block/10of1000000-4 880381562 882846038 +0.28%
BenchmarkQuerierSelect/Block/100of1000000-4 887519357 881016916 -0.73%
BenchmarkQuerierSelect/Block/1000of1000000-4 902194205 883433524 -2.08%
BenchmarkQuerierSelect/Block/10000of1000000-4 892321964 885130170 -0.81%
BenchmarkQuerierSelect/Block/100000of1000000-4 938604466 933527150 -0.54%
BenchmarkQuerierSelect/Block/1000000of1000000-4 1313510845 1295881124 -1.34%
benchmark old allocs new allocs delta
BenchmarkQuerierSelect/Head/1of1000000-4 4000056 4000018 -0.00%
BenchmarkQuerierSelect/Head/10of1000000-4 4000074 4000036 -0.00%
BenchmarkQuerierSelect/Head/100of1000000-4 4000254 4000216 -0.00%
BenchmarkQuerierSelect/Head/1000of1000000-4 4002054 4002016 -0.00%
BenchmarkQuerierSelect/Head/10000of1000000-4 4020054 4020016 -0.00%
BenchmarkQuerierSelect/Head/100000of1000000-4 4200054 4200016 -0.00%
BenchmarkQuerierSelect/Head/1000000of1000000-4 6000054 6000016 -0.00%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 4000071 4000071 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 4000089 4000089 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 4000269 4000269 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 4002069 4002069 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 4020069 4020069 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 4200069 4200069 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 6000069 6000069 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 6000023 6000022 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 6000059 6000058 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 6000419 6000418 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 6004019 6004018 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 6040019 6040018 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 6400019 6400018 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 10000020 10000019 -0.00%
benchmark old bytes new bytes delta
BenchmarkQuerierSelect/Head/1of1000000-4 229192200 176001176 -23.21%
BenchmarkQuerierSelect/Head/10of1000000-4 229193352 176002328 -23.21%
BenchmarkQuerierSelect/Head/100of1000000-4 229204872 176013848 -23.21%
BenchmarkQuerierSelect/Head/1000of1000000-4 229320072 176129048 -23.20%
BenchmarkQuerierSelect/Head/10000of1000000-4 230472072 177281048 -23.08%
BenchmarkQuerierSelect/Head/100000of1000000-4 241992072 188801048 -21.98%
BenchmarkQuerierSelect/Head/1000000of1000000-4 357192072 304001048 -14.89%
BenchmarkQuerierSelect/SortedHead/1of1000000-4 229193928 229193928 +0.00%
BenchmarkQuerierSelect/SortedHead/10of1000000-4 229195080 229195080 +0.00%
BenchmarkQuerierSelect/SortedHead/100of1000000-4 229206600 229206600 +0.00%
BenchmarkQuerierSelect/SortedHead/1000of1000000-4 229321800 229321800 +0.00%
BenchmarkQuerierSelect/SortedHead/10000of1000000-4 230473800 230473800 +0.00%
BenchmarkQuerierSelect/SortedHead/100000of1000000-4 241993800 241993800 +0.00%
BenchmarkQuerierSelect/SortedHead/1000000of1000000-4 357193800 357193800 +0.00%
BenchmarkQuerierSelect/Block/1of1000000-4 227201516 227201500 -0.00%
BenchmarkQuerierSelect/Block/10of1000000-4 227202924 227202908 -0.00%
BenchmarkQuerierSelect/Block/100of1000000-4 227217036 227217020 -0.00%
BenchmarkQuerierSelect/Block/1000of1000000-4 227358156 227358140 -0.00%
BenchmarkQuerierSelect/Block/10000of1000000-4 228769356 228769340 -0.00%
BenchmarkQuerierSelect/Block/100000of1000000-4 242881356 242881340 -0.00%
BenchmarkQuerierSelect/Block/1000000of1000000-4 384001616 384001600 -0.00%
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-01-28 09:14:56 +00:00
de0a772b8e
Port tsdb to use pkg/labels. ( #6326 )
...
* Port tsdb to use pkg/labels.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
* Get tests passing.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
* Remove useless cast.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
* Appease linters.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
* Fix review comments
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-11-18 11:53:33 -08:00
5ecef3542d
Cleanup after merging tsdb into prometheus
...
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-08-13 14:04:14 +05:30
06f1ba73eb
Provide flag to compress the tsdb WAL
...
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-07-03 08:03:29 -06:00
91d7175eaa
Add storage.Warnings to LabelValues and LabelNames ( #5673 )
...
Fixes #5661
Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2019-06-17 08:31:17 +01:00
1684dc750a
updated tsdb to 0.6.0 ( #5292 )
...
* updated tsdb to 0.6.0
as part of the update also added the new storage.tsdb.allow-overlapping-blocks flag and mark it as experimental.
2019-03-04 21:42:45 +02:00
384cba1211
Add flag for size based retention ( #5109 )
...
* Add flag for size based retention
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Deprecate the old retention flag for a new one.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Add ability to take a suffix for size flag
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Address feedback
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2019-01-18 19:18:36 +05:30
3bd41cc92c
Udpate tsdb to 0.4 ( #5110 )
...
* update tsdb to v0.4.0
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* remove unused struct field
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2019-01-18 16:32:14 +05:30
5ddde1965b
tune the "Wal segment size" with a flag ( #5029 )
...
Add WALSegmentSize as an option, and the corresponding flag "storage.tsdb.wal-segment-size" to tune the max size of wal segment files.
The addressed base problem is to reduce the disk space used by wal segment files : on a raspberry pi, for instance, we often want to reduce write load of the sd card, then, the wal directory is mounted on a memory (space limited) partition.
the default value of the segment max file size, pushed the size of directory to 128 MB for each segment , which is too much ram consumption on a rasp.
the initial discussion is at https://github.com/prometheus/tsdb/pull/450
2019-01-03 17:13:21 +03:00
6e08029b56
Move err to be the last return value from storage.Select. ( #5054 )
...
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-01-02 11:10:13 +00:00
f0e9196dca
Return warnings on a remote read fail ( #4832 )
...
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
2018-11-30 14:27:12 +00:00
bd100182b2
added tsdb/head mint maxt metrics ( #4888 )
...
added the head metrics with the correct suffix.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
2018-11-21 12:57:32 +02:00
ca93fd544b
/api/v1/labels endpoint for getting all label names ( #4835 )
...
* vendor: update tsdb
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* /api/v1/labels endpoint
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* regex matchers for API
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add docs
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Matchers behaving as OR
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Removed the matchers
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* vendor: update tsdb using go mod
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* vendor update: tsdb
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Added LabelNames() to storage.Querier
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Test for api.labelNames
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Nits
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2018-11-19 15:51:14 +05:30
ec8e4d8a7c
all: remove unnecessary type conversions ( #3992 )
...
excep promql due to not to create conflict with #3966 .
2018-03-21 09:25:22 +00:00
7ccd4b39b8
*: implement query params
...
This adds a parameter to the storage selection interface which allows
query engine(s) to pass information about the operations surrounding a
data selection.
This can for example be used by remote storage backends to infer the
correct downsampling aggregates that need to be provided.
2018-02-13 12:17:22 +01:00
83cd270ea4
*: adapt to storage interface changes
2017-11-23 19:05:04 +01:00
30e777d10d
tsdb: default too small max block duration
2017-10-30 12:09:56 +01:00
bf56ad4233
Merge branch 'master' into master
2017-10-26 09:06:12 +02:00
c4c3205d76
storage/tsdb: check that max block duration is larger than min
...
If the user accidentally sets the max block duration smaller than the min,
the current error is not informative. This change just performs the check
earlier and improves the error message.
2017-10-25 19:24:49 -05:00
fc4406201e
Tsdb StartTime : Use a simplier way to compute StartTime
2017-10-25 17:41:00 +02:00
a57ea79660
Close index reader properly
2017-10-23 21:59:18 +02:00
bf4a279a91
Remote storage reads based on oldest timestamp in primary storage ( #3129 )
...
Currently all read queries are simply pushed to remote read clients.
This is fine, except for remote storage for wich it unefficient and
make query slower even if remote read is unnecessary.
So we need instead to compare the oldest timestamp in primary/local
storage with the query range lower boundary. If the oldest timestamp
is older than the mint parameter, then there is no need for remote read.
This is an optionnal behavior per remote read client.
Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>
2017-10-18 12:08:14 +01:00
721050c6cb
Update prometheus/tsdb dependency
2017-10-16 15:36:25 +02:00
33c1171b9c
Don't add anchoring to exported `Value` matcher field
...
Instead, just make the anchoring part of the internal regex. This helps because
some users will want to read back the `Value` field and expect it to be the
same as the input value (e.g. some tests in Cortex), or use the value in
another context which is already expected to add its own anchoring, leading to
superfluous double anchoring (such as when we translate matchers into remote
read request matchers).
2017-10-10 10:10:21 -07:00
f7e8348a88
Re-add contexts to storage.Storage.Querier() ( #3230 )
...
* Re-add contexts to storage.Storage.Querier()
These are needed when replacing the storage by a multi-tenant
implementation where the tenant is stored in the context.
The 1.x query interfaces already had contexts, but they got lost in 2.x.
* Convert promql.Engine to use native contexts
2017-10-04 21:04:15 +02:00
7b02bfee0a
web: start web handler while TSDB is starting up
2017-09-20 15:03:19 +02:00
d21f149745
*: migrate to go-kit/log
2017-09-08 22:01:51 +05:30
0efecea6d4
Adapt storage APIs to uint64 references
2017-09-07 14:14:41 +02:00
0c81d5f719
storage: instantiate correct block ranges
2017-08-24 12:36:07 +02:00
2037778d14
vendor: update TSDB
2017-08-10 14:51:02 +02:00
db8128ceeb
Add label set as first parameter to AddFast, ingored by TSDB adapter.
2017-07-12 15:20:12 +01:00
16464c3a33
Merge pull request #2910 from prometheus/adminapi
...
Admin API
2017-07-11 17:15:49 +02:00
ccf9e62972
*: add admin grpc API
2017-07-10 09:14:14 +02:00
243419c007
Return tsdb.ErrOutOfBounds as storage.ErrOutOfBounds
...
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-07-06 14:18:31 +02:00
d407bd150c
Consolidate the duration params in CLI
...
* All CLI params moved to model.Duration
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-06-16 20:20:57 +05:30
d289dc55c3
storage: update TSDB
2017-05-22 11:53:08 +02:00
9b175d48cb
Add flag to disable TSDB lock file
2017-05-09 12:56:51 +02:00
5c9a6ce747
Add license to files.
...
This should fix CI for dev-2.0.
2017-04-19 13:46:22 +01:00
cfb2a7f1d5
vendor: sync organisation migration of tsdb
2017-04-04 11:33:51 +02:00
b586781283
*: update tsdb vendoring and add retention flag
2017-03-17 16:06:04 +01:00
8a8eb12985
storage/tsdb: don't use partitioned DB.
2017-03-07 11:51:30 +01:00
4397b4d508
*: pass Prometheus registry into storage
2017-02-28 09:33:14 +01:00
ea3ba338dd
main: add flags for new storage
2017-02-05 18:22:06 +01:00
5772f1a7ba
retrieval/storage: adapt to new interface
...
This simplifies the interface to two add methods for
appends with labels or faster reference numbers.
2017-02-02 13:05:46 +01:00
c691895a0f
retrieval: cache series references, use pkg/textparse
...
With this change the scraping caches series references and only
allocates label sets if it has to retrieve a new reference.
pkg/textparse is used to do the conditional parsing and reduce
allocations from 900B/sample to 0 in the general case.
2017-01-16 12:03:57 +01:00
ad9bc62e4c
storage: extend appender and adapt it
2017-01-13 14:48:01 +01:00
304cae9928
tsdb: Use PartitionedDB constructor
2017-01-06 12:34:54 +01:00
bc20d93f0a
storage: rename iterator value getters to At()
2017-01-02 13:33:37 +01:00
f8fc1f5bb2
*: migrate ingestion to new batch Appender
2016-12-29 11:03:56 +01:00
71fe0c58a8
promql: misc fixes
2016-12-28 11:32:15 +01:00
Brian Brazil
Tom Wilkie
Ganesh Vernekar
Chris Marchbanks
Thomas Jackson
Krasi Georgiev
Goutham Veeramachaneni
glutamatt
mknapphrt
ferhat elmas
Fabian Reinartz
Paul Gier
Thibault Chataigner
Thibault Chataigner
Tobias Schmidt
Julius Volz
Tom Wilkie
Goutham Veeramachaneni