RepoMirrors/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus synced 2024-12-25 07:52:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
8,562 Commits 243 Branches 444 Tags 653 MiB
7c26642460
Commit Graph

455 Commits

Author SHA1 Message Date
jessicagreben
7c26642460 add block alignment and write in 2 hr blocks 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2021-03-14 10:10:55 -07:00
jessicagreben
9fc53b7edf fix appender.Add -> appender.Append 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2021-03-01 05:49:49 -08:00
jessicagreben
78e84aed89 resolve merge conflict 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2021-02-24 09:47:29 -08:00
jessicagreben
f2db9dc722 add multi rule integration tests 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2021-02-24 09:42:31 -08:00
Julien Pivotto
8c8de46003
Merge pull request #8036 from dgl/promtool-alert-err 
promtool: Don't end alert tests early, in some failure situations
 2021-02-20 22:35:00 +01:00
Tom Wilkie
7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489) 
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
 2021-02-18 17:37:00 +05:30
Julien Pivotto
1fac1c783b
Merge pull request #8504 from rbauduin/require_alertname 
promtool: alert_rule_test items require alertname
 2021-02-17 22:07:52 +01:00
Julien Pivotto
2d172d0896
Merge pull request #8508 from prometheus/release-2.25 
Merge back release 2.25
 2021-02-17 16:26:34 +01:00
Raphael Bauduin
a7d64cad21 promtool: alert_rule_test items require alertname 
Accepting alert_rule_test without alertname is confusing as it will
always pass with empty exp_alerts, and never with non-empty exp_alerts.

Signed-off-by: Raphael Bauduin <raphael.bauduin@tessares.net>
 2021-02-17 16:23:12 +01:00
Ganesh Vernekar
c4536fa28c
Increase block writer size for backfilling 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2021-02-17 15:45:41 +05:30
Julien Pivotto
a419b75abd
Merge pull request #8485 from hryniuk/promtool-query-errors-details 
Print details of API errors received by promtool
 2021-02-16 22:47:08 +01:00
Łukasz Hryniuk
ab41de68b4 Print details of API errors 
Signed-off-by: Łukasz Hryniuk <code@hryniuk.pl>
 2021-02-15 23:42:06 +01:00
David Leadbeater
3e30f72af1 promtool: Add more negative alert tests 
Signed-off-by: David Leadbeater <dgl@dgl.cx>
 2021-02-15 17:00:49 +00:00
Julien Pivotto
e29b47b39e
Merge pull request #8440 from mishamo/master 
Add optional name property to testgroup for better test failure output
 2021-02-09 21:23:24 +01:00
misha
1c3e7b4241 Use strings.Builder for neater error formatting 
Signed-off-by: misha <DL-OTTCloudPlatform-Nova@bskyb.internal>
 2021-02-09 15:00:26 +00:00
Tom Wilkie
d479151f1f Various enhancements and refactorings for remote write receiver: 
- Remove unrelated changes
- Refactor code out of the API module - that is already getting pretty crowded.
- Don't track reference for AddFast in remote write.  This has the potential to consume unlimited server-side memory if a malicious client pushes a different label set for every series.  For now, its easier and safer to always use the 'slow' path.
- Return 400 on out of order samples.
- Use remote.DecodeWriteRequest in the remote write adapters.
- Put this behing the 'remote-write-server' feature flag
- Add some (very) basic docs.
- Used named return & add test for commit error propagation

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
 2021-02-08 20:41:23 +00:00
fuling
72475b8a0c [ENHANCEMENT] remote storage:Add default api implementation of remote write 
Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>
 2021-02-07 18:12:48 +00:00
misha
c2c5aeb16b Add optional name property to testgroup for better test failure output 
Signed-off-by: misha <DL-OTTCloudPlatform-Nova@bskyb.internal>
 2021-02-04 10:07:22 +00:00
Julien Pivotto
c1f8bd9944
Merge pull request #8432 from roidelapluie/backfillpanic 
backfill: move checkErr before we close the mmaped file
 2021-02-03 16:32:35 +01:00
Julien Pivotto
9334269f2b backfill: move checkErr before we close the mmaped file 
When printing the error, we still need access to the mmapped byte array
of the file. Therefore, we make sure that we run it before closing the
file.

I could have done something more complex like a defer, or not closing
the file, knowing that we would exit the program anyway. However, I
think that in case we extend this in the future, or this is copy/paster
elsewhere, we should continue closing the file. As it is small enough, I
went for the solution to call the function 3 times instead of playing
with a defer.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-02-01 21:18:42 +01:00
Jeremy Albinet
4a1f2c097e Typo on plural in checkRules/checkDuplicates 
Signed-off-by: Jeremy Albinet <jalbinet@synthesio.com>
 2021-02-01 15:43:05 +01:00
Julien Pivotto
2316062d4e Deprecate --alertmanager.timeout 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-01-25 12:36:13 +01:00
Ganesh Vernekar
9199fcb8d1
'@ <timestamp>' modifier (#8121) 
This commit adds `@ <timestamp>` modifier as per this design doc: https://docs.google.com/document/d/1uSbD3T2beM-iX4-Hp7V074bzBRiRNlqUdcWP6JTDQSs/edit.

An example query:

```
rate(process_cpu_seconds_total[1m]) 
  and
topk(7, rate(process_cpu_seconds_total[1h] @ 1234))
```

which ranks based on last 1h rate and w.r.t. unix timestamp 1234 but actually plots the 1m rate.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2021-01-20 16:27:39 +05:30
Julien Pivotto
ac2626757c Update exporter-toolkit to 0.5.0 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2021-01-13 21:49:54 +01:00
Guangwen Feng
2df1a482da
Fix misspelled word in comment (#8348) 
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
 2021-01-07 10:01:08 +00:00
Julien Pivotto
bc9f9ee3aa
Backfilling: fast-path for non-consecutive blocks (#8324) 
* Backfilling: optimize for non-consecutive blocks

When you have missing data for > 2 hours, you spend a lot of time
re-reading the complete file. It is not optimal.

This introduces a fastpath for this scenario.

Next, we do parse the metric even when we know we will not use it, based
on its timestamp. This only computes the metric when we know its
timestamp is right.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-30 02:06:41 +01:00
Julien Pivotto
003d6451fc Promtool: add web config validation 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-29 16:55:29 +01:00
Julien Pivotto
5b4f46a348 Add TLS and basic authentication 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-28 21:33:44 +01:00
Ben Kochie
5055dfbbe4 Listen on web early in startup 
Avoid starting up components like the TSDB if we can't bind
to the web listening port.

Signed-off-by: Ben Kochie <superq@gmail.com>
 2020-12-28 20:13:05 +01:00
beorn7
6bfa33308e promtool: Print block meta-data slightly more nicely 
I initially thought I could somehow rescue the current column layout
by recycling the tabwriter, but flushing completely blanks
it. However, by setting a minimum width of 13, we get a slightly
broader DURATION column but otherwise nice formatting, unless numbers
get really big, but that's OK, I guess.

Before:

```
BLOCK ULID                  MIN TIME                       MAX TIME                       DURATION  NUM SAMPLES  NUM CHUNKS  NUM SERIES  SIZE
01ETN0KGNP5WWK9T5QMQGBG9F1  2020-11-19 07:39:17 +0000 UTC  2020-11-19 07:44:17 +0000 UTC  5m0.001s  8            2           2           624B
01ETN0KGQSFF0AB2QDZVQG3CWC  2020-11-19 10:25:57 +0000 UTC  2020-11-19 10:30:57 +0000 UTC  5m0.001s  8  2  2  622B
01ETN0KGSW8KYP3YPG4X20P60Z  2020-11-19 13:12:37 +0000 UTC  2020-11-19 13:17:37 +0000 UTC  5m0.001s  8  2  2  625B
```

After:

```
BLOCK ULID                  MIN TIME                       MAX TIME                       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01ETN0R72SXN9A1FG732P7KFFN  2020-11-19 07:39:17 +0000 UTC  2020-11-19 07:44:17 +0000 UTC  5m0.001s     8            2            2            624B
01ETN0R74Y9AG1A1MKN4MZK7WM  2020-11-19 10:25:57 +0000 UTC  2020-11-19 10:30:57 +0000 UTC  5m0.001s     8            2            2            622B
01ETN0R76KXZ5VQECMDNES49J6  2020-11-19 13:12:37 +0000 UTC  2020-11-19 13:17:37 +0000 UTC  5m0.001s     8            2            2            625B
```

After without the `-r` flag:

```
BLOCK ULID                  MIN TIME       MAX TIME       DURATION     NUM SAMPLES  NUM CHUNKS   NUM SERIES   SIZE
01ETN0RFFJ42274NWR1GH0RTV6  1605771557000  1605771857001  5m0.001s     8            2            2            624
01ETN0RFJ1MZCHHS2SBZS8XC27  1605781557000  1605781857001  5m0.001s     8            2            2            622
01ETN0RFM98N3V4KD2DZXFGHGN  1605791557000  1605791857001  5m0.001s     8            2            2            625
```

Signed-off-by: beorn7 <beorn@grafana.com>
 2020-12-28 16:55:12 +01:00
beorn7
651b57b9ab Merge branch 'backfillhr' of git://github.com/roidelapluie/prometheus into review 2020-12-28 16:18:00 +01:00
yeya24
cedd2dbec9 create output directory before backfilling 
Signed-off-by: yeya24 <yb532204897@gmail.com>
 2020-12-24 23:36:36 -05:00
Julien Pivotto
53480c168d Backfill: print created blocks only, add human-readable option 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-23 20:42:30 +01:00
AdaephonBen
dca6954b0a
promtool: Add URL scheme when not provided (#7956) 
Signed-off-by: AdaephonBen <ma18btech11011@iith.ac.in>
 2020-12-23 19:52:04 +01:00
lzhfromustc
27a6e1e174
test: add buffer to channel to avoid goroutine leak (#8274) 
Signed-off-by: lzhfromustc <lzhfromustc@gmail.com>
 2020-12-10 09:09:21 +00:00
Julien Pivotto
7957731339 Inline defer 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-09 09:23:39 +01:00
Julien Pivotto
82b5f1d8b1 Backfill: Use mmap to reuse parser code 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-12-08 23:48:31 +01:00
jessicagreben
e32e4fcc53 fix unit test 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-30 11:02:45 -08:00
jessicagreben
cec3515fa3 fix linter 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-30 08:17:51 -08:00
jessicagreben
2e9946e4d7 add test 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-28 07:58:33 -08:00
jessicagreben
ac06d0a657 merge master/resolve conflict 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-26 08:43:07 -08:00
jessicagreben
ee85c22adb flush samples to disk every 5k samples 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-26 08:30:06 -08:00
Atibhi Agrawal
b317b6ab9c
Backfill from OpenMetrics format (#8084) 
* get parser working

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* import file created

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Find min and max ts

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make two passes over file and write to tsdb

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* print error messages

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Max and Min initializer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Start with unit tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reset file read

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* align blocks to two hour range

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add cleanup test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove .ds_store

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add license to import_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix Circle CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Refactor code
Move backfill from tsdb to promtool directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix gitignore

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove panic
Rename ContenType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* adjust mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return statement

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go modules

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added unit test for backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix file handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close DB

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Close directory

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Error Handling

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* inline err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix command line flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add spaces before func
fix pointers

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add defer'd calls

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* move openmetrics.go content to backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changed args to flags

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add tests for wrong OM files

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Added additional tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add comment to warn of func reuse

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Make input required in main.go

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer blockwriter close

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Remove contentType

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove defer from backfilltest

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix defer remove in backfill_test

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* changes to fix CI errors

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix go.mod

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change package name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* assert->require

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix format

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix createblock

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix defer

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix return

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* check err for anon func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* update comment

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix for the Flush Bug

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix formatting, comments, names

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* cleanup

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor test to take care of multiple samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* refactor tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* I dont know what I fixed

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fix tests, add test description, print blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* commit after 5000 samples

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* reviews part 1

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Series Count

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix CI

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove extra func

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* make timestamp into sec

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Reviews 2

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add Todo

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Fixes

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fixes reviews

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* =0

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove backfill.om

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add global err var, remove stuff

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* change var name

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* sampleLimit pass as parameter

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Add test when number of samples greater than batch size

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Change name of batchsize

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* revert export

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* remove

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add comment, remove newline,consistent err

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Print Blocks

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* Modify comments

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* db.Querier

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* add sanity check , get maxt and mint

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* ci error

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* comment change

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* nits

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* NoError

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>

* fix

Signed-off-by: aSquare14 <atibhi.a@gmail.com>
 2020-11-26 10:37:06 +05:30
jessicagreben
5dd3577424 change name of promtool subcommand to create-blocks-from 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-22 15:05:02 -08:00
jessicagreben
19dee0a569 add name and labels to metric, eval all rules for each block 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-22 14:24:38 -08:00
gotjosh
4eca4dffb8
Allow metric metadata to be propagated via Remote Write. (#6815) 
* Introduce a metadata watcher

Similarly to the WAL watcher, its purpose is to observe the scrape manager and pull metadata. Then, send it to a remote storage.

Signed-off-by: gotjosh <josue@grafana.com>

* Additional fixes after rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Rework samples/metadata metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Use more descriptive variable names in MetadataWatcher collect.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix issues caused during rebasing.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix missing metric add and unneeded config code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix metrics and docs

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Replace assert with require

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Bring back max_samples_per_send metric

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-11-19 20:53:03 +05:30
jessicagreben
75654715d3 fix panics 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-11-01 07:54:04 -08:00
jessicagreben
61c9a89120 use milliseconds for blocksize 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-10-31 07:11:54 -07:00
jessicagreben
6980bcf671 unexport backfiller 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-10-31 06:40:56 -07:00
jessicagreben
3ed6457dd4 use blockwriter, rm multiwriter code 
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-10-31 06:32:07 -07:00