bd24e2fb92
Merge pull request #9303 from LeviHarrison/backfill-return-1
...
Rules backfill: return 1 if unsuccessful
2021-09-07 18:00:42 +02:00
34e1b47968
Fixed error handling
...
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-09-06 21:55:57 -04:00
5edec40d60
promtool: Speed up checking for duplicate rules
...
Trade space for speed. Convert all rules into our temporary struct, sort
and then iterate. This is a significant when having many rules.
Signed-off-by: Holger Hans Peter Freyther <holger@moiji-mobile.com>
2021-09-06 23:10:26 +08:00
3b5257d869
Changed disabled_features to feature_flags
...
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-07-13 22:03:51 -04:00
7c125aa5fb
Promtool: Add support for compaction analysis ( #8940 )
...
* Extend promtool to support compaction analysis
This commit extends the promtool tsdb analyze command to help
troubleshoot high Prometheus disk usage. The command now plots a
distribution of how full chunks are relative to the maximum capacity of
120 samples per chunk.
Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>
* Update cmd/promtool/tsdb.go
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2021-07-02 11:08:52 +01:00
90976e7505
Promtool: Add feature flags to unit tests ( #8958 )
...
* Added feature flag support to unit tests
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Added/fixed tests
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Addressed review comments
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-30 22:43:39 +01:00
ca1896c15b
Promtool: Validate service discovery files ( #8950 )
...
* Check SD files in promtool
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-29 17:32:59 +02:00
fd6c852567
promtool: backfill: allow configuring block duration ( #8919 )
...
* promtool: backfill: allow configuring block duration
When backfilling large amounts of data across long periods of time, it
may in certain circumstances be useful to use a longer block duration to
increase the efficiency and speed of the backfilling process. This patch
adds a flag --block-duration-power to allow a user to choose the power N
where the block duration is 2^(N+1)h.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
* promtool: use sub-tests in backfill testing
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
* backfill: add messages to tests for clarity
When someone new breaks a test, seeing "expected: false, got: true" is
really not useful. A nice message helps here.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
* backfill: test long block durations
A test that uses a long block duration to write bigger blocks is added.
The check to make sure all blocks are the default duration is removed.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2021-06-29 14:53:38 +05:30
ba76bceb6b
Merge pull request #8917 from stevekuznetsov/skuznets/silence-backfill
...
promtool: backfill: allow silencing output
2021-06-14 23:27:18 +02:00
b5f6f8fb36
Switched to go-kit/log
...
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2021-06-11 12:28:36 -04:00
ee771a2a66
promtool: backfill: allow silencing output
...
When using the backfill command to add data to an ephemeral/test
Prometheus instance, it is not important to see which data was added as
it is often generated ahead of time and mostly irrelevant to the
use-case. The current approach prints information about each block that
is written, but does so in a generally inefficient and costly manner.
This patch adds a `--quiet` flag that allows a user to opt out of this
behavior.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2021-06-10 15:31:16 -07:00
731545ad34
Add documentation for recording rule backfiller ( #8674 )
...
* add docs for rule backfiller
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-04-01 22:38:00 +02:00
e635ca834b
Add environment variable expansion in external label values
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-30 01:36:28 +02:00
9549a15c6f
Merge pull request #7675 from JessicaGreben/jg/11-retroactive-rule-eval
...
Add rule importer to backfill
2021-03-29 19:09:21 +02:00
f0bccba1c3
Update Go modules for 2.26 ( #8636 )
...
* Update Go modules for 2.26
Bump all Go modules to the latest upstream.
Signed-off-by: Ben Kochie <superq@gmail.com>
* Fix promtool for new client_golang
LabelValues now requires a list of string matchers.
Signed-off-by: Ben Kochie <superq@gmail.com>
2021-03-24 09:41:12 +00:00
8de4da3716
add changes per comments, fix tests
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-20 12:38:30 -07:00
e3a8132bb3
fix block alignment, add sample alignment
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-15 12:44:58 -07:00
7c26642460
add block alignment and write in 2 hr blocks
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-03-14 10:10:55 -07:00
b1ac4a45e6
Add num scrapes as tsdb write benchmark command flag
...
By default same value that was hardcoded is used, but with the
new flag added the number of scrapes can be increased to any value.
Signed-off-by: Pau Freixes <pfreixes@gmail.com>
2021-03-10 11:17:07 +01:00
537c0aff49
Prometheus and Promtool binaries now print help and usage to stdout ( #8542 )
...
Signed-off-by: ArthurSens <arthursens2005@gmail.com>
2021-02-25 19:52:34 +01:00
78e84aed89
resolve merge conflict
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-02-24 09:47:29 -08:00
ab41de68b4
Print details of API errors
...
Signed-off-by: Łukasz Hryniuk <code@hryniuk.pl>
2021-02-15 23:42:06 +01:00
c1f8bd9944
Merge pull request #8432 from roidelapluie/backfillpanic
...
backfill: move checkErr before we close the mmaped file
2021-02-03 16:32:35 +01:00
9334269f2b
backfill: move checkErr before we close the mmaped file
...
When printing the error, we still need access to the mmapped byte array
of the file. Therefore, we make sure that we run it before closing the
file.
I could have done something more complex like a defer, or not closing
the file, knowing that we would exit the program anyway. However, I
think that in case we extend this in the future, or this is copy/paster
elsewhere, we should continue closing the file. As it is small enough, I
went for the solution to call the function 3 times instead of playing
with a defer.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-02-01 21:18:42 +01:00
4a1f2c097e
Typo on plural in checkRules/checkDuplicates
...
Signed-off-by: Jeremy Albinet <jalbinet@synthesio.com>
2021-02-01 15:43:05 +01:00
ac2626757c
Update exporter-toolkit to 0.5.0
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-01-13 21:49:54 +01:00
003d6451fc
Promtool: add web config validation
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-29 16:55:29 +01:00
651b57b9ab
Merge branch 'backfillhr' of git://github.com/roidelapluie/prometheus into review
2020-12-28 16:18:00 +01:00
53480c168d
Backfill: print created blocks only, add human-readable option
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-23 20:42:30 +01:00
dca6954b0a
promtool: Add URL scheme when not provided ( #7956 )
...
Signed-off-by: AdaephonBen <ma18btech11011@iith.ac.in>
2020-12-23 19:52:04 +01:00
cec3515fa3
fix linter
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-30 08:17:51 -08:00
2e9946e4d7
add test
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-28 07:58:33 -08:00
ac06d0a657
merge master/resolve conflict
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-26 08:43:07 -08:00
ee85c22adb
flush samples to disk every 5k samples
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-26 08:30:06 -08:00
b317b6ab9c
Backfill from OpenMetrics format ( #8084 )
...
* get parser working
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* import file created
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Find min and max ts
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make two passes over file and write to tsdb
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* print error messages
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Max and Min initializer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Start with unit tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reset file read
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* align blocks to two hour range
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add cleanup test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove .ds_store
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add license to import_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Circle CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Refactor code
Move backfill from tsdb to promtool directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix gitignore
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove panic
Rename ContenType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* adjust mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return statement
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go modules
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added unit test for backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix file handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close DB
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Error Handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* inline err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix command line flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add spaces before func
fix pointers
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add defer'd calls
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* move openmetrics.go content to backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changed args to flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add tests for wrong OM files
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added additional tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add comment to warn of func reuse
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Make input required in main.go
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer blockwriter close
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove contentType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove defer from backfilltest
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix defer remove in backfill_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changes to fix CI errors
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go.mod
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change package name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* assert->require
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix format
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix createblock
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* check err for anon func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* update comment
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix for the Flush Bug
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix formatting, comments, names
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* cleanup
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor test to take care of multiple samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* I dont know what I fixed
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests, add test description, print blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* commit after 5000 samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reviews part 1
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Series Count
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove extra func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make timestamp into sec
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Reviews 2
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add Todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fixes
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fixes reviews
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* =0
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove backfill.om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add global err var, remove stuff
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change var name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* sampleLimit pass as parameter
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add test when number of samples greater than batch size
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Change name of batchsize
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* revert export
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add comment, remove newline,consistent err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Modify comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* db.Querier
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add sanity check , get maxt and mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* ci error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* comment change
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* NoError
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-11-26 10:37:06 +05:30
5dd3577424
change name of promtool subcommand to create-blocks-from
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-22 15:05:02 -08:00
75654715d3
fix panics
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-11-01 07:54:04 -08:00
6980bcf671
unexport backfiller
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-10-31 06:40:56 -07:00
3ed6457dd4
use blockwriter, rm multiwriter code
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-10-31 06:32:07 -07:00
4e5b1722b3
Move away from testutil, refactor imports ( #8087 )
...
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
36ac0b68f1
merge master, fix conflicts
2020-10-17 08:20:21 -07:00
2e526cf2a7
add output dir parameter
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-09-13 08:38:32 -07:00
dfa510086b
add alignment, mv rule importer to promtool dir, add queryRange
...
Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2020-09-13 08:07:59 -07:00
442b3364d7
Promtool: add evaluation time to instant query ( #7829 )
...
* Promtool: add evaluation time to instant query
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
* Apply suggestion
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-25 11:32:25 +01:00
4e6a94a27d
Invert service discovery dependencies ( #7701 )
...
This also fixes a bug in query_log_file, which now is relative to the config file like all other paths.
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
2020-08-20 13:48:26 +01:00
7504b5ce7c
add rule importer with tsdb block writer
...
Signed-off-by: jessicagreben <Jessica.greben1+github@gmail.com>
2020-07-27 07:44:49 -07:00
50c261502e
add tsdb cmds into promtool ( #6088 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
update tsdb cli in makefile and promu
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove building tsdb bin
Signed-off-by: yeya24 <yb532204897@gmail.com>
remove useless func
Signed-off-by: yeya24 <yb532204897@gmail.com>
refactor analyzeBlock
Signed-off-by: yeya24 <yb532204897@gmail.com>
Fix Makefile
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2020-07-23 19:35:50 +01:00
e6ea798c32
promtool range query should exit when fail to parse time ( #7505 )
...
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-07-16 23:53:04 +01:00
797e48c1a3
support time range in promtool query labels
...
Updated prometheus/client_golang and json-iterator/go
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-07-03 11:29:39 -04:00
c7e82274c6
replace github.com/prometheus/prometheus/testutil/promlint by github.com/prometheus/client_golang/prometheus/testutil/promlint from our codebase ( #7209 )
...
Signed-off-by: RainbowMango <renhongcai@huawei.com>
2020-05-07 11:34:39 +01:00
Julien Pivotto
Levi Harrison
Holger Hans Peter Freyther
Filip Petkovski
Steve Kuznetsov
Jess G
Björn Rabenstein
Ben Kochie
Pau Freixes
Arthur Silva Sens
Łukasz Hryniuk
Jeremy Albinet
AdaephonBen
Atibhi Agrawal
Andy Bursavich
jessicagreben
Ben Ye
Hongcai Ren