bc9f9ee3aa
Backfilling: fast-path for non-consecutive blocks ( #8324 )
...
* Backfilling: optimize for non-consecutive blocks
When you have missing data for > 2 hours, you spend a lot of time
re-reading the complete file. It is not optimal.
This introduces a fastpath for this scenario.
Next, we do parse the metric even when we know we will not use it, based
on its timestamp. This only computes the metric when we know its
timestamp is right.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-12-30 02:06:41 +01:00
b317b6ab9c
Backfill from OpenMetrics format ( #8084 )
...
* get parser working
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* import file created
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Find min and max ts
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make two passes over file and write to tsdb
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* print error messages
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Max and Min initializer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Start with unit tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reset file read
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* align blocks to two hour range
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add cleanup test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove .ds_store
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add license to import_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix Circle CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Refactor code
Move backfill from tsdb to promtool directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix gitignore
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove panic
Rename ContenType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* adjust mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return statement
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go modules
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added unit test for backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix file handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close DB
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Close directory
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Error Handling
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* inline err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix command line flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add spaces before func
fix pointers
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add defer'd calls
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* move openmetrics.go content to backfill
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changed args to flags
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add tests for wrong OM files
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Added additional tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add comment to warn of func reuse
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Make input required in main.go
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer blockwriter close
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Remove contentType
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove defer from backfilltest
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix defer remove in backfill_test
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* changes to fix CI errors
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix go.mod
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change package name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* assert->require
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix format
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix createblock
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix defer
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix return
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* check err for anon func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* update comment
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix for the Flush Bug
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix formatting, comments, names
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* cleanup
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor test to take care of multiple samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* refactor tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* I dont know what I fixed
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fix tests, add test description, print blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* commit after 5000 samples
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* reviews part 1
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Series Count
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix CI
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove extra func
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* make timestamp into sec
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Reviews 2
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add Todo
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Fixes
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fixes reviews
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* =0
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove backfill.om
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add global err var, remove stuff
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* change var name
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* sampleLimit pass as parameter
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Add test when number of samples greater than batch size
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Change name of batchsize
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* revert export
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* remove
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add comment, remove newline,consistent err
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Print Blocks
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* Modify comments
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* db.Querier
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* add sanity check , get maxt and mint
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* ci error
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* comment change
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* nits
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* NoError
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
* fix
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-11-26 10:37:06 +05:30
Ganesh Vernekar
Julien Pivotto
Atibhi Agrawal