RepoMirrors/prometheus
1
0
Fork 0
mirror of https://github.com/prometheus/prometheus synced 2024-12-26 08:33:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
8,309 Commits 243 Branches 444 Tags 653 MiB
dff967286e
Commit Graph

271 Commits

Author SHA1 Message Date
Ganesh Vernekar
dff967286e
Set the min time of Head properly after truncation (#8212) 
* Set the min time of Head properly after truncation

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix lint

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Enhance compaction plan logic for completely deleted small block

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-11-25 18:33:30 +05:30
Chris Marchbanks
c8f810083d
Cache label name and last value when adding series (#8192) 
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
 2020-11-19 11:18:12 -07:00
Marco Pracucci
db19e05d93
Add option to customise head chunks write buffer size (#8201) 
* Add option to customise head chunks write buffer size

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed tests

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-11-19 18:30:47 +05:30
Bartlomiej Plotka
4513537034
Exposed DeletionIterator and CompactMetas functions. (#8161) 
* Exposed DeletionIterator and CompactMetas functions.

Required for CLI for deletions in Thanos: https://github.com/thanos-io/thanos/pull/3421

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed Thanos usage mentions.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-11-09 16:51:25 +00:00
Julien Pivotto
8bc369bf9b
Calculate head chunk size based on actual disk usage (#8139) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-11-03 15:34:59 +05:30
Marco Pracucci
63be30dcee
Fixed WAL corruption on partial writes within a page (#8125) 
* Fixed WAL corruption on partial writes

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Renamed variable

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Fixed test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Addressed review comments

Signed-off-by: Marco Pracucci <marco@pracucci.com>
 2020-10-29 16:07:03 +05:30
Julien Pivotto
6c56a1faaa
Testify: move to require (#8122) 
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-29 09:43:23 +00:00
Bartlomiej Plotka
3d8826a3d4
MultiError: Refactored MultiError for more concise and safe usage. (#8066) 
* MultiError: Refactored MultiError for more concise and safe usage.

* Less lines
* Goland IDE was marking every usage of old MultiError "potential nil" error
* It was easy to forgot using Err() when error was returned, now it's safely assured on compile time.

NOTE: Potentially I would rename package to merrors. (: In different PR.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed review comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fix after rebase.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-28 15:24:58 +00:00
johncming
28ca0965f0
tsdb/chunkenc: fix typo of return error. (#7670) 
* tsdb/chunkenc: fix typo of return error.

Signed-off-by: johncming <johncming@yahoo.com>

* tsdb: fix typo of function in markdonw.

Signed-off-by: johncming <johncming@yahoo.com>
 2020-10-28 12:03:11 +00:00
Ganesh Vernekar
3245b3267b
Don't use returned DB to close resources on TSDB startup error (#8113) 
* Don't use returned DB to close resources on TSDB startup error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add unit test and fix another panic

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comment

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-28 15:39:03 +05:30
Julien Pivotto
1282d1b39c
Refactor test assertions (#8110) 
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-27 11:06:53 +01:00
Julien Pivotto
4e5b1722b3
Move away from testutil, refactor imports (#8087) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-22 11:00:08 +02:00
Arthur Silva Sens
c5a832b394
Close resources after failing to startup TSDB (#8031) 
* Close resources after failing to startup TSDB

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Return close error instead of logging

Signed-off-by: arthursens <arthursens2005@gmail.com>

* Change named return's name

Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-10-21 20:38:28 +05:30
Brian Brazil
fdf1c29224
Fix metric description of prometheus_tsdb_symbol_table_size_bytes. (#8080) 
This is how much memory we use to load in the on-disk
symbol tables, not the size of the tables themselves.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-21 14:35:40 +01:00
Ganesh Vernekar
ce4b3ac282
Simplify TestHeadReadWriter_Truncate (#7437) 
* Simplify TestHeadReadWriter_Truncate

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:41:39 +05:30
Ganesh Vernekar
2624d827fa
Read repair empty last file in chunks_head (#8061) 
* Read repair empty file in chunks_head

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Refactor and introduce repairLastChunkFile

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Attempt windows test fix

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-21 18:27:13 +05:30
Bartlomiej Plotka
2fe1e9fa93
Create a checkpoint only at the end of Compact call (#8067) 
* Create a checkpoint only at the end of Compact call

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix Bartek's offline reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Introduce TruncateInMemory and TruncateWAL

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Small enhancements and test fixing attempts

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix tests

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Add TestOneCheckpointPerCompactCall

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Don't truncate WAL on block compaction

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Simplified the algo.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Better protection around calling truncateWAL, truncate WAL on Head compaction error

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 20:57:08 +05:30
Ganesh Vernekar
d9cd913219
Add links to TSDB blog post series in tsdb/README.md (#8078) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 17:17:10 +05:30
Julien Pivotto
59733b1a26
TSDB: use blocks instead of db.blocks in condition (#8068) 
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
 2020-10-19 16:51:54 +05:30
Ganesh Vernekar
5de7d15879
Consistent naming and better names for fields in TSDB format docs (#8077) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-19 14:27:19 +05:30
Bartlomiej Plotka
9981b3f3ee
labels: Reduce allocated memory and latency of Hash method; Added tests. (#8025) 
* labels: Reduce allocated memory by Hash method in edge cases; Added tests.

Old:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 5366161	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 1700371	       767 ns/op	    2048 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	     356	   3743115 ns/op	10523442 B/op	       1 allocs/op
PASS

New:
/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	 4758883	       259 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	 3324492	       357 ns/op	      80 B/op	       1 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	    1087	   1083949 ns/op	    9734 B/op	       1 allocs/op
PASS

Process finished with exit code 0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Kemal's comment.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Moved to v2 xxhash for improvements.

New:

/tmp/___BenchmarkLabels_Hash_in_github_com_prometheus_prometheus_pkg_labels -test.v -test.bench ^\QBenchmarkLabels_Hash\E$ -test.run ^$ -test.benchtime 10s
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/pkg/labels
BenchmarkLabels_Hash
BenchmarkLabels_Hash/typical_labels_under_1KB
BenchmarkLabels_Hash/typical_labels_under_1KB-12         	53447894	       221 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/bigger_labels_over_1KB
BenchmarkLabels_Hash/bigger_labels_over_1KB-12           	42341754	       326 ns/op	       0 B/op	       0 allocs/op
BenchmarkLabels_Hash/extremely_large_label_value_10MB
BenchmarkLabels_Hash/extremely_large_label_value_10MB-12 	   10000	   1248546 ns/op	       0 B/op	       0 allocs/op
PASS

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Removed old xxhash package.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Pined minor version.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-10-15 11:31:28 +01:00
Ganesh Vernekar
4ed3ec7b46
More info in mmap error message (#8058) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-15 12:58:13 +05:30
frbimo
260cd84d8f
log on empty head compaction result (#8037) 
Signed-off-by: frbimo <fr.bimo@gmail.com>
 2020-10-14 18:59:59 +05:30
Gayathri Venkatesh
73e2ce1bd6
Do not ignore reload errors in compactHead (#8051) 
* Modified unknownRefs to unknownRefs.Load()

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Modified db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Revert "Modified unknownRefs to unknownRefs.Load()"

This reverts commit 79caf595fa9b9193878dc0dd9dec17d58851ae42.

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>

* Made changes to reload error in db.go

Signed-off-by: GayathriVenkatesh <gayaa2010@gmail.com>
 2020-10-14 15:05:24 +05:30
Goutham Veeramachaneni
1cc02930b6
Implement series API query hint to not hit chunks (#8050) 
Fixes https://github.com/prometheus/prometheus/issues/5547

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
 2020-10-14 14:36:17 +05:30
Ling Jin
9145200842
tsdb: fix unkown ref in log (#8048) 
Signed-off-by: JinLingChristopher <jinl1037@hotmail.com>
 2020-10-13 20:03:16 +05:30
Arthur Silva Sens
4f45e201cc
Promtool tsdb list now prints block sizes (#7993) 
* promtool tsdb list now prints blocks' size

Signed-off-by: arthursens <arthursens2005@gmail.com>
 2020-10-12 23:15:40 +02:00
Jess G
90680b092c
Add a tsdb writer interface (#7950) 
* add a tsdb writer interface

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in tsdbutil block creator

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* pass thru context

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use block writer in blockutil and also in tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* fix link err

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add err handling for defer

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* add blockwriter test

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* mv initHead into NewBlockWriter, update tests

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>

* use blocksize arg, make test more concise

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
 2020-10-12 17:04:20 +01:00
garanews
c38816828f
fix few typo (#8023) 
Signed-off-by: garanews <puntogtg@tiscali.it>
 2020-10-07 16:51:31 +01:00
Brian Brazil
073e93c768
Gracefully handle unknown WAL record types. (#8004) 
As we're looking to expand what's in the WAL,
having old Prometheus servers ignore the new record types
rather than treating them as corruption allows for better
upgrade/downgrade paths.

Adjust some tests accordingly, so they're still testing what they're
meant to test.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 2020-10-05 10:09:59 +01:00
Ganesh Vernekar
50ae4e298b
Fix magic number in docs (#7998) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-10-01 15:05:01 +05:30
Ganesh Vernekar
f0d87b5d86
Encode and Decode method for tombstones (#7967) 
* Encode and Decode method for tombstones

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-09-30 10:25:09 +05:30
Max Neverov
7e1c27b853
Add tsdb startup duration metric (#7737) 
* Add tsdb wal replay duration metric

Signed-off-by: Max Neverov <neverov.max@gmail.com>
 2020-09-21 18:25:05 +02:00
Jorge Vallecillo
7aa5fb01bf
tsdb/chunks/head_chunks_test.go: Fix typo (#7953) 
tsdb/db_test.go: Fix typo

Signed-off-by: Jorge Vallecillo <jorgevallecilloc@gmail.com>
 2020-09-20 18:42:01 +01:00
Tobias Klauser
b5b3bc34da
tsdb/fileutil: Use FcntlFstore from golang.org/x/sys/unix (#7949) 
Direct syscalls using syscall.Syscall(SYS_*, ...) should no longer be
used on darwin, see [1]. Instead, use the FcntlFstore libSystem wrapper
provided by the golang.org/x/sys/unix package to implement
preallocFixed.

[1] https://golang.org/doc/go1.12#darwin

Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
 2020-09-19 10:57:13 +01:00
Xiaochao Dong
a282d25099
tsdb: remove duplicate values set to reduce memory usage(map overhead) (#7915) 
Signed-off-by: Xiaochao Dong (@damnever) <dxc.wolf@gmail.com>
 2020-09-10 20:35:47 +05:30
zhulongcheng
5ea6bdca1f
tsdb: sync temporary checkpoint directory before rename (#7004) 
Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>
 2020-09-07 20:04:49 +05:30
johncming
75ae384192
tsdb: remove redundant fields. (#7869) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-09-02 17:03:21 +01:00
Ganesh Vernekar
2255b6f62f
Refactor WAL.Segments method to be part of the wal package (#6477) 
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-09-01 14:46:57 +05:30
johncming
252265fa81
tsdb/chunks: remove un-used code. (#6944) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-31 16:55:47 +02:00
Ganesh Vernekar
c806262206
Fix 'chunks.HeadReadWriter: maxt of the files are not set' error (#7856) 
* Fix chunks.HeadReadWriter: maxt of the files are not set

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
 2020-08-26 19:59:18 +02:00
Yukun Sun
cfd4e05c9e
fix: return a corruption error when iterator function find a chunk that is out of sequence (#7855) 
Signed-off-by: sunyukun <sunyukun@didiglobal.com>

Co-authored-by: sunyukun <sunyukun@didiglobal.com>
 2020-08-26 20:36:27 +05:30
Max Neverov
bb5c6b38e2
Fix Possible Race Condition in TSDB (#7815) 
* Replace tsdb chunk mapper size with atomic; protect mmappedChunkFiles with read path mutex on DeleteCorrupted

Signed-off-by: Max Neverov <neverov.max@gmail.com>

* PR fixes

Signed-off-by: Max Neverov <neverov.max@gmail.com>
 2020-08-26 14:22:48 +05:30
johncming
3e7b463908
add testcase for rangeForTimestamp. (#6454) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-25 16:16:43 +01:00
Guangwen Feng
9439b2150d
Fix golint warning caused by misspell (#7842) 
Signed-off-by: Guangwen Feng <fenggw-fnst@cn.fujitsu.com>
 2020-08-25 09:17:41 +01:00
johncming
b521612042
tsdb: simplify code. (#7792) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-14 15:15:08 +05:30
gastonqiu
5a7d398d19
Chore: Log segment number when segment read failed (#7687) 
* Chore: Log segment number when segment read failed

To manually fix the WAL files, it is good to know where the corrupt
happened so we should log the segment number when the read failed.

Related Issue #7506

Signed-off-by: gaston.qiu <gaston.qiu@umbocv.com>
 2020-08-13 11:01:21 -07:00
johncming
d19fc71903
tsdb: use NewRangeHead instead. (#7793) 
Signed-off-by: johncming <johncming@yahoo.com>
 2020-08-13 10:55:35 +01:00
Harold Dost
21a753c4e2
Make file permissions set to allow for wider umask options. (#7782) 
0644 -> 0666 on all non vendored code.

Fixes #7717

Signed-off-by: Harold Dost <harolddost@gmail.com>
 2020-08-12 23:23:17 +02:00
Bartlomiej Plotka
f16cbc20d6
tsdb: Bug fix for further continued deletions after crash deletions; added more tests. (#7777) 
* tsdb: Bug fix for further continued after crash deletions; added more tests.

Additionally: Added log line for block removal.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comment.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
 2020-08-11 15:53:23 +01:00