RepoMirrors
/
prometheus
mirror of https://github.com/prometheus/prometheus
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
prometheus/tsdb
History
Brian Brazil 48d25e6fe7 Reduce memory used by postings offset table. 
Rather than keeping the offset of each postings list, instead
keep the nth offset of the offset of the posting list. As postings
list offsets have always been sorted, we can then get to the closest
entry before the one we want an iterate forwards.

I haven't done much tuning on the 32 number, it was chosen to try
not to read through more than a 4k page of data.

Switch to a bulk interface for fetching postings. Use it to avoid having
to re-read parts of the posting offset table when querying lots of it.

For a index with what BenchmarkHeadPostingForMatchers uses RAM
for r.postings drops from 3.79MB to 80.19kB or about 48x.
Bytes allocated go down by 30%, and suprisingly CPU usage drops by
4-6% for typical queries too.

benchmark                                                               old ns/op      new ns/op      delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              35231          36673          +4.09%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      563380         540627         -4.04%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      536782         534186         -0.48%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     533990         541550         +1.42%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            113374598      117969608      +4.05%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            146329884      139651442      -4.56%
BenchmarkPostingsForMatchers/Block/i=~""-4                              50346510       44961127       -10.70%
BenchmarkPostingsForMatchers/Block/i!=""-4                              41261550       35356165       -14.31%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              112544418      116904010      +3.87%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       112487086      116864918      +3.89%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        41094758       35457904       -13.72%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                41906372       36151473       -13.73%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              147262414      140424800      -4.64%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             28615629       27872072       -2.60%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       147117177      140462403      -4.52%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     175096826      167902298      -4.11%

benchmark                                                               old allocs     new allocs     delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              4              6              +50.00%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      7              11             +57.14%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      7              11             +57.14%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     15             17             +13.33%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            100010         100012         +0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            200069         200040         -0.01%
BenchmarkPostingsForMatchers/Block/i=~""-4                              200072         200045         -0.01%
BenchmarkPostingsForMatchers/Block/i!=""-4                              200070         200041         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              100013         100017         +0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       100017         100023         +0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        200073         200046         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                200075         200050         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              200074         200049         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             111165         111150         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       200078         200055         -0.01%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     311282         311238         -0.01%

benchmark                                                               old bytes     new bytes     delta
BenchmarkPostingsForMatchers/Block/n="1"-4                              264           296           +12.12%
BenchmarkPostingsForMatchers/Block/n="1",j="foo"-4                      360           424           +17.78%
BenchmarkPostingsForMatchers/Block/j="foo",n="1"-4                      360           424           +17.78%
BenchmarkPostingsForMatchers/Block/n="1",j!="foo"-4                     520           552           +6.15%
BenchmarkPostingsForMatchers/Block/i=~".*"-4                            1600461       1600482       +0.00%
BenchmarkPostingsForMatchers/Block/i=~".+"-4                            24900801      17259077      -30.69%
BenchmarkPostingsForMatchers/Block/i=~""-4                              24900836      17259151      -30.69%
BenchmarkPostingsForMatchers/Block/i!=""-4                              24900760      17259048      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",j="foo"-4              1600557       1600621       +0.00%
BenchmarkPostingsForMatchers/Block/n="1",i=~".*",i!="2",j="foo"-4       1600717       1600813       +0.01%
BenchmarkPostingsForMatchers/Block/n="1",i!=""-4                        24900856      17259176      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i!="",j="foo"-4                24900952      17259304      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",j="foo"-4              24900993      17259333      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~"1.+",j="foo"-4             3788311       3142630       -17.04%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!="2",j="foo"-4       24901137      17259509      -30.69%
BenchmarkPostingsForMatchers/Block/n="1",i=~".+",i!~"2.*",j="foo"-4     28693086      20405680      -28.88%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
 		2019-12-11 19:59:31 +00:00
..
chunkenc Correct spelling mistakes 2019-10-10 18:46:27 +08:00
chunks return err instead of panic for corrupted chunk (#6040) 2019-12-04 09:37:49 +02:00
cmd/tsdb Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
docs/format Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
encoding Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
errors Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
fileutil Compute WAL size and use it during retention size checks (#5886) 2019-11-12 09:40:16 +07:00
goversion fix the wrong word (#6069) 2019-09-30 09:54:55 -06:00
index Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
record Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
test Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
testdata Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
tombstones remove debug fmt.Println in tombstones. (#6135) 2019-10-14 14:45:26 +03:00
tsdbutil Removed extra tsdb/testutil after merge. 2019-08-14 10:12:32 +01:00
wal Simplify duration check for watcher WAL replay. 2019-11-26 16:53:11 -08:00
.gitignore Moving tsdb into its own subdirectory 2019-08-13 13:58:49 +05:30
CHANGELOG.md fix-up tsdb-typo (#5954) 2019-08-28 14:43:02 +01:00
README.md Edit TSDB README badges 2019-10-24 15:35:47 +05:30
block.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
block_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
compact.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
compact_test.go Don't buffer up tables in memory on compaction. (#6422) 2019-12-11 12:49:13 +00:00
db.go tsdb: add error details in log. (#6415) 2019-12-09 10:37:01 +00:00
db_test.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
head.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
head_bench_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
head_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
mocks_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
querier.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
querier_bench_test.go Extend PostingsForMatchers benchmark to cover Blocks too. 2019-12-11 19:59:31 +00:00
querier_test.go Reduce memory used by postings offset table. 2019-12-11 19:59:31 +00:00
repair.go tsdb: add block meta version constant (#5994) 2019-09-09 12:28:01 +03:00
repair_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
tsdbblockutil.go Added CreateBlock and CreateHead functions to new file (#6331) 2019-11-21 19:10:25 +07:00
wal.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00
wal_test.go Port tsdb to use pkg/labels. (#6326) 2019-11-18 11:53:33 -08:00

README.md

TSDB

GoDoc

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.

See also the format documentation.