It's a prefectly valid use case to have a sparse histogram with a zero
threshold of zero (i.e. only observations of exactly zero go into the
zero bucket). Even if the current PoC implementation of client_golang
doesn't allow that, such a case should be ingested properly.
However, there is now the edge case af a sparse histogram with a zero
threshold of zero and no observations yet. Such a histogram would look
the same if it was meant to be a conventional histogram. For now, we
ingest this case as a conventional histogram, but the final format
should have means to unambiguously express if a histogram is meant to
be ingested as a sparse histogram or as a conventional histogram.
Signed-off-by: beorn7 <beorn@grafana.com>
Parser now supports summaries and legacy histograms including
exemplars.
It also adds the option of specifying exemplars together with a sparse
histogram by simply using the legacy bucket section, too. The buckets
will be ignored, but the exemplars will be ingested.
Signed-off-by: beorn7 <beorn@grafana.com>
* Update querier.go to support Head compaction with histograms
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Add test for Head compaction with histograms
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix tests
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Do not panic on histoAppender.Append
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* M-map all chunks on shutdown
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Support negative schema for querying
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* bucketIterator which returns all valid bucket indices for a []span
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* support for comparing []spans and generating interjections
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* add license header
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* assert order fix
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* handle pathological 0-length span case more gracefully
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* stale todo
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* decode-recode histograms when new buckets appear
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* factor out recoding and also add it to the fallback case
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* make linter happy
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
This "brings back" protobuf parsing, with the only goal to play with
the new sparse histograms.
The Prom-2.x style parser is highly adapted to the structure of the
Prometheus text format (and later OpenMetrics). Some jumping through
hoops is required to feed protobuf into it.
This is not meant to be a model for the final implementation. It
should just enable sparse histogram ingestion at a reasonable
efficiency.
Following known shortcomings and flaws:
- No tests yet.
- Summaries and legacy histograms, i.e. without sparse buckets, are
ignored.
- Staleness doesn't work (but this could be fixed in the appender, to
be discussed).
- No tricks have been tried that would be similar to the tricks the
text parsers do (like direct pointers into the HTTP response
body). That makes things weird here. Tricky optimizations only make
sense once the final format is specified, which will almost
certainly not be the old protobuf format. (Interestingly, I expect
this implementation to be in fact much more efficient than the
original protobuf ingestion in Prom-1.x.)
- This is using a proto3 version of metrics.proto (mostly to be
consistent with the other protobuf uses). However, proto3 sees no
difference between an unset field. We depend on that to distinguish
between an unset timestamp and the timestamp 0 (1970-01-01, 00:00:00
UTC). In this experimental code, we just assume that timestamp is
never specified and therefore a timestamp of 0 always is interpreted
as "not set".
Signed-off-by: beorn7 <beorn@grafana.com>
* factor out different varbit schemes and include Beorn's "optimum" for buckets
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* use more compact dod encoding scheme for SHS chunk columns
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* remove FB VB and xor dod encoding because we won't use it
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* HistoChunk metadata encoding
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* add SparseHistogram.Copy()
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* histogram test: test appending a few histograms
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* add license headers
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* Append sparse histograms into the Head block
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Add AtHistogram() to Iterator interface. Make HistoChunk conform to Chunk interface.
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* integer types and timestamp separation
1) unify types to int64. as suggested by beorn. we want to support
counters going down (resets) even if we plan to create new chunks for
now, in that case
2) histogram type doesn't know its own timestamp. include it separately
in appending and iteration
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* correction: count and zeroCount to remain unsigned
to make api more resilient and that's what we use in protobuf anyway
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
* temp hack. Ganesh will fix
Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
We cannot just use prometheus/client_model directly because we want to
stay consistent with the use of gogo-protobuf. So this converts
metrics.proto to proto3 and edits it lightly so that it fits into
the framework how prometheus/prometheus handles protobuf.
Note that metrics.proto couldn't be merged into the prompb package
because prompb already has an Exemplar type, which is different from
the Exemplar type in metrics.proto. The directory structure seems to
play a role in the protobuf world, so I better kept it.
Signed-off-by: beorn7 <beorn@grafana.com>
* Added MaxSamplesPerSend
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Added tests
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Fixed order of require
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Added docs
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* writes -> writesReceived
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Improved send loop
Signed-off-by: Levi Harrison <git@leviharrison.dev>
* Cut v2.28.0-rc.0 (#8954)
* Cut v2.28.0-rc.0
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Changelog fixup
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Address review comments
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Downgrade some features to enhancements
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Adjust release date to today
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Migrate HTTP SD docs from docs repo (#8972)
See discussion in https://github.com/prometheus/docs/pull/1975
Signed-off-by: Julius Volz <julius.volz@gmail.com>
* Cut Prometheus v2.28.0 (#8973)
Signed-off-by: Julius Volz <julius.volz@gmail.com>
beorn7
Ganesh Vernekar
Ganesh Vernekar
Dieter Plaetinck
Julien Pivotto
Levi Harrison
Ben Kochie
Julius Volz