* Additional logging in compact.go - logged time needed for writing blocks to disk
Signed-off-by: Radoslaw Lesniewski <Radoslaw.Lesniewski@sabre.com>
* Additional logging in compact.go - code formatted
Signed-off-by: Radoslaw Lesniewski <Radoslaw.Lesniewski@sabre.com>
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
unexported NewMemTombstones as this returns unexported memTombstones
type which will not be shows in godoc.
Added missing comments for exported methods.
Removed unused RecordLogger,RecordReader interfaces.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
* compact: Verify for chunks outside of compacted time range.
Unit test for populateBlocs.
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Co-authored-by: Krasi Georgiev <kgeorgie@redhat.com>
* Fix filehandling for windows
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Fix more windows filehandling issues
Windows: Close files before deleting Checkpoints.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close writers in case of errors so they can be deleted
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close block so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close file to delete it
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: Close dir so that it can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Windows: close files so that they can be deleted.
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
* Review feedback
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
This reverts commit 98fe30438c.
After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.
Signed-off-by: beorn7 <beorn@soundcloud.com>
Blocks are half-open intervals [a, b), while all other intervals
(chunks, head, ...) are closed intervals [a, b].
Make that distinction explicit by defining `OverlapsClosedInterval()`
methods for blocks and chunks, and using them in place of the more
generic `intervalOverlap()` function.
This change also fixes `db.Querier()` and `db.Delete()`, which could
previously return one extraneous block at the end of the specified
interval.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
This is based on my experience while debugging https://github.com/prometheus/prometheus/issues/3943.
I needed to deduct few things, and all that would be just bit easier with these two logs:
- new block's ULID on each compaction.
- actual list of Blocks (ulid + time range) on Prometheus startup (easy to log that while repairing blocks).
We don't really need blocks that takes part in compaction - that can be deducted easily based on time ranges of blocks we have currently in system.
What do you think?
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Aligning postings list for similar keys close to each other improves
page cache hit rates in typical queries that select postings for
multiple label pairs with the same name.
This commit introduces error returns in various places and is explicit
about closing persisted blocks.
{Index,Chunk,Tombstone}Readers are more consistent about their Close()
method. Whenever a reader is retrieved, the corresponding close method
must eventually be called. We use this to track pending readers against
persisted blocks.
Querier's against the DB no longer hold a read lock for their entire
lifecycle. This avoids long running queriers to starve new ones when we
have to acquire a write lock when reloading blocks.
IDs for new series are handed out before the postings are locked. Thus
series are not indexed in order of their IDs, which could result in only
partially sorted postings list.
Iterating over those silently skipped elements as the sort invariant was
violated.
This adds various new locks to replace the single big lock on
the head. All parts now must be COW as they may be held by clients
after initial retrieval.
Series by ID and hashes are now held in a stripe lock to reduce
contention and total holding time during GC. This should reduce
starvation of readers.
This changes the structure to a single WAL backed by a single head
block.
Parts of the head block can be compacted. This relieves us from any head
amangement and greatly simplifies any consistency and isolation concerns
by just having a single head.
radek_lesniewski
Ganesh Vernekar
mknapphrt
Brian Brazil
Krasi Georgiev
Bartek Płotka
Goutham Veeramachaneni
Julius Volz
beorn7
codwu
Benoît Knecht
Fabian Reinartz
Fabian Reinartz
Goutham Veeramachaneni
Simon Pasquier
Shubheksha Jalan
ranbochen