From 3de734d8de3c6ddf8d037d5d4fd1bf4e1e0cc9a3 Mon Sep 17 00:00:00 2001 From: Frederic Branczyk Date: Tue, 29 Jan 2019 13:51:16 +0100 Subject: [PATCH 1/2] docs: Add filesystem POSIX requirement Signed-off-by: Frederic Branczyk --- docs/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/storage.md b/docs/storage.md index 9f6b0e913..55c8275b7 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -64,7 +64,7 @@ needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes To tune the rate of ingested samples per second, you can either reduce the number of time series you scrape (fewer targets or fewer series per target), or you can increase the scrape interval. However, reducing the number of series is likely more effective, due to compression of samples within a series. -If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. However, you can also try removing individual block directories to resolve the problem. This means losing a time window of around two hours worth of data per block directory. Again, Prometheus's local storage is not meant as durable long-term storage. +If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. Non POSIX compliant filesystems are not supported by Prometheus's local storage, corruptions may happen, without possibility to recover. However, you can also try removing individual block directories to resolve the problem. This means losing a time window of around two hours worth of data per block directory. Again, Prometheus's local storage is not meant as durable long-term storage. If both time and size retention policies are specified, whichever policy triggers first will be used at that instant. From 32079f351f12b4564fdfef013d8fea3179e79f65 Mon Sep 17 00:00:00 2001 From: Frederic Branczyk Date: Thu, 31 Jan 2019 12:57:48 +0100 Subject: [PATCH 2/2] docs: Specifically call out NFS and POSIX Signed-off-by: Frederic Branczyk --- docs/storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/storage.md b/docs/storage.md index 55c8275b7..8ee9366ec 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -64,7 +64,7 @@ needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes To tune the rate of ingested samples per second, you can either reduce the number of time series you scrape (fewer targets or fewer series per target), or you can increase the scrape interval. However, reducing the number of series is likely more effective, due to compression of samples within a series. -If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. Non POSIX compliant filesystems are not supported by Prometheus's local storage, corruptions may happen, without possibility to recover. However, you can also try removing individual block directories to resolve the problem. This means losing a time window of around two hours worth of data per block directory. Again, Prometheus's local storage is not meant as durable long-term storage. +If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. Non POSIX compliant filesystems are not supported by Prometheus's local storage, corruptions may happen, without possibility to recover. NFS is only potentially POSIX, most implementations are not. You can try removing individual block directories to resolve the problem, this means losing a time window of around two hours worth of data per block directory. Again, Prometheus's local storage is not meant as durable long-term storage. If both time and size retention policies are specified, whichever policy triggers first will be used at that instant.