ceph/doc/dev/mon-osdmap-prune.rst

===========================
FULL OSDMAP VERSION PRUNING
===========================

For each incremental osdmap epoch, the monitor will keep a full osdmap
epoch in the store.

While this is great when serving osdmap requests from clients, allowing
us to fulfill their request without having to recompute the full osdmap
from a myriad of incrementals, it can also become a burden once we start
keeping an unbounded number of osdmaps.

The monitors will attempt to keep a bounded number of osdmaps in the store.
This number is defined (and configurable) via ``mon_min_osdmap_epochs``, and
defaults to 500 epochs. Generally speaking, we will remove older osdmap
epochs once we go over this limit.

However, there are a few constraints to removing osdmaps. These are all
defined in ``OSDMonitor::get_trim_to()``.

In the event one of these conditions is not met, we may go over the bounds
defined by ``mon_min_osdmap_epochs``. And if the cluster does not meet the
trim criteria for some time (e.g., unclean pgs), the monitor may start
keeping a lot of osdmaps. This can start putting pressure on the underlying
key/value store, as well as on the available disk space.

One way to mitigate this problem would be to stop keeping full osdmap
epochs on disk. We would have to rebuild osdmaps on-demand, or grab them
from cache if they had been recently served. We would still have to keep
at least one osdmap, and apply all incrementals on top of either this
oldest map epoch kept in the store or a more recent map grabbed from cache.
While this would be feasible, it seems like a lot of cpu (and potentially
IO) would be going into rebuilding osdmaps.

Additionally, this would prevent the aforementioned problem going forward,
but would do nothing for stores currently in a state that would truly
benefit from not keeping osdmaps.

This brings us to full osdmap pruning.

Instead of not keeping full osdmap epochs, we are going to prune some of
them when we have too many.

Deciding whether we have too many will be dictated by a configurable option
``mon_osdmap_full_prune_min`` (default: 10000). The pruning algorithm will be
engaged once we go over this threshold.

We will not remove all ``mon_osdmap_full_prune_min`` full osdmap epochs
though. Instead, we are going to poke some holes in the sequence of full
maps. By default, we will keep one full osdmap per 10 maps since the last
map kept; i.e., if we keep epoch 1, we will also keep epoch 10 and remove
full map epochs 2 to 9. The size of this interval is configurable with
``mon_osdmap_full_prune_interval``.

Essentially, we are proposing to keep ~10% of the full maps, but we will
always honour the minimum number of osdmap epochs, as defined by
``mon_min_osdmap_epochs``, and these won't be used for the count of the
minimum versions to prune. For instance, if we have on-disk versions
[1..50000], we would allow the pruning algorithm to operate only over
osdmap epochs [1..49500); but, if have on-disk versions [1..10200], we
won't be pruning because the algorithm would only operate on versions
[1..9700), and this interval contains less versions than the minimum
required by ``mon_osdmap_full_prune_min``.


ALGORITHM
=========

Say we have 50,000 osdmap epochs in the store, and we're using the
defaults for all configurable options.

::

    -----------------------------------------------------------
    |1|2|..|10|11|..|100|..|1000|..|10000|10001|..|49999|50000|
    -----------------------------------------------------------
     ^ first                                            last ^

We will prune when all the following constraints are met:

1. number of versions is greater than ``mon_min_osdmap_epochs``;

2. the number of versions between ``first`` and ``prune_to`` is greater (or
   equal) than ``mon_osdmap_full_prune_min``, with ``prune_to`` being equal to
   ``last`` minus ``mon_min_osdmap_epochs``.

If any of these conditions fails, we will *not* prune any maps.

Furthermore, if it is known that we have been pruning, but since then we
are no longer satisfying at least one of the above constraints, we will
not continue to prune. In essence, we only prune full osdmaps if the
number of epochs in the store so warrants it.

As pruning will create gaps in the sequence of full maps, we need to keep
track of the intervals of missing maps. We do this by keeping a manifest of
pinned maps -- i.e., a list of maps that, by being pinned, are not to be
pruned.

While pinned maps are not removed from the store, maps between two consecutive
pinned maps will; and the number of maps to be removed will be dictated by the
configurable option ``mon_osdmap_full_prune_interval``. The algorithm makes an
effort to keep pinned maps apart by as many maps as defined by this option,
but in the event of corner cases it may allow smaller intervals. Additionally,
as this is a configurable option that is read any time a prune iteration
occurs, there is the possibility this interval will change if the user changes
this config option.

Pinning maps is performed lazily: we will be pinning maps as we are removing
maps. This grants us more flexibility to change the prune interval while
pruning is happening, but also simplifies considerably the algorithm, as well
as the information we need to keep in the manifest. Below we show a simplified
version of the algorithm:::

    manifest.pin(first)
    last_to_prune = last - mon_min_osdmap_epochs

    while manifest.get_last_pinned() + prune_interval < last_to_prune AND
          last_to_prune - first > mon_min_osdmap_epochs AND
          last_to_prune - first > mon_osdmap_full_prune_min AND
          num_pruned < mon_osdmap_full_prune_txsize:
      
      last_pinned = manifest.get_last_pinned()
      new_pinned = last_pinned + prune_interval
      manifest.pin(new_pinned)
      for e in (last_pinned .. new_pinned):
        store.erase(e)
        ++num_pruned

In essence, the algorithm ensures that the first version in the store is
*always* pinned. After all, we need a starting point when rebuilding maps, and
we can't simply remove the earliest map we have; otherwise we would be unable
to rebuild maps for the very first pruned interval.

Once we have at least one pinned map, each iteration of the algorithm can
simply base itself on the manifest's last pinned map (which we can obtain by
reading the element at the tail of the manifest's pinned maps list).

We'll next need to determine the interval of maps to be removed: all the maps
from ``last_pinned`` up to ``new_pinned``, which in turn is nothing more than
``last_pinned`` plus ``mon_osdmap_full_prune_interval``. We know that all maps
between these two values, ``last_pinned`` and ``new_pinned`` can be removed,
considering ``new_pinned`` has been pinned.

The algorithm ceases to execute as soon as one of the two initial
preconditions is not met, or if we do not meet two additional conditions that
have no weight on the algorithm's correctness:

1. We will stop if we are not able to create a new pruning interval properly
   aligned with ``mon_osdmap_full_prune_interval`` that is lower than
   ``last_pruned``. There is no particular technical reason why we enforce
   this requirement, besides allowing us to keep the intervals with an
   expected size, and preventing small, irregular intervals that would be
   bound to happen eventually (e.g., pruning continues over the course of
   several iterations, removing one or two or three maps each time).

2. We will stop once we know that we have pruned more than a certain number of
   maps. This value is defined by ``mon_osdmap_full_prune_txsize``, and
   ensures we don't spend an unbounded number of cycles pruning maps. We don't
   enforce this value religiously (deletes do not cost much), but we make an
   effort to honor it.

We could do the removal in one go, but we have no idea how long that would
take. Therefore, we will perform several iterations, removing at most
``mon_osdmap_full_prune_txsize`` osdmaps per iteration.

In the end, our on-disk map sequence will look similar to::

    ------------------------------------------
    |1|10|20|30|..|49500|49501|..|49999|50000|
    ------------------------------------------
     ^ first                           last ^


Because we are not pruning all versions in one go, we need to keep state
about how far along on our pruning we are. With that in mind, we have
created a data structure, ``osdmap_manifest_t``, that holds the set of pinned
maps:::

    struct osdmap_manifest_t:
        set<version_t> pinned;

Given we are only pinning maps while we are pruning, we don't need to keep
track of additional state about the last pruned version. We know as a matter
of fact that we have pruned all the intermediate maps between any two
consecutive pinned maps.

The question one could ask, though, is how can we be sure we pruned all the
intermediate maps if, for instance, the monitor crashes. To ensure we are
protected against such an event, we always write the osdmap manifest to disk
on the same transaction that is deleting the maps. This way we have the
guarantee that, if the monitor crashes, we will read the latest version of the
manifest: either containing the newly pinned maps, meaning we also pruned the
in-between maps; or we will find the previous version of the osdmap manifest,
which will not contain the maps we were pinning at the time we crashed, given
the transaction on which we would be writing the updated osdmap manifest was
not applied (alongside with the maps removal).

The osdmap manifest will be written to the store each time we prune, with an
updated list of pinned maps. It is written in the transaction effectively
pruning the maps, so we guarantee the manifest is always up to date. As a
consequence of this criteria, the first time we will write the osdmap manifest
is the first time we prune. If an osdmap manifest does not exist, we can be
certain we do not hold pruned map intervals.

We will rely on the manifest to ascertain whether we have pruned maps
intervals. In theory, this will always be the on-disk osdmap manifest, but we
make sure to read the on-disk osdmap manifest each time we update from paxos;
this way we always ensure having an up to date in-memory osdmap manifest.

Once we finish pruning maps, we will keep the manifest in the store, to
allow us to easily find which maps have been pinned (instead of checking
the store until we find a map). This has the added benefit of allowing us to
quickly figure out which is the next interval we need to prune (i.e., last
pinned plus the prune interval). This doesn't however mean we will forever
keep the osdmap manifest: the osdmap manifest will no longer be required once
the monitor trims osdmaps and the earliest available epoch in the store is
greater than the last map we pruned.

The same conditions from ``OSDMonitor::get_trim_to()`` that force the monitor
to keep a lot of osdmaps, thus requiring us to prune, may eventually change
and allow the monitor to remove some of its oldest maps.

MAP TRIMMING
------------

If the monitor trims maps, we must then adjust the osdmap manifest to
reflect our pruning status, or remove the manifest entirely if it no longer
makes sense to keep it. For instance, take the map sequence from before, but
let us assume we did not finish pruning all the maps.::

    -------------------------------------------------------------
    |1|10|20|30|..|490|500|501|502|..|49500|49501|..|49999|50000|
    -------------------------------------------------------------
     ^ first            ^ pinned.last()                   last ^

    pinned = {1, 10, 20, ..., 490, 500}

Now let us assume that the monitor will trim up to epoch 501. This means
removing all maps prior to epoch 501, and updating the ``first_committed``
pointer to ``501``. Given removing all those maps would invalidate our
existing pruning efforts, we can consider our pruning has finished and drop
our osdmap manifest. Doing so also simplifies starting a new prune, if all
the starting conditions are met once we refreshed our state from the
store.

We would then have the following map sequence: ::

    ---------------------------------------
    |501|502|..|49500|49501|..|49999|50000|
    ---------------------------------------
     ^ first                        last ^

However, imagine a slightly more convoluted scenario: the monitor will trim
up to epoch 491. In this case, epoch 491 has been previously pruned from the
store.

Given we will always need to have the oldest known map in the store, before
we trim we will have to check whether that map is in the prune interval
(i.e., if said map epoch belongs to ``[ pinned.first()..pinned.last() )``).
If so, we need to check if this is a pinned map, in which case we don't have
much to be concerned aside from removing lower epochs from the manifest's
pinned list. On the other hand, if the map being trimmed to is not a pinned
map, we will need to rebuild said map and pin it, and only then will we remove
the pinned maps prior to the map's epoch. 

In this case, we would end up with the following sequence:::

    -----------------------------------------------
    |491|500|501|502|..|49500|49501|..|49999|50000|
    -----------------------------------------------
     ^   ^- pinned.last()                   last ^
     `- first

There is still an edge case that we should mention. Consider that we are
going to trim up to epoch 499, which is the very last pruned epoch.

Much like the scenario above, we would end up writing osdmap epoch 499 to
the store; but what should we do about pinned maps and pruning?

The simplest solution is to drop the osdmap manifest. After all, given we
are trimming to the last pruned map, and we are rebuilding this map, we can
guarantee that all maps greater than e 499 are sequential (because we have
not pruned any of them). In essence, dropping the osdmap manifest in this
case is essentially the same as if we were trimming over the last pruned
epoch: we can prune again later if we meet the required conditions.

And, with this, we have fully dwelled into full osdmap pruning. Later in this
document one can find detailed `REQUIREMENTS, CONDITIONS & INVARIANTS` for the
whole algorithm, from pruning to trimming. Additionally, the next section
details several additional checks to guarantee the sanity of our configuration
options. Enjoy.


CONFIGURATION OPTIONS SANITY CHECKS
-----------------------------------

We perform additional checks before pruning to ensure all configuration
options involved are sane:

1. If ``mon_osdmap_full_prune_interval`` is zero we will not prune; we
   require an actual positive number, greater than one, to be able to prune
   maps. If the interval is one, we would not actually be pruning any maps, as
   the interval between pinned maps would essentially be a single epoch. This
   means we would have zero maps in-between pinned maps, hence no maps would
   ever be pruned.

2. If ``mon_osdmap_full_prune_min`` is zero we will not prune; we require a
   positive, greater than zero, value so we know the threshold over which we
   should prune. We don't want to guess.

3. If ``mon_osdmap_full_prune_interval`` is greater than
   ``mon_osdmap_full_prune_min`` we will not prune, as it is impossible to
   ascertain a proper prune interval.

4. If ``mon_osdmap_full_prune_txsize`` is lower than
   ``mon_osdmap_full_prune_interval`` we will not prune; we require a
   ``txsize`` with a value at least equal than ``interval``, and (depending on
   the value of the latter) ideally higher.


REQUIREMENTS, CONDITIONS & INVARIANTS
-------------------------------------

REQUIREMENTS
~~~~~~~~~~~~

* All monitors in the quorum need to support pruning.

* Once pruning has been enabled, monitors not supporting pruning will not be
  allowed in the quorum, nor will be allowed to synchronize.

* Removing the osdmap manifest results in disabling the pruning feature quorum
  requirement. This means that monitors not supporting pruning will be allowed
  to synchronize and join the quorum, granted they support any other features
  required.


CONDITIONS & INVARIANTS
~~~~~~~~~~~~~~~~~~~~~~~

* Pruning has never happened, or we have trimmed past its previous
  intervals:::

    invariant: first_committed > 1

    condition: pinned.empty() AND !store.exists(manifest)


* Pruning has happened at least once:::

    invariant: first_committed > 0
    invariant: !pinned.empty())
    invariant: pinned.first() == first_committed
    invariant: pinned.last() < last_committed

      precond: pinned.last() < prune_to AND
               pinned.last() + prune_interval < prune_to

     postcond: pinned.size() > old_pinned.size() AND
               (for each v in [pinned.first()..pinned.last()]:
                 if pinned.count(v) > 0: store.exists_full(v)
                 else: !store.exists_full(v)
               )


* Pruning has finished:::

    invariant: first_committed > 0
    invariant: !pinned.empty()
    invariant: pinned.first() == first_committed
    invariant: pinned.last() < last_committed

    condition: pinned.last() == prune_to OR
               pinned.last() + prune_interval < prune_to


* Pruning intervals can be trimmed:::

    precond:   OSDMonitor::get_trim_to() > 0

    condition: !pinned.empty()

    invariant: pinned.first() == first_committed
    invariant: pinned.last() < last_committed
    invariant: pinned.first() <= OSDMonitor::get_trim_to()
    invariant: pinned.last() >= OSDMonitor::get_trim_to()

* Trim pruned intervals:::

    invariant: !pinned.empty()
    invariant: pinned.first() == first_committed
    invariant: pinned.last() < last_committed
    invariant: pinned.first() <= OSDMonitor::get_trim_to()
    invariant: pinned.last() >= OSDMonitor::get_trim_to()

    postcond:  pinned.empty() OR
               (pinned.first() == OSDMonitor::get_trim_to() AND
                pinned.last() > pinned.first() AND
                (for each v in [0..pinned.first()]:
                  !store.exists(v) AND
                  !store.exists_full(v)
                ) AND
                (for each m in [pinned.first()..pinned.last()]:
                  if pinned.count(m) > 0: store.exists_full(m)
                  else: !store.exists_full(m) AND store.exists(m)
                )
               )
    postcond:  !pinned.empty() OR
               (!store.exists(manifest) AND
                (for each v in [pinned.first()..pinned.last()]:
                  !store.exists(v) AND
                  !store.exists_full(v)
                )
               )