mirror of
https://github.com/ceph/ceph
synced 2024-12-18 09:25:49 +00:00
0d094f8f52
Rationale: There are many threshold limits for split and
merge in this doc that just says like:
"A directory fragment is eligible for splitting
when its size exceeds `mds_bal_split_size`
(default 10000)". Need to clarify what 10000 actually
means. This applies to all other such entries in this
doc.
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
102 lines
4.5 KiB
ReStructuredText
102 lines
4.5 KiB
ReStructuredText
|
|
===================================
|
|
Configuring Directory fragmentation
|
|
===================================
|
|
|
|
In CephFS, directories are *fragmented* when they become very large
|
|
or very busy. This splits up the metadata so that it can be shared
|
|
between multiple MDS daemons, and between multiple objects in the
|
|
metadata pool.
|
|
|
|
In normal operation, directory fragmentation is invisible to
|
|
users and administrators, and all the configuration settings mentioned
|
|
here should be left at their default values.
|
|
|
|
While directory fragmentation enables CephFS to handle very large
|
|
numbers of entries in a single directory, application programmers should
|
|
remain conservative about creating very large directories, as they still
|
|
have a resource cost in situations such as a CephFS client listing
|
|
the directory, where all the fragments must be loaded at once.
|
|
|
|
.. tip:: The root directory cannot be fragmented.
|
|
|
|
All directories are initially created as a single fragment. This fragment
|
|
may be *split* to divide up the directory into more fragments, and these
|
|
fragments may be *merged* to reduce the number of fragments in the directory.
|
|
|
|
Splitting and merging
|
|
=====================
|
|
|
|
When an MDS identifies a directory fragment to be split, it does not
|
|
do the split immediately. Because splitting interrupts metadata IO,
|
|
a short delay is used to allow short bursts of client IO to complete
|
|
before the split begins. This delay is configured with
|
|
``mds_bal_fragment_interval``, which defaults to 5 seconds.
|
|
|
|
When the split is done, the directory fragment is broken up into
|
|
a power of two number of new fragments. The number of new
|
|
fragments is given by two to the power ``mds_bal_split_bits``, i.e.
|
|
if ``mds_bal_split_bits`` is 2, then four new fragments will be
|
|
created. The default setting is 3, i.e. splits create 8 new fragments.
|
|
|
|
The criteria for initiating a split or a merge are described in the
|
|
following sections.
|
|
|
|
Size thresholds
|
|
===============
|
|
|
|
A directory fragment is eligible for splitting when its size exceeds
|
|
``mds_bal_split_size`` (default 10000 directory entries). Ordinarily this
|
|
split is delayed by ``mds_bal_fragment_interval``, but if the fragment size
|
|
exceeds a factor of ``mds_bal_fragment_fast_factor`` the split size,
|
|
the split will happen immediately (holding up any client metadata
|
|
IO on the directory).
|
|
|
|
``mds_bal_fragment_size_max`` is the hard limit on the size of
|
|
directory fragments. If it is reached, clients will receive
|
|
ENOSPC errors if they try to create files in the fragment. On
|
|
a properly configured system, this limit should never be reached on
|
|
ordinary directories, as they will have split long before. By default,
|
|
this is set to 10 times the split size, giving a dirfrag size limit of
|
|
100000 directory entries. Increasing this limit may lead to oversized
|
|
directory fragment objects in the metadata pool, which the OSDs may not
|
|
be able to handle.
|
|
|
|
A directory fragment is eligible for merging when its size is less
|
|
than ``mds_bal_merge_size``. There is no merge equivalent of the
|
|
"fast splitting" explained above: fast splitting exists to avoid
|
|
creating oversized directory fragments, there is no equivalent issue
|
|
to avoid when merging. The default merge size is 50 directory entries.
|
|
|
|
Activity thresholds
|
|
===================
|
|
|
|
In addition to splitting fragments based
|
|
on their size, the MDS may split directory fragments if their
|
|
activity exceeds a threshold.
|
|
|
|
The MDS maintains separate time-decaying load counters for read and write
|
|
operations on directory fragments. The decaying load counters have an
|
|
exponential decay based on the ``mds_decay_halflife`` setting.
|
|
|
|
On writes, the write counter is
|
|
incremented, and compared with ``mds_bal_split_wr``, triggering a
|
|
split if the threshold is exceeded. Write operations include metadata IO
|
|
such as renames, unlinks and creations.
|
|
|
|
The ``mds_bal_split_rd`` threshold is applied based on the read operation
|
|
load counter, which tracks readdir operations.
|
|
|
|
The ``mds_bal_split_rd`` and ``mds_bal_split_wr`` configs represent the
|
|
popularity threshold. In the MDS these are measured as "read/write temperatures"
|
|
which is closely related to the number of respective read/write operations.
|
|
By default, the read threshold is 25000 operations and the write
|
|
threshold is 10000 operations, i.e. 2.5x as many reads as writes would be
|
|
required to trigger a split.
|
|
|
|
After fragments are split due to the activity thresholds, they are only
|
|
merged based on the size threshold (``mds_bal_merge_size``), so
|
|
a spike in activity may cause a directory to stay fragmented
|
|
forever unless some entries are unlinked.
|
|
|