mirror of
https://github.com/ceph/ceph
synced 2025-03-11 02:39:05 +00:00
doc: instructions and guidance for multimds
Inspired by http://tracker.ceph.com/issues/19135 Signed-off-by: John Spray <john.spray@redhat.com>
This commit is contained in:
parent
f903f2f3fd
commit
76589ed9e1
@ -98,6 +98,7 @@ authentication keyring.
|
||||
Client authentication <client-auth>
|
||||
Upgrading old filesystems <upgrading>
|
||||
Configuring directory fragmentation <dirfrags>
|
||||
Configuring multiple active MDS daemons <multimds>
|
||||
|
||||
.. raw:: html
|
||||
|
||||
|
109
doc/cephfs/multimds.rst
Normal file
109
doc/cephfs/multimds.rst
Normal file
@ -0,0 +1,109 @@
|
||||
|
||||
Configuring multiple active MDS daemons
|
||||
---------------------------------------
|
||||
|
||||
*Also known as: multi-mds, active-active MDS*
|
||||
|
||||
Each CephFS filesystem is configured for a single active MDS daemon
|
||||
by default. To scale metadata performance for large scale systems, you
|
||||
may enable multiple active MDS daemons, which will share the metadata
|
||||
workload with one another.
|
||||
|
||||
When should I use multiple active MDS daemons?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You should configure multiple active MDS daemons when your metadata performance
|
||||
is bottlenecked on the single MDS that runs by default.
|
||||
|
||||
Adding more daemons may not increase performance on all workloads. Typically,
|
||||
a single application running on a single client will not benefit from an
|
||||
increased number of MDS daemons unless the application is doing a lot of
|
||||
metadata operations in parallel.
|
||||
|
||||
Workloads that typically benefit from a larger number of active MDS daemons
|
||||
are those with many clients, perhaps working on many separate directories.
|
||||
|
||||
|
||||
Increasing the MDS active cluster size
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Each CephFS filesystem has a *max_mds* setting, which controls
|
||||
how many ranks will be created. The actual number of ranks
|
||||
in the filesystem will only be increased if a spare daemon is
|
||||
available to take on the new rank. For example, if there is only one MDS daemon running, and max_mds is set to two, no second rank will be created.
|
||||
|
||||
Set ``max_mds`` to the desired number of ranks. In the following examples
|
||||
the "fsmap" line of "ceph status" is shown to illustrate the expected
|
||||
result of commands.
|
||||
|
||||
::
|
||||
|
||||
# fsmap e5: 1/1/1 up {0=a=up:active}, 2 up:standby
|
||||
|
||||
ceph fs set max_mds 2
|
||||
|
||||
# fsmap e8: 2/2/2 up {0=a=up:active,1=c=up:creating}, 1 up:standby
|
||||
# fsmap e9: 2/2/2 up {0=a=up:active,1=c=up:active}, 1 up:standby
|
||||
|
||||
The newly created rank (1) will pass through the 'creating' state
|
||||
and then enter this 'active state'.
|
||||
|
||||
Standby daemons
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Even with multiple active MDS daemons, a highly available system **still
|
||||
requires standby daemons** to take over if any of the servers running
|
||||
an active daemon fail.
|
||||
|
||||
Consequently, the practical maximum of ``max_mds`` for highly available systems
|
||||
is one less than the total number of MDS servers in your system.
|
||||
|
||||
To remain available in the event of multiple server failures, increase the
|
||||
number of standby daemons in the system to match the number of server failures
|
||||
you wish to withstand.
|
||||
|
||||
Decreasing the number of ranks
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All ranks, including the rank(s) to be removed must first be active. This
|
||||
means that you must have at least max_mds MDS daemons available.
|
||||
|
||||
First, set max_mds to a lower number, for example we might go back to
|
||||
having just a single active MDS:
|
||||
|
||||
::
|
||||
|
||||
# fsmap e9: 2/2/2 up {0=a=up:active,1=c=up:active}, 1 up:standby
|
||||
ceph fs set max_mds 1
|
||||
# fsmap e10: 2/2/1 up {0=a=up:active,1=c=up:active}, 1 up:standby
|
||||
|
||||
Note that we still have two active MDSs: the ranks still exist even though
|
||||
we have decreased max_mds, because max_mds only restricts creation
|
||||
of new ranks.
|
||||
|
||||
Next, use the ``ceph mds deactivate <rank>`` command to remove the
|
||||
unneeded rank:
|
||||
|
||||
::
|
||||
|
||||
ceph mds deactivate cephfs_a:1
|
||||
telling mds.1:1 172.21.9.34:6806/837679928 to deactivate
|
||||
|
||||
# fsmap e11: 2/2/1 up {0=a=up:active,1=c=up:stopping}, 1 up:standby
|
||||
# fsmap e12: 1/1/1 up {0=a=up:active}, 1 up:standby
|
||||
# fsmap e13: 1/1/1 up {0=a=up:active}, 2 up:standby
|
||||
|
||||
The deactivated rank will first enter the stopping state for a period
|
||||
of time while it hands off its share of the metadata to the remaining
|
||||
active daemons. This phase can take from seconds to minutes. If the
|
||||
MDS appears to be stuck in the stopping state then that should be investigated
|
||||
as a possible bug.
|
||||
|
||||
If an MDS daemon crashes or is killed while in the 'stopping' state, a
|
||||
standby will take over and the rank will go back to 'active'. You can
|
||||
try to deactivate it again once it has come back up.
|
||||
|
||||
When a daemon finishes stopping, it will respawn itself and go
|
||||
back to being a standby.
|
||||
|
||||
|
@ -9,6 +9,8 @@ or *FSCID*.
|
||||
|
||||
Each CephFS filesystem has a number of *ranks*, one by default,
|
||||
which start at zero. A rank may be thought of as a metadata shard.
|
||||
Controlling the number of ranks in a filesystem is described
|
||||
in :doc:`/cephfs/multimds`
|
||||
|
||||
Each CephFS ceph-mds process (a *daemon*) initially starts up
|
||||
without a rank. It may be assigned one by the monitor cluster.
|
||||
@ -19,11 +21,6 @@ If a rank is not associated with a daemon, the rank is
|
||||
considered *failed*. Once a rank is assigned to a daemon,
|
||||
the rank is considered *up*.
|
||||
|
||||
Each CephFS filesystem has a *max_mds* setting, which controls
|
||||
how many ranks will be created. The actual number of ranks
|
||||
in the filesystem will only be increased if a spare daemon is
|
||||
available to take on the new rank.
|
||||
|
||||
A daemon has a *name* that is set statically by the administrator
|
||||
when the daemon is first configured. Typical configurations
|
||||
use the hostname where the daemon runs as the daemon name.
|
||||
|
Loading…
Reference in New Issue
Block a user