ceph/doc/cephfs/eviction.rst

Ceph filesystem client eviction
===============================

When a filesystem client is unresponsive or otherwise misbehaving, it
may be necessary to forcibly terminate its access to the filesystem.  This
process is called *eviction*.

This process is somewhat thorough in order to protect against data inconsistency
resulting from misbehaving clients.

OSD blacklisting
----------------

First, prevent the client from performing any more data operations by *blacklisting*
it at the RADOS level.  You may be familiar with this concept as *fencing* in other
storage systems.

Identify the client to evict from the MDS session list:

::

    # ceph daemon mds.a session ls
    [
        { "id": 4117,
          "num_leases": 0,
          "num_caps": 1,
          "state": "open",
          "replay_requests": 0,
          "reconnecting": false,
          "inst": "client.4117 172.16.79.251:0\/3271",
          "client_metadata": { "entity_id": "admin",
              "hostname": "fedoravm.localdomain",
              "mount_point": "\/home\/user\/mnt"}}]

In this case the 'fedoravm' client has address ``172.16.79.251:0/3271``, so we blacklist
it as follows:

::

    # ceph osd blacklist add 172.16.79.251:0/3271
    blacklisting 172.16.79.251:0/3271 until 2014-12-09 13:09:56.569368 (3600 sec)

OSD epoch barrier
-----------------

While the evicted client is now marked as blacklisted in the central (mon) copy of the OSD
map, it is now necessary to ensure that this OSD map update has propagated to all daemons
involved in subsequent filesystem I/O.  To do this, use the ``osdmap barrier`` MDS admin
socket command.

First read the latest OSD epoch:

::

    # ceph osd dump
    epoch 12
    fsid fd61ca96-53ff-4311-826c-f36b176d69ea
    created 2014-12-09 12:03:38.595844
    modified 2014-12-09 12:09:56.619957
    ...

In this case it is 12.  Now request the MDS to barrier on this epoch:

::

    # ceph daemon mds.a osdmap barrier 12

MDS session eviction
--------------------

Finally, it is safe to evict the client's MDS session, such that any capabilities it held
may be issued to other clients.  The ID here is the ``id`` attribute from the ``session ls``
output:

::

    # ceph daemon mds.a session evict 4117

That's it!  The client has now been evicted, and any resources it had locked will
now be available for other clients.

Background: OSD epoch barrier
-----------------------------

The purpose of the barrier is to ensure that when we hand out any
capabilities which might allow touching the same RADOS objects, the
clients we hand out the capabilities to must have a sufficiently recent
OSD map to not race with cancelled operations (from ENOSPC) or
blacklisted clients (from evictions)

More specifically, the cases where we set an epoch barrier are:

 * Client eviction (where the client is blacklisted and other clients
   must wait for a post-blacklist epoch to touch the same objects)
 * OSD map full flag handling in the client (where the client may
   cancel some OSD ops from a pre-full epoch, so other clients must
   wait until the full epoch or later before touching the same objects).
 * MDS startup, because we don't persist the barrier epoch, so must
   assume that latest OSD map is always required after a restart.

Note that this is a global value for simplicity: we could maintain this on
a per-inode basis.  We don't, because:

 * It would be more complicated
 * It would use an extra 4 bytes of memory for every inode
 * It would not be much more efficient as almost always everyone has the latest
   OSD map anyway, in most cases everyone will breeze through this barrier
   rather than waiting.
 * We only do this barrier in very rare cases, so any benefit from per-inode
   granularity would only very rarely be seen.

The epoch barrier is transmitted along with all capability messages, and
instructs the receiver of the message to avoid sending any more RADOS
operations to OSDs until it has seen this OSD epoch.  This mainly applies
to clients (doing their data writes directly to files), but also applies
to the MDS because things like file size probing and file deletion are
done directly from the MDS.