ceph/doc/rados/configuration/ceph-conf.rst

631 lines
22 KiB
ReStructuredText
Raw Normal View History

==================
Configuring Ceph
==================
When you start the Ceph service, the initialization process activates a series
of daemons that run in the background. The hosts in a typical Ceph cluster run
at least one of four daemons:
- Object Storage Device (``ceph-osd``)
- Monitor (``ceph-mon``)
- Metadata Server (``ceph-mds``)
- Ceph Gateway (``radosgw``)
For your convenience, each daemon has a series of default values (*i.e.*, many
are set by ``ceph/src/common/config_opts.h``). You may override these settings
with a Ceph configuration file.
.. _ceph-conf-file:
The Configuration File
======================
When you start a Ceph cluster, each daemon looks for a Ceph configuration file
(i.e., ``ceph.conf`` by default) that provides the cluster's configuration
settings. For manual deployments, you need to create a Ceph configuration file.
For third party tools that create configuration files for you (*e.g.*, Chef),
you may use the information contained herein as a reference. The Ceph
Configuration file defines:
- Authentication settings
- Cluster membership
- Host names
- Host addresses
- Paths to keyrings
- Paths to journals
- Paths to data
- Other runtime options
The default ``ceph.conf`` locations in sequential order include:
#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable)
#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
#. ``/etc/ceph/ceph.conf``
#. ``~/.ceph/config``
#. ``./ceph.conf`` (*i.e.,* in the current working directory)
The Ceph configuration file uses an *ini* style syntax. You can add comments
by preceding comments with a semi-colon (;) or a pound sign (#). For example:
.. code-block:: ini
# <--A number (#) sign precedes a comment.
; A comment may be anything.
# Comments always follow a semi-colon (;) or a pound (#) on each line.
# The end of the line terminates a comment.
# We recommend that you provide comments in your configuration file(s).
.. _ceph-conf-settings:
Config Sections
===============
The configuration file can configure all daemons in a cluster, or all daemons of
a particular type. To configure a series of daemons, the settings must be
included under the processes that will receive the configuration as follows:
``[global]``
:Description: Settings under ``[global]`` affect all daemons in a Ceph cluster.
:Example: ``auth supported = cephx``
``[osd]``
:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in
the cluster.
:Example: ``osd journal size = 1000``
``[mon]``
:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in
the cluster.
:Example: ``mon addr = 10.0.0.101:6789``
``[mds]``
:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in
the cluster.
:Example: ``host = myserver01``
``[client]``
:Description: Settings under ``[client]`` affect all clients (e.g., mounted
CephFS filesystems, mounted block devices, etc.)
:Example: ``log file = /var/log/ceph/radosgw.log``
Global settings affect all instances of all daemon in the cluster. Use the ``[global]``
setting for values that are common for all daemons in the cluster. You can override each
``[global]`` setting by:
#. Changing the setting in a particular process type
(*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ).
#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` )
Overriding a global setting affects all child processes, except those that
you specifically override.
A typical global setting involves activating authentication. For example:
.. code-block:: ini
[global]
#Enable authentication between hosts within the cluster.
#v 0.54 and earlier
auth supported = cephx
#v 0.55 and after
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
You can specify settings that apply to a particular type of daemon. When you
specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a
particular instance, the setting will apply to all OSDs, monitors or metadata
daemons respectively.
You may specify settings for particular instances of a daemon. You may specify
an instance by entering its type, delimited by a period (.) and by the
instance ID. The instance ID for an OSD is always numeric, but it may be
alphanumeric for monitors and metadata servers.
.. code-block:: ini
[osd.1]
# settings affect osd.1 only.
[mon.a]
# settings affect mon.a only.
[mds.b]
# settings affect mds.b only.
.. _ceph-metavariables:
Metavariables
=============
Metavariables simplify cluster configuration dramatically. When a metavariable
is set in a configuration value, Ceph expands the metavariable into a concrete
value. Metavariables are very powerful when used within the ``[global]``,
``[osd]``, ``[mon]`` or ``[mds]`` sections of your configuration file. Ceph
metavariables are similar to Bash shell expansion.
Ceph supports the following metavariables:
``$cluster``
:Description: Expands to the cluster name. Useful when running multiple
clusters on the same hardware.
:Example: ``/etc/ceph/$cluster.keyring``
:Default: ``ceph``
``$type``
:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the
type of the current daemon.
:Example: ``/var/lib/ceph/$type``
``$id``
:Description: Expands to the daemon identifier. For ``osd.0``, this would be
``0``; for ``mds.a``, it would be ``a``.
:Example: ``/var/lib/ceph/$type/$cluster-$id``
``$host``
:Description: Expands to the host name of the current daemon.
``$name``
:Description: Expands to ``$type.$id``.
:Example: ``/var/run/ceph/$cluster-$name.asok``
.. _ceph-conf-common-settings:
Common Settings
===============
The `Hardware Recommendations`_ section provides some hardware guidelines for
configuring the cluster. It is possible for a single host to run multiple
daemons. For example, a single host with multiple disks or RAIDs may run one
``ceph-osd`` for each disk or RAID. Additionally, a host may run both a
``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have
a host for a particular type of process. For example, one host may run
``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other
hosts may run ``ceph-mon`` daemons.
Each host has a name identified by the ``host`` setting. Monitors also specify
a network address and port (i.e., domain name or IP address) identified by the
``addr`` setting. A basic configuration file will typically specify only
minimal settings for each instance of a daemon. For example:
.. code-block:: ini
[mon.a]
host = hostName
mon addr = 150.140.130.120:6789
[osd.0]
host = hostName
.. important:: The ``host`` setting is the short name of the host (i.e., not
an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on
the command line to retrieve the name of the host. Also, this setting is
**ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT**
be used with ``chef`` or ``ceph-deploy``.
.. _ceph-network-config:
Networks
========
See the `Network Configuration Reference`_ for a detailed discussion about
configuring a network for use with Ceph.
Authentication
==============
.. versionadded:: Bobtail 0.56
For Bobtail (v 0.56) and beyond, you should expressly enable or disable authentication
in the ``[global]`` section of your Ceph configuration file. ::
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
Additionally, you should enable message signing. See `Cephx Config Reference`_
and `Cephx Authentication`_ for details.
.. important:: When upgrading, we recommend expressly disabling authentication first,
then perform the upgrade. Once the upgrade is complete, re-enable authentication.
.. _Cephx Authentication: ../../operations/authentication
.. _Cephx Config Reference: ../auth-config-ref
.. _ceph-monitor-config:
Monitors
========
Ceph production clusters typically deploy with a minimum 3 monitors to ensure
high availability should a monitor instance crash. An odd number of monitors (3)
ensures that the Paxos algorithm can determine which version of the cluster map
is the most recent from a quorum of monitors.
.. note:: You may deploy Ceph with a single monitor, but if the instance fails,
the lack of a monitor may interrupt data service availability.
Ceph monitors typically listen on port ``6789``. For example:
.. code-block:: ini
[mon.a]
host = hostName
mon addr = 150.140.130.120:6789
By default, Ceph expects that you will store a monitor's data under the following path::
/var/lib/ceph/mon/$cluster-$id
You must create the corresponding directory yourself. With metavariables fully
expressed and a cluster named "ceph", the foregoing directory would evaluate to::
/var/lib/ceph/mon/ceph-a
You may override this path using the ``mon data`` setting. We don't recommend
changing the default location. Create the default directory on your new monitor host. ::
ssh {new-mon-host}
sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter}
.. _ceph-osd-config:
OSDs
====
Ceph production clusters typically deploy OSDs where one host has one OSD daemon
running a filestore on one data disk. A typical deployment specifies a journal
size and whether the file store's extended attributes (XATTRs) use an
object map (i.e., when running on the ``ext4`` filesystem). For example:
.. code-block:: ini
[osd]
osd journal size = 10000
filestore xattr use omap = true #enables the object map. Only if running ext4.
[osd.0]
host = {hostname}
By default, Ceph expects that you will store an OSD's data with the following path::
/var/lib/ceph/osd/$cluster-$id
You must create the corresponding directory yourself. With metavariables fully
expressed and a cluster named "ceph", the foregoing directory would evaluate to::
/var/lib/ceph/osd/ceph-0
You may override this path using the ``osd data`` setting. We don't recommend
changing the default location. Create the default directory on your new OSD host. ::
ssh {new-osd-host}
sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
The ``osd data`` path ideally leads to a mount point with a hard disk that is
separate from the hard disk storing and running the operating system and
daemons. If the OSD is for a disk other than the OS disk, prepare it for
use with Ceph, and mount it to the directory you just created::
ssh {new-osd-host}
sudo mkfs -t {fstype} /dev/{disk}
sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
We recommend using the ``xfs`` file system or the ``btrfs`` file system when
running :command:mkfs.
By default, Ceph expects that you will store an OSDs journal with the
following path::
/var/lib/ceph/osd/$cluster-$id/journal
Without performance optimization, Ceph stores the journal on the same disk as
the OSDs data. An OSD optimized for performance may use a separate disk to store
journal data (e.g., a solid state drive delivers high performance journaling).
Ceph's default ``osd journal size`` is 0, so you will need to set this in your
``ceph.conf`` file. A journal size should find the product of the ``filestore
max sync interval`` and the expected throughput, and multiply the product by
two (2)::
osd journal size = {2 * (expected throughput * filestore max sync interval)}
The expected throughput number should include the expected disk throughput
(i.e., sustained data transfer rate), and network throughput. For example,
a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()``
of the disk and network throughput should provide a reasonable expected
throughput. Some users just start off with a 10GB journal size. For
example::
osd journal size = 10000
.. _ceph-logging-and-debugging:
Logs / Debugging
================
Ceph is still on the leading edge, so you may encounter situations that require
modifying logging output and using Ceph's debugging. To activate Ceph's
debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your
configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is
terse and 20 is verbose.
.. note:: See `Debugging and Logging`_ for details on log rotation.
.. _Debugging and Logging: ../../operations/debug
Subsystems common to each daemon may be set under ``[global]`` in your
configuration file. Subsystems for particular daemons are set under the daemon
section in your configuration file (*e.g.*, ``[mon]``, ``[osd]``, ``[mds]``).
For example::
[global]
debug ms = 1
[mon]
debug mon = 20
debug paxos = 20
debug auth = 20
[osd]
debug osd = 20
debug filestore = 20
debug journal = 20
debug monc = 20
[mds]
debug mds = 20
debug mds balancer = 20
debug mds log = 20
debug mds migrator = 20
When your system is running well, choose appropriate logging levels and remove
unnecessary debugging settings to ensure your cluster runs optimally. Logging
debug output messages is relatively slow, and a waste of resources when
operating your cluster.
.. tip: When debug output slows down your system, the latency can hide
race conditions.
Each subsystem has a logging level for its output logs, and for its logs
in-memory. You may set different values for each of these subsystems by setting
a log file level and a memory level for debug logging. For example::
debug {subsystem} {log-level}/{memory-level}
#for example
debug mds log 1/20
+--------------------+-----------+--------------+
| Subsystem | Log Level | Memory Level |
+====================+===========+==============+
| ``default`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``lockdep`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``context`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``crush`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds balancer`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds locker`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds log`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds log expire`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``mds migrator`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``buffer`` | 0 | 0 |
+--------------------+-----------+--------------+
| ``timer`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``filer`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``objecter`` | 0 | 0 |
+--------------------+-----------+--------------+
| ``rados`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``rbd`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``journaler`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``objectcacher`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``client`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``osd`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``optracker`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``objclass`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``filestore`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``journal`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``ms`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``mon`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``monc`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``paxos`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``tp`` | 0 | 5 |
+--------------------+-----------+--------------+
| ``auth`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``finisher`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``heartbeatmap`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``perfcounter`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``rgw`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``hadoop`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``asok`` | 1 | 5 |
+--------------------+-----------+--------------+
| ``throttle`` | 1 | 5 |
+--------------------+-----------+--------------+
Example ceph.conf
=================
.. literalinclude:: demo-ceph.conf
:language: ini
.. _ceph-runtime-config:
Runtime Changes
===============
Ceph allows you to make changes to the configuration of an ``ceph-osd``,
``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite
useful for increasing/decreasing logging output, enabling/disabling debug
settings, and even for runtime optimization. The following reflects runtime
configuration usage::
ceph {daemon-type} tell {id or *} injectargs '--{name} {value} [--{name} {value}]'
Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply
the runtime setting to all daemons of a particular type with ``*``, or specify
a specific daemon's ID (i.e., its number or letter). For example, to increase
debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following::
ceph osd tell 0 injectargs '--debug-osd 20 --debug-ms 1'
In your ``ceph.conf`` file, you may use spaces when specifying a
setting name. When specifying a setting name on the command line,
ensure that you use an underscore or hyphen (``_`` or ``-``) between
terms (e.g., ``debug osd`` becomes ``debug-osd``).
Viewing a Configuration at Runtime
==================================
If your Ceph cluster is running, and you would like to see the configuration
settings from a running daemon, execute the following::
ceph --admin-daemon {/path/to/admin/socket} config show | less
The default path for the admin socket for each daemon is::
/var/run/ceph/$cluster-$name.asok
At real time, the metavariables will evaluate to the actual cluster name
and daemon name. For example, if the cluster name is ``ceph`` (it is by default)
and you want to retrieve the configuration for ``osd.0``, use the following::
ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
Running Multiple Clusters
=========================
With Ceph, you can run multiple clusters on the same hardware. Running multiple
clusters provides a higher level of isolation compared to using different pools
on the same cluster with different CRUSH rulesets. A separate cluster will have
separate monitor, OSD and metadata server processes. When running Ceph with
default settings, the default cluster name is ``ceph``, which means you would
save your Ceph configuration file with the file name ``ceph.conf`` in the
``/etc/ceph`` default directory.
When you run multiple clusters, you must name your cluster and save the Ceph
configuration file with the name of the cluster. For example, a cluster named
``openstack`` will have a Ceph configuration file with the file name
``openstack.conf`` in the ``/etc/ceph`` default directory.
.. important:: Cluster names must consist of letters a-z and digits 0-9 only.
Separate clusters imply separate data disks and journals, which are not shared
between clusters. Referring to `Metavariables`_, the ``$cluster`` metavariable
evaluates to the cluster name (i.e., ``openstack`` in the foregoing example).
Various settings use the ``$cluster`` metavariable, including:
- ``keyring``
- ``admin socket``
- ``log file``
- ``pid file``
- ``mon data``
- ``mon cluster log file``
- ``osd data``
- ``osd journal``
- ``mds data``
- ``rgw data``
See `General Settings`_, `OSD Settings`_, `Monitor Settings`_, `MDS Settings`_,
`RGW Settings`_ and `Log Settings`_ for relevant path defaults that use the
``$cluster`` metavariable.
.. _General Settings: ../general-config-ref
.. _OSD Settings: ../osd-config-ref
.. _Monitor Settings: ../mon-config-ref
.. _MDS Settings: ../../../cephfs/mds-config-ref
.. _RGW Settings: ../../../radosgw/config-ref/
.. _Log Settings: ../log-and-debug-ref
When deploying the Ceph configuration file, ensure that you use the cluster name
in your command line syntax. For example::
ssh myserver01 sudo tee /etc/ceph/openstack.conf < /etc/ceph/openstack.conf
When creating default directories or files, you should also use the cluster
name at the appropriate places in the path. For example::
sudo mkdir /var/lib/ceph/osd/openstack-0
sudo mkdir /var/lib/ceph/mon/openstack-a
.. important:: When running monitors on the same host, you should use
different ports. By default, monitors use port 6789. If you already
have monitors using port 6789, use a different port for your other cluster(s).
To invoke a cluster other than the default ``ceph`` cluster, use the
``--cluster=clustername`` option with the ``ceph`` command. For example::
ceph --cluster=openstack health
.. _Hardware Recommendations: ../../../install/hardware-recommendations
.. _hardware recommendations: ../../../install/hardware-recommendations
.. _Network Configuration Reference: ../network-config-ref