ceph/doc/dev/config.rst

=================================
 Configuration Management System
=================================

The configuration management system exists to provide every daemon with the
proper configuration information. The configuration can be viewed as a set of
key-value pairs.

How can the configuration be set? Well, there are several sources:

 - the ceph configuration file, usually named ceph.conf
 - command line arguments::

    --debug-ms=1
    --debug-monc=10

   etc.
 - arguments injected at runtime using ``injectargs`` or ``config set``


The Configuration File
======================

Most configuration settings originate in the Ceph configuration file.

How do we find the configuration file? Well, in order, we check:

 - the default locations
 - the environment variable ``CEPH_CONF``
 - the command line argument ``-c``

Each stanza of the configuration file describes the key-value pairs that will be in
effect for a particular subset of the daemons. The "global" stanza applies to
everything. The "mon", "osd", and "mds" stanzas specify settings to take effect
for all monitors, all OSDs, and all mds servers, respectively.  A stanza of the
form ``mon.$name``, ``osd.$name``, or ``mds.$name`` gives settings for the monitor, OSD, or
MDS of that name, respectively. Configuration values that appear later in the
file win over earlier ones.

A sample configuration file can be found in src/sample.ceph.conf.


Metavariables
=============

The configuration system allows any configuration value to be
substituted into another value using the ``$varname`` syntax, similar
to how bash shell expansion works.

A few additional special metavariables are also defined:

 - $host: expands to the current hostname
 - $type: expands to one of "mds", "osd", "mon", or "client"
 - $id: expands to the daemon identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``; for ``client.admin``, it would be ``admin``.
 - $num: same as $id
 - $name: expands to $type.$id


Reading configuration values
====================================================

There are two ways for Ceph code to get configuration values. One way is to
read it directly from a variable named ``g_conf``, or equivalently,
``g_ceph_ctx->_conf``. The other is to register an observer that will be called
every time the relevant configuration values change. This observer will be
called soon after the initial configuration is read, and every time after that
when one of the relevant values changes. Each observer tracks a set of keys
and is invoked only when one of the relevant keys changes.

The interface to implement is found in ``common/config_obs.h``.

The observer method should be preferred in new code because

 - It is more flexible, allowing the code to do whatever reinitialization needs
   to be done to implement the new configuration value.
 - It is the only way to create a std::string configuration variable that can
   be changed by injectargs.
 - Even for int-valued configuration options, changing the values in one thread
   while another thread is reading them can lead to subtle and
   impossible-to-diagnose bugs.

For these reasons, reading directly from ``g_conf`` should be considered deprecated
and not done in new code.  Do not ever alter ``g_conf``.

Changing configuration values
====================================================

Configuration values can be changed by calling ``g_conf()->set_val``. After changing
the configuration, you should call ``g_conf()->apply_changes`` to re-run all the
affected configuration observers. For convenience, you can call
``g_conf()->set_val_or_die`` to make a configuration change which you think should
never fail.

``injectargs``, ``parse_argv``, and ``parse_env`` are three other functions which modify
the configuration. Just like with set_val, you should call apply_changes after
calling these functions to make sure your changes get applied.


Defining config options
=======================

Config options are defined in ``common/options/*.yaml.in``. The options are categorized
by their consumers. If an option is only used by ceph-osd, it should go to
``osd.yaml.in``. All the ``.yaml.in`` files are translated into ``.cc`` and ``.h`` files
at build time by ``y2c.py``.

Each option is represented using a YAML mapping (dictionary). A typical option looks like

.. code-block:: yaml

   - name: public_addr
     type: addr
     level: basic
     desc: public-facing address to bind to
     long_desc: The IP address for the public (front-side) network.
       Set for each daemon.
     services:
     - mon
     - mds
     - osd
     - mgr
     flags:
     - startup
     with_legacy: true

In which, following keys are allowed:

level
-----

The ``level`` property of an option is an indicator for the probability the
option is adjusted by an operator or a developer:

.. describe:: basic

   for basic config options that a normal operator is likely to adjust.

.. describe:: advanced

   for options that an operator *can* adjust, but should not touch unless they
   understand what they are doing. Adjusting advanced options poorly can lead to
   problems (performance or even data loss) if done incorrectly.

.. describe:: dev

   for options in place for use by developers only, either for testing purposes,
   or to describe constants that no user should adjust but we prefer not to compile
   into the code.

``desc``, ``long_desc`` and ``fmt_desc``
----------------------------------------

.. describe:: desc

   Short description of the option. Sentence fragment. e.g.

   .. code-block:: yaml

      desc: Default checksum algorithm to use

.. describe:: long_desc

   The long description is complete sentences, perhaps even multiple
   paragraphs, and may include other detailed information or notes. e.g.

   .. code-block:: yaml

      long_desc: crc32c, xxhash32, and xxhash64 are available.  The _16 and _8 variants use
        only a subset of the bits for more compact (but less reliable) checksumming.

.. describe:: fmt_desc

   The description formatted using reStructuredText. This property is
   only used by the ``confval`` directive to render an option in the
   document. e.g.:

   .. code-block:: yaml

      fmt_desc: The interval for "deep" scrubbing (fully reading all data). The
        ``osd_scrub_load_threshold`` does not affect this setting.

Default values
--------------

There is a default value for every config option. In some cases, there may
also be a *daemon default* that only applies to code that declares itself
as a daemon (in this case, the regular default only applies to non-daemons). Like:

.. code-block:: yaml

   default: crc32c

Some literal postfixes are allowed when options with type of ``float``, ``size``
and ``secs``, like:

.. code-block:: yaml

   - name: mon_scrub_interval
     type: secs
     default: 1_day
   - name: osd_journal_size
     type: size
     default: 5_K

For better readability, it is encouraged to use these literal postfixes when
adding or updating the default value for an option.

Service
-------

Service is a component name, like "common", "osd", "rgw", "mds", etc. It may
be a list of components, like:

.. code-block:: yaml

   services:
   - mon
   - mds
   - osd
   - mgr

For example, the rocksdb options affect both the osd and mon. If an option is put
into a service specific ``.yaml.in`` file, the corresponding service is added to
its ``services`` property automatically. For instance, ``osd_scrub_begin_hour``
option is located in ``osd.yaml.in``, even its ``services`` is not specified
explicitly in this file, this property still contains ``osd``.

Tags
----

Tags identify options across services that relate in some way. For example:

network
  options affecting network configuration
mkfs
  options that only matter at mkfs time

Like:

.. code-block:: yaml

   tags:
   - network

Enums
-----

For options with a defined set of allowed values:

.. code-block:: yaml

   enum_values:
   - none
   - crc32c
   - crc32c_16
   - crc32c_8
   - xxhash32
   - xxhash64

Flags
-----

.. describe:: runtime

   the value can be updated at runtime

.. describe:: no_mon_update

   Daemons/clients do not pull this value from the monitor config database.  We
   disallow setting this option via ``ceph config set ...``.  This option should
   be configured via ``ceph.conf`` or via the command line.

.. describe:: startup

   option takes effect only during daemon startup

.. describe:: cluster_create

   option only affects cluster creation

.. describe:: create

   option only affects daemon creation