mirror of
https://github.com/ceph/ceph
synced 2024-12-25 21:03:31 +00:00
442 lines
14 KiB
ReStructuredText
442 lines
14 KiB
ReStructuredText
.. _cephadm_deploying_new_cluster:
|
|
|
|
============================
|
|
Deploying a new Ceph cluster
|
|
============================
|
|
|
|
Cephadm creates a new Ceph cluster by "bootstrapping" on a single
|
|
host, expanding the cluster to encompass any additional hosts, and
|
|
then deploying the needed services.
|
|
|
|
.. highlight:: console
|
|
|
|
.. _cephadm-host-requirements:
|
|
|
|
Requirements
|
|
============
|
|
|
|
- Python 3
|
|
- Systemd
|
|
- Podman or Docker for running containers
|
|
- Time synchronization (such as chrony or NTP)
|
|
- LVM2 for provisioning storage devices
|
|
|
|
Any modern Linux distribution should be sufficient. Dependencies
|
|
are installed automatically by the bootstrap process below.
|
|
|
|
See the section :ref:`Compatibility With Podman
|
|
Versions<cephadm-compatibility-with-podman>` for a table of Ceph versions that
|
|
are compatible with Podman. Not every version of Podman is compatible with
|
|
Ceph.
|
|
|
|
|
|
|
|
.. _get-cephadm:
|
|
|
|
Install cephadm
|
|
===============
|
|
|
|
There are two ways to install ``cephadm``:
|
|
|
|
#. a :ref:`curl-based installation<cephadm_install_curl>` method
|
|
#. :ref:`distribution-specific installation methods<cephadm_install_distros>`
|
|
|
|
|
|
.. _cephadm_install_curl:
|
|
|
|
curl-based installation
|
|
-----------------------
|
|
|
|
* Use ``curl`` to fetch the most recent version of the
|
|
standalone script.
|
|
|
|
.. prompt:: bash #
|
|
:substitutions:
|
|
|
|
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/|stable-release|/src/cephadm/cephadm
|
|
|
|
Make the ``cephadm`` script executable:
|
|
|
|
.. prompt:: bash #
|
|
|
|
chmod +x cephadm
|
|
|
|
This script can be run directly from the current directory:
|
|
|
|
.. prompt:: bash #
|
|
|
|
./cephadm <arguments...>
|
|
|
|
* Although the standalone script is sufficient to get a cluster started, it is
|
|
convenient to have the ``cephadm`` command installed on the host. To install
|
|
the packages that provide the ``cephadm`` command, run the following
|
|
commands:
|
|
|
|
.. prompt:: bash #
|
|
:substitutions:
|
|
|
|
./cephadm add-repo --release |stable-release|
|
|
./cephadm install
|
|
|
|
Confirm that ``cephadm`` is now in your PATH by running ``which``:
|
|
|
|
.. prompt:: bash #
|
|
|
|
which cephadm
|
|
|
|
A successful ``which cephadm`` command will return this:
|
|
|
|
.. code-block:: bash
|
|
|
|
/usr/sbin/cephadm
|
|
|
|
.. _cephadm_install_distros:
|
|
|
|
distribution-specific installations
|
|
-----------------------------------
|
|
|
|
.. important:: The methods of installing ``cephadm`` in this section are distinct from the curl-based method above. Use either the curl-based method above or one of the methods in this section, but not both the curl-based method and one of these.
|
|
|
|
Some Linux distributions may already include up-to-date Ceph packages. In
|
|
that case, you can install cephadm directly. For example:
|
|
|
|
In Ubuntu:
|
|
|
|
.. prompt:: bash #
|
|
|
|
apt install -y cephadm
|
|
|
|
In CentOS Stream:
|
|
|
|
.. prompt:: bash #
|
|
:substitutions:
|
|
|
|
dnf search release-ceph
|
|
dnf install --assumeyes centos-release-ceph-|stable-release|
|
|
dnf install --assumeyes cephadm
|
|
|
|
In Fedora:
|
|
|
|
.. prompt:: bash #
|
|
|
|
dnf -y install cephadm
|
|
|
|
In SUSE:
|
|
|
|
.. prompt:: bash #
|
|
|
|
zypper install -y cephadm
|
|
|
|
|
|
|
|
Bootstrap a new cluster
|
|
=======================
|
|
|
|
What to know before you bootstrap
|
|
---------------------------------
|
|
|
|
The first step in creating a new Ceph cluster is running the ``cephadm
|
|
bootstrap`` command on the Ceph cluster's first host. The act of running the
|
|
``cephadm bootstrap`` command on the Ceph cluster's first host creates the Ceph
|
|
cluster's first "monitor daemon", and that monitor daemon needs an IP address.
|
|
You must pass the IP address of the Ceph cluster's first host to the ``ceph
|
|
bootstrap`` command, so you'll need to know the IP address of that host.
|
|
|
|
.. note:: If there are multiple networks and interfaces, be sure to choose one
|
|
that will be accessible by any host accessing the Ceph cluster.
|
|
|
|
Running the bootstrap command
|
|
-----------------------------
|
|
|
|
Run the ``ceph bootstrap`` command:
|
|
|
|
.. prompt:: bash #
|
|
|
|
cephadm bootstrap --mon-ip *<mon-ip>*
|
|
|
|
This command will:
|
|
|
|
* Create a monitor and manager daemon for the new cluster on the local
|
|
host.
|
|
* Generate a new SSH key for the Ceph cluster and add it to the root
|
|
user's ``/root/.ssh/authorized_keys`` file.
|
|
* Write a copy of the public key to ``/etc/ceph/ceph.pub``.
|
|
* Write a minimal configuration file to ``/etc/ceph/ceph.conf``. This
|
|
file is needed to communicate with the new cluster.
|
|
* Write a copy of the ``client.admin`` administrative (privileged!)
|
|
secret key to ``/etc/ceph/ceph.client.admin.keyring``.
|
|
* Add the ``_admin`` label to the bootstrap host. By default, any host
|
|
with this label will (also) get a copy of ``/etc/ceph/ceph.conf`` and
|
|
``/etc/ceph/ceph.client.admin.keyring``.
|
|
|
|
Further information about cephadm bootstrap
|
|
-------------------------------------------
|
|
|
|
The default bootstrap behavior will work for most users. But if you'd like
|
|
immediately to know more about ``cephadm bootstrap``, read the list below.
|
|
|
|
Also, you can run ``cephadm bootstrap -h`` to see all of ``cephadm``'s
|
|
available options.
|
|
|
|
* By default, Ceph daemons send their log output to stdout/stderr, which is picked
|
|
up by the container runtime (docker or podman) and (on most systems) sent to
|
|
journald. If you want Ceph to write traditional log files to ``/var/log/ceph/$fsid``,
|
|
use the ``--log-to-file`` option during bootstrap.
|
|
|
|
* Larger Ceph clusters perform better when (external to the Ceph cluster)
|
|
public network traffic is separated from (internal to the Ceph cluster)
|
|
cluster traffic. The internal cluster traffic handles replication, recovery,
|
|
and heartbeats between OSD daemons. You can define the :ref:`cluster
|
|
network<cluster-network>` by supplying the ``--cluster-network`` option to the ``bootstrap``
|
|
subcommand. This parameter must define a subnet in CIDR notation (for example
|
|
``10.90.90.0/24`` or ``fe80::/64``).
|
|
|
|
* ``cephadm bootstrap`` writes to ``/etc/ceph`` the files needed to access
|
|
the new cluster. This central location makes it possible for Ceph
|
|
packages installed on the host (e.g., packages that give access to the
|
|
cephadm command line interface) to find these files.
|
|
|
|
Daemon containers deployed with cephadm, however, do not need
|
|
``/etc/ceph`` at all. Use the ``--output-dir *<directory>*`` option
|
|
to put them in a different directory (for example, ``.``). This may help
|
|
avoid conflicts with an existing Ceph configuration (cephadm or
|
|
otherwise) on the same host.
|
|
|
|
* You can pass any initial Ceph configuration options to the new
|
|
cluster by putting them in a standard ini-style configuration file
|
|
and using the ``--config *<config-file>*`` option. For example::
|
|
|
|
$ cat <<EOF > initial-ceph.conf
|
|
[global]
|
|
osd crush chooseleaf type = 0
|
|
EOF
|
|
$ ./cephadm bootstrap --config initial-ceph.conf ...
|
|
|
|
* The ``--ssh-user *<user>*`` option makes it possible to choose which SSH
|
|
user cephadm will use to connect to hosts. The associated SSH key will be
|
|
added to ``/home/*<user>*/.ssh/authorized_keys``. The user that you
|
|
designate with this option must have passwordless sudo access.
|
|
|
|
* If you are using a container on an authenticated registry that requires
|
|
login, you may add the argument:
|
|
|
|
* ``--registry-json <path to json file>``
|
|
|
|
example contents of JSON file with login info::
|
|
|
|
{"url":"REGISTRY_URL", "username":"REGISTRY_USERNAME", "password":"REGISTRY_PASSWORD"}
|
|
|
|
Cephadm will attempt to log in to this registry so it can pull your container
|
|
and then store the login info in its config database. Other hosts added to
|
|
the cluster will then also be able to make use of the authenticated registry.
|
|
|
|
* See :ref:`cephadm-deployment-scenarios` for additional examples for using ``cephadm bootstrap``.
|
|
|
|
.. _cephadm-enable-cli:
|
|
|
|
Enable Ceph CLI
|
|
===============
|
|
|
|
Cephadm does not require any Ceph packages to be installed on the
|
|
host. However, we recommend enabling easy access to the ``ceph``
|
|
command. There are several ways to do this:
|
|
|
|
* The ``cephadm shell`` command launches a bash shell in a container
|
|
with all of the Ceph packages installed. By default, if
|
|
configuration and keyring files are found in ``/etc/ceph`` on the
|
|
host, they are passed into the container environment so that the
|
|
shell is fully functional. Note that when executed on a MON host,
|
|
``cephadm shell`` will infer the ``config`` from the MON container
|
|
instead of using the default configuration. If ``--mount <path>``
|
|
is given, then the host ``<path>`` (file or directory) will appear
|
|
under ``/mnt`` inside the container:
|
|
|
|
.. prompt:: bash #
|
|
|
|
cephadm shell
|
|
|
|
* To execute ``ceph`` commands, you can also run commands like this:
|
|
|
|
.. prompt:: bash #
|
|
|
|
cephadm shell -- ceph -s
|
|
|
|
* You can install the ``ceph-common`` package, which contains all of the
|
|
ceph commands, including ``ceph``, ``rbd``, ``mount.ceph`` (for mounting
|
|
CephFS file systems), etc.:
|
|
|
|
.. prompt:: bash #
|
|
:substitutions:
|
|
|
|
cephadm add-repo --release |stable-release|
|
|
cephadm install ceph-common
|
|
|
|
Confirm that the ``ceph`` command is accessible with:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph -v
|
|
|
|
|
|
Confirm that the ``ceph`` command can connect to the cluster and also
|
|
its status with:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph status
|
|
|
|
Adding Hosts
|
|
============
|
|
|
|
Next, add all hosts to the cluster by following :ref:`cephadm-adding-hosts`.
|
|
|
|
By default, a ``ceph.conf`` file and a copy of the ``client.admin`` keyring
|
|
are maintained in ``/etc/ceph`` on all hosts with the ``_admin`` label, which is initially
|
|
applied only to the bootstrap host. We usually recommend that one or more other hosts be
|
|
given the ``_admin`` label so that the Ceph CLI (e.g., via ``cephadm shell``) is easily
|
|
accessible on multiple hosts. To add the ``_admin`` label to additional host(s),
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph orch host label add *<host>* _admin
|
|
|
|
Adding additional MONs
|
|
======================
|
|
|
|
A typical Ceph cluster has three or five monitor daemons spread
|
|
across different hosts. We recommend deploying five
|
|
monitors if there are five or more nodes in your cluster.
|
|
|
|
Please follow :ref:`deploy_additional_monitors` to deploy additional MONs.
|
|
|
|
Adding Storage
|
|
==============
|
|
|
|
To add storage to the cluster, either tell Ceph to consume any
|
|
available and unused device:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph orch apply osd --all-available-devices
|
|
|
|
See :ref:`cephadm-deploy-osds` for more detailed instructions.
|
|
|
|
Enabling OSD memory autotuning
|
|
------------------------------
|
|
|
|
.. warning:: By default, cephadm enables ``osd_memory_target_autotune`` on bootstrap, with ``mgr/cephadm/autotune_memory_target_ratio`` set to ``.7`` of total host memory.
|
|
|
|
See :ref:`osd_autotune`.
|
|
|
|
To deploy hyperconverged Ceph with TripleO, please refer to the TripleO documentation: `Scenario: Deploy Hyperconverged Ceph <https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/cephadm.html#scenario-deploy-hyperconverged-ceph>`_
|
|
|
|
In other cases where the cluster hardware is not exclusively used by Ceph (hyperconverged),
|
|
reduce the memory consumption of Ceph like so:
|
|
|
|
.. prompt:: bash #
|
|
|
|
# hyperconverged only:
|
|
ceph config set mgr mgr/cephadm/autotune_memory_target_ratio 0.2
|
|
|
|
Then enable memory autotuning:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph config set osd osd_memory_target_autotune true
|
|
|
|
|
|
Using Ceph
|
|
==========
|
|
|
|
To use the *Ceph Filesystem*, follow :ref:`orchestrator-cli-cephfs`.
|
|
|
|
To use the *Ceph Object Gateway*, follow :ref:`cephadm-deploy-rgw`.
|
|
|
|
To use *NFS*, follow :ref:`deploy-cephadm-nfs-ganesha`
|
|
|
|
To use *iSCSI*, follow :ref:`cephadm-iscsi`
|
|
|
|
.. _cephadm-deployment-scenarios:
|
|
|
|
Different deployment scenarios
|
|
==============================
|
|
|
|
Single host
|
|
-----------
|
|
|
|
To configure a Ceph cluster to run on a single host, use the
|
|
``--single-host-defaults`` flag when bootstrapping. For use cases of this, see
|
|
:ref:`one-node-cluster`.
|
|
|
|
The ``--single-host-defaults`` flag sets the following configuration options::
|
|
|
|
global/osd_crush_chooseleaf_type = 0
|
|
global/osd_pool_default_size = 2
|
|
mgr/mgr_standby_modules = False
|
|
|
|
For more information on these options, see :ref:`one-node-cluster` and
|
|
``mgr_standby_modules`` in :ref:`mgr-administrator-guide`.
|
|
|
|
.. _cephadm-airgap:
|
|
|
|
Deployment in an isolated environment
|
|
-------------------------------------
|
|
|
|
You might need to install cephadm in an environment that is not connected
|
|
directly to the internet (such an environment is also called an "isolated
|
|
environment"). This can be done if a custom container registry is used. Either
|
|
of two kinds of custom container registry can be used in this scenario: (1) a
|
|
Podman-based or Docker-based insecure registry, or (2) a secure registry.
|
|
|
|
The practice of installing software on systems that are not connected directly
|
|
to the internet is called "airgapping" and registries that are not connected
|
|
directly to the internet are referred to as "airgapped".
|
|
|
|
Make sure that your container image is inside the registry. Make sure that you
|
|
have access to all hosts that you plan to add to the cluster.
|
|
|
|
#. Run a local container registry:
|
|
|
|
.. prompt:: bash #
|
|
|
|
podman run --privileged -d --name registry -p 5000:5000 -v /var/lib/registry:/var/lib/registry --restart=always registry:2
|
|
|
|
#. If you are using an insecure registry, configure Podman or Docker with the
|
|
hostname and port where the registry is running.
|
|
|
|
.. note:: You must repeat this step for every host that accesses the local
|
|
insecure registry.
|
|
|
|
#. Push your container image to your local registry. Here are some acceptable
|
|
kinds of container images:
|
|
|
|
* Ceph container image. See :ref:`containers`.
|
|
* Prometheus container image
|
|
* Node exporter container image
|
|
* Grafana container image
|
|
* Alertmanager container image
|
|
|
|
#. Create a temporary configuration file to store the names of the monitoring
|
|
images. (See :ref:`cephadm_monitoring-images`):
|
|
|
|
.. prompt:: bash $
|
|
|
|
cat <<EOF > initial-ceph.conf
|
|
|
|
::
|
|
|
|
[mgr]
|
|
mgr/cephadm/container_image_prometheus = *<hostname>*:5000/prometheus
|
|
mgr/cephadm/container_image_node_exporter = *<hostname>*:5000/node_exporter
|
|
mgr/cephadm/container_image_grafana = *<hostname>*:5000/grafana
|
|
mgr/cephadm/container_image_alertmanager = *<hostname>*:5000/alertmanger
|
|
|
|
#. Run bootstrap using the ``--image`` flag and pass the name of your
|
|
container image as the argument of the image flag. For example:
|
|
|
|
.. prompt:: bash #
|
|
|
|
cephadm --image *<hostname>*:5000/ceph/ceph bootstrap --mon-ip *<mon-ip>*
|
|
|
|
.. _cluster network: ../rados/configuration/network-config-ref#cluster-network
|