ceph/doc/install/manual-deployment.rst

===================
 Manual Deployment
===================

All Ceph clusters require at least one monitor, and at least as many OSDs as
copies of an object stored on the cluster.  Bootstrapping the initial monitor(s)
is the first step in deploying a Ceph Storage Cluster. Monitor deployment also
sets important criteria for the entire cluster, such as the number of replicas
for pools, the number of placement groups per OSD, the heart beat intervals,
whether authentication is required, etc. Most of these values are set by
default, so it's useful to know about them when setting up your cluster for
production.

Following the same configuration as `Installation (Quick)`_, we will set up a
cluster with ``node1`` as  the monitor node, and ``node2`` and ``node3`` for 
OSD nodes.



.. ditaa:: 
           /------------------\         /----------------\
           |    Admin Node    |         |     node1      |
           |                  +-------->+                |
           |                  |         | cCCC           |
           \---------+--------/         \----------------/
                     |
                     |                  /----------------\
                     |                  |     node2      |
                     +----------------->+                |
                     |                  | cCCC           |
                     |                  \----------------/
                     |
                     |                  /----------------\
                     |                  |     node3      |
                     +----------------->|                |
                                        | cCCC           |
                                        \----------------/


Monitor Bootstrapping
=====================

Bootstrapping a monitor (a Ceph Storage Cluster, in theory) requires
a number of things:

- **Unique Identifier:** The ``fsid`` is a unique identifier for the cluster, 
  and stands for File System ID from the days when the Ceph Storage Cluster was 
  principally for the Ceph Filesystem. Ceph now supports native interfaces, 
  block device, and object storage gateway interfaces too, so ``fsid`` is a bit 
  of a misnomer.

- **Cluster Name:** All Ceph clusters have a name, but the default name is
  ``ceph``. Setting the cluster name to something other than ``ceph`` is 
  especially useful when you are working with multiple clusters and you need to 
  clearly understand which cluster your are working with. For example, when you 
  run multiple clusters on the same hardware, or if you are running Ceph in 
  federated architectures, the cluster name defines which cluster you are
  using (e.g., ``us-west``, ``us-east``). 
  
- **Monitor Name:** Each monitor instance within a cluster has a unique name. 
  In common practice, the Ceph Monitor name is the host name (we recommend one
  Ceph Monitor per host, and no commingling of Ceph OSD Daemons with 
  Ceph Monitors).

- **Cluster Map:** Bootstrapping the initial monitor(s) requires you to 
  generate a cluster map. The cluster map requires the ``fsid``, the cluster 
  name (or uses the default), and at least one host name and its IP address.

- **Monitor Keyring**: Monitors communicate with each other via a 
  secret key. You must generate a keyring with a monitor secret and provide 
  it when bootstrapping the initial monitor(s).
  
- **Administrator Keyring**: To use the ``ceph`` CLI tools, you must have
  a ``client.admin`` user. So you must generate the admin user and keyring,
  and you must also add the ``client.admin`` user to the monitor keyring.

The foregoing requirements do not imply the creation of a Ceph Configuration 
file. However, as a best practice, we recommend creating a Ceph configuration 
file and populating it with the ``fsid``, the ``mon initial members`` and the
``mon host`` settings. 

You can get and set all of the monitor settings at runtime as well. However,
a Ceph Configuration file may contain only those settings that override the 
default values. When you add settings to a Ceph configuration file, these
settings override the default settings. Maintaining those settings in a 
Ceph configuration file makes it easier to maintain your cluster.

The procedure is as follows:


#. Log in to the initial monitor node(s)::

	ssh {hostname}
	ssh node1


#. Ensure you have a directory for the Ceph configuration file. By default, 
   Ceph uses ``/etc/ceph``. When you install ``ceph``, the directory will 
   get created automatically. ::

	ls /etc/ceph   


#. Create a Ceph configuration file (optional). By default, Ceph uses 
   ``ceph.conf``, where ``ceph`` reflects the cluster name. ::

	sudo vim /etc/ceph/ceph.conf


#. Generate a unique ID (i.e., ``fsid``) for your cluster. :: 

	uuidgen
	

#. Add the unique ID to your Ceph configuration file (optional). :: 

	fsid = {UUID}
	fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993


#. Add the initial monitor(s) to your Ceph configuration file (optional). :: 

	mon initial members = {hostname}[,{hostname}]
	mon initial members = node1


#. Add the IP address(es) of the initial monitor(s) to your Ceph configuration 
   file and save the file (optional). :: 

	mon host = {ip-address}[,{ip-address}]
	mon host = 192.168.0.1


#. Create a keyring for your cluster and generate a monitor secret key. ::

	ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'


#. Generate an administrator keyring, generate a ``client.admin`` user and add
   the user to the keyring. :: 

	ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'


#. Add the ``client.admin`` key to the ``ceph.mon.keyring``. :: 

	ceph-authtool ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring


#. Generate a monitor map using the hostname(s), host IP address(es) and the FSID. 
   Save it as ``/tmp/monmap``:: 

	monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap
	monmaptool --create --add node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap


#. Create a default data directory (or directories) on the monitor host(s). ::

	sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}
	sudo mkdir /var/lib/ceph/mon/ceph-node1


#. Populate the monitor daemon(s) with the monitor map and keyring. ::

	ceph-mon --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
	ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring


#. Consider settings for a Ceph configuration file. Common settings include 
   the following::

	[global]
	fsid = {cluster-id}
	mon initial members = {hostname}[, {hostname}]
	mon host = {ip-address}[, {ip-address}]
	public network = {network}[, {network}]
	cluster network = {network}[, {network}]
	auth cluster required = cephx
	auth service required = cephx
	auth client required = cephx
	osd journal size = {n}
	filestore xattr use omap = true
	osd pool default size = {n}  # Write an object n times.
	osd pool default min size = {n} # Allow writing n copy in a degraded state.
	osd pool default pg num = {n}
	osd pool default pgp num = {n}	
	osd crush chooseleaf type = {n}

   In the foregoing example, the ``[global]`` section of the configuration might
   look like this::

	[global]
	fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
	mon initial members = node1
	mon host = 192.168.0.1
	public network = 192.168.0.0/24
	auth cluster required = cephx
	auth service required = cephx
	auth client required = cephx
	osd journal size = 1024
	filestore xattr use omap = true
	osd pool default size = 2
	osd pool default min size = 1
	osd pool default pg num = 333
	osd pool default pgp num = 333	
	osd crush chooseleaf type = 1


#. Start the monitor(s).

   For Debian/Ubuntu, use Upstart::

	sudo start ceph-mon id=node1

   For CentOS/RHEL, use sysvinit::

	sudo /etc/init.d/ceph start mon.node1


#. Verify that Ceph created the default pools. ::

	ceph osd lspools

   You should see output like this::

	0 data,1 metadata,2 rbd,


#. Verify that the monitor is running. :: 

	ceph -s

   You should see output that looks something like this:: 

	cluster a7f64266-0894-4f1e-a635-d0aeaca0e993
	  health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds
	  monmap e1: 1 mons at {node1=192.168.0.1:6789/0}, election epoch 1, quorum 0 node1
	  osdmap e1: 0 osds: 0 up, 0 in
	  pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects
	     0 kB used, 0 kB / 0 kB avail
	     192 creating



Adding OSDs
===========

Once you have your initial monitor(s) running, you should add OSDs. Your cluster
cannot reach an ``active + clean`` state until you have enough OSDs to handle the
number of copies of an object (e.g., ``osd pool default size = 2`` requires at
least two OSDs). After bootstrapping your monitor, your cluster has a default
CRUSH map; however, the CRUSH map doesn't have any Ceph OSD Daemons mapped to 
a Ceph Node.

To create the first two OSDs, perform this procedure on ``node2`` and ``node3``:


#. Generate a UUID for the OSD. ::

	uuidgen


#. Create the OSD. If no UUID is given, it will be set automatically when the 
   OSD starts up. The following command will output the OSD number, which you 
   will need for subsequent steps. ::
	
	ceph osd create [{uuid}]


#. Create the default directory on your new OSD. :: 

	ssh {new-osd-host}
	sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
	

#. If the OSD is for a drive other than the OS drive, prepare it 
   for use with Ceph, and mount it to the directory you just created:: 

	ssh {new-osd-host}
	sudo mkfs -t {fstype} /dev/{drive}
	sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}

	
#. Initialize the OSD data directory. :: 

	ssh {new-osd-host}
	sudo ceph-osd -i {osd-num} --mkfs --mkkey
	
   The directory must be empty before you can run ``ceph-osd``.


#. Register the OSD authentication key. The value of ``ceph`` for 
   ``ceph-{osd-num}`` in the path is the ``$cluster-$id``.  If your 
   cluster name differs from ``ceph``, use your cluster name instead.::

	sudo ceph auth add osd.{osd-num} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring


#. Add your Ceph Node to the CRUSH map. ::

	ceph osd crush add-bucket {hostname} host
	ceph osd crush add-bucket node1 host


#. Place the Ceph Node under the root ``default``. ::

	ceph osd crush move node1 root=default


#. Add the OSD to the CRUSH map so that it can begin receiving data. You may
   also decompile the CRUSH map, add the OSD to the device list, add the host as a
   bucket (if it's not already in the CRUSH map), add the device as an item in the
   host, assign it a weight, recompile it and set it. ::

	ceph osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...]
	ceph osd crush add osd.0 1.0 host=node1


#. After you add an OSD to Ceph, the OSD is in your configuration. However, 
   it is not yet running. The OSD is ``down`` and ``in``. You must start 
   your new OSD before it can begin receiving data.

   For Debian/Ubuntu, use Upstart::

	sudo start ceph-osd id={osd-num}
	sudo start ceph-osd id=0
	sudo start ceph-osd id=1

   For CentOS/RHEL, use sysvinit::

	sudo /etc/init.d/ceph start osd.{osd-num}
	sudo /etc/init.d/ceph start osd.0
	sudo /etc/init.d/ceph start osd.1

   Once you start your OSD, it is ``up`` and ``in``.


Summary
=======

Once you have your monitor and two OSDs up and running, you can watch the
placement groups peer by executing the following:: 

	ceph -w

To view the tree, execute the following:: 

	ceph osd tree
	
You should see output that looks something like this:: 

	# id	weight	type name	up/down	reweight
	-1	2	root default
	-2	2		host node1
	0	1			osd.0	up	1
	-3	1		host node2
	1	1			osd.1	up	1	

To add (or remove) additional monitors, see `Add/Remove Monitors`_. 
To add (or remove) additional Ceph OSD Daemons, see `Add/Remove OSDs`_.

.. _Installation (Quick): ../../start
.. _Add/Remove Monitors: ../../rados/operations/add-or-rm-mons
.. _Add/Remove OSDs: ../../rados/operations/add-or-rm-osds