2018-08-02 14:32:38 +00:00
|
|
|
|
|
|
|
|
|
.. _orchestrator-cli-module:
|
|
|
|
|
|
|
|
|
|
================
|
|
|
|
|
Orchestrator CLI
|
|
|
|
|
================
|
|
|
|
|
|
|
|
|
|
This module provides a command line interface (CLI) to orchestrator
|
|
|
|
|
modules (ceph-mgr modules which interface with external orchestation services)
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
As the orchestrator CLI unifies different external orchestrators, a common nomenclature
|
2019-01-29 14:51:16 +00:00
|
|
|
|
for the orchestrator module is needed.
|
|
|
|
|
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
| host | hostname (not DNS name) of the |
|
|
|
|
|
| | physical host. Not the podname, |
|
|
|
|
|
| | container name, or hostname inside |
|
|
|
|
|
| | the container. |
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
| service type | The type of the service. e.g., nfs, |
|
|
|
|
|
| | mds, osd, mon, rgw, mgr, iscsi |
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
| service | A logical service, Typically |
|
|
|
|
|
| | comprised of multiple service |
|
|
|
|
|
| | instances on multiple hosts for HA |
|
|
|
|
|
| | |
|
|
|
|
|
| | * ``fs_name`` for mds type |
|
|
|
|
|
| | * ``rgw_zone`` for rgw type |
|
|
|
|
|
| | * ``ganesha_cluster_id`` for nfs type |
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
| service instance | A single instance of a service. |
|
|
|
|
|
| | Usually a daemon, but maybe not |
|
|
|
|
|
| | (e.g., might be a kernel service |
|
|
|
|
|
| | like LIO or knfsd or whatever) |
|
|
|
|
|
| | |
|
|
|
|
|
| | This identifier should |
|
|
|
|
|
| | uniquely identify the instance |
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
| daemon | A running process on a host; use |
|
|
|
|
|
| | “service instance” instead |
|
|
|
|
|
+--------------------------------------+---------------------------------------+
|
|
|
|
|
|
|
|
|
|
The relation between the names is the following:
|
|
|
|
|
|
|
|
|
|
* a service belongs to a service type
|
|
|
|
|
* a service instance belongs to a service type
|
|
|
|
|
* a service instance belongs to a single service group
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-08-02 14:32:38 +00:00
|
|
|
|
Configuration
|
|
|
|
|
=============
|
|
|
|
|
|
2019-07-15 08:52:12 +00:00
|
|
|
|
To enable the orchestrator, please select the orchestrator module to use
|
|
|
|
|
with the ``set backend`` command::
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
|
|
|
|
ceph orchestrator set backend <module>
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
For example, to enable the Rook orchestrator module and use it with the CLI::
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
|
|
|
|
ceph mgr module enable rook
|
|
|
|
|
ceph orchestrator set backend rook
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
You can then check backend is properly configured::
|
2018-08-06 15:20:39 +00:00
|
|
|
|
|
|
|
|
|
ceph orchestrator status
|
|
|
|
|
|
2019-07-15 08:52:12 +00:00
|
|
|
|
Disable the Orchestrator
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
To disable the orchestrator again, use the empty string ``""``::
|
|
|
|
|
|
|
|
|
|
ceph orchestrator set backend ""``
|
2019-07-15 11:25:39 +00:00
|
|
|
|
ceph mgr module disable rook
|
2019-07-15 08:52:12 +00:00
|
|
|
|
|
2018-08-02 14:32:38 +00:00
|
|
|
|
Usage
|
|
|
|
|
=====
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
.. warning::
|
|
|
|
|
|
|
|
|
|
The orchestrator CLI is unfinished and work in progress. Some commands will not
|
|
|
|
|
exist, or return a different result.
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
Orchestrator modules may only implement a subset of the commands listed below.
|
|
|
|
|
Also, the implementation of the commands are orchestrator module dependent and will
|
|
|
|
|
differ between implementations.
|
|
|
|
|
|
|
|
|
|
Status
|
|
|
|
|
~~~~~~
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
ceph orchestrator status
|
|
|
|
|
|
|
|
|
|
Show current orchestrator mode and high-level status (whether the module able
|
|
|
|
|
to talk to it)
|
|
|
|
|
|
|
|
|
|
Also show any in-progress actions.
|
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
Host Management
|
|
|
|
|
~~~~~~~~~~~~~~~
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
List hosts associated with the cluster::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
ceph orchestrator host ls
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
Add and remove hosts::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
ceph orchestrator host add <host>
|
|
|
|
|
ceph orchestrator host rm <host>
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
|
|
|
|
OSD Management
|
|
|
|
|
~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
List Devices
|
|
|
|
|
^^^^^^^^^^^^
|
|
|
|
|
|
2018-08-02 14:32:38 +00:00
|
|
|
|
Print a list of discovered devices, grouped by node and optionally
|
|
|
|
|
filtered to a particular node:
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
ceph orchestrator device ls [--host=...] [--refresh]
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-04-11 08:51:50 +00:00
|
|
|
|
Example::
|
|
|
|
|
|
|
|
|
|
# ceph orchestrator device ls
|
|
|
|
|
Host 192.168.121.206:
|
|
|
|
|
Device Path Type Size Rotates Available Model
|
|
|
|
|
/dev/sdb hdd 50.0G True True ATA/QEMU HARDDISK
|
|
|
|
|
/dev/sda hdd 50.0G True False ATA/QEMU HARDDISK
|
|
|
|
|
|
|
|
|
|
Host 192.168.121.181:
|
|
|
|
|
Device Path Type Size Rotates Available Model
|
|
|
|
|
/dev/sdb hdd 50.0G True True ATA/QEMU HARDDISK
|
|
|
|
|
/dev/sda hdd 50.0G True False ATA/QEMU HARDDISK
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
Output form Ansible orchestrator
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
Create OSDs
|
|
|
|
|
^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Create OSDs on a group of devices on a single host::
|
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
ceph orchestrator osd create <host>:<drive>
|
|
|
|
|
ceph orchestrator osd create -i <path-to-drive-group.json>
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
|
|
|
|
|
2018-12-19 14:17:55 +00:00
|
|
|
|
The output of ``osd create`` is not specified and may vary between orchestrator backends.
|
|
|
|
|
|
2019-07-08 08:29:58 +00:00
|
|
|
|
Where ``drive.group.json`` is a JSON file containing the fields defined in
|
|
|
|
|
:class:`ceph.deployment_utils.drive_group.DriveGroupSpec`
|
2019-01-29 14:51:16 +00:00
|
|
|
|
|
2019-04-11 08:51:50 +00:00
|
|
|
|
Example::
|
|
|
|
|
|
|
|
|
|
# ceph orchestrator osd create 192.168.121.206:/dev/sdc
|
|
|
|
|
{"status": "OK", "msg": "", "data": {"event": "playbook_on_stats", "uuid": "7082f3ba-f5b7-4b7c-9477-e74ca918afcb", "stdout": "\r\nPLAY RECAP *********************************************************************\r\n192.168.121.206 : ok=96 changed=3 unreachable=0 failed=0 \r\n", "counter": 932, "pid": 10294, "created": "2019-05-28T22:22:58.527821", "end_line": 1170, "runner_ident": "083cad3c-8197-11e9-b07a-2016b900e38f", "start_line": 1166, "event_data": {"ignored": 0, "skipped": {"192.168.121.206": 186}, "ok": {"192.168.121.206": 96}, "artifact_data": {}, "rescued": 0, "changed": {"192.168.121.206": 3}, "pid": 10294, "dark": {}, "playbook_uuid": "409364a6-9d49-4e44-8b7b-c28e5b3adf89", "playbook": "add-osd.yml", "failures": {}, "processed": {"192.168.121.206": 1}}, "parent_uuid": "409364a6-9d49-4e44-8b7b-c28e5b3adf89"}}
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
Output form Ansible orchestrator
|
2019-01-29 14:51:16 +00:00
|
|
|
|
|
2018-12-19 14:17:55 +00:00
|
|
|
|
Decommission an OSD
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
ceph orchestrator osd rm <osd-id> [osd-id...]
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
Removes one or more OSDs from the cluster and the host, if the OSDs are marked as
|
2018-12-19 14:17:55 +00:00
|
|
|
|
``destroyed``.
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-04-11 08:51:50 +00:00
|
|
|
|
Example::
|
|
|
|
|
|
|
|
|
|
# ceph orchestrator osd rm 4
|
|
|
|
|
{"status": "OK", "msg": "", "data": {"event": "playbook_on_stats", "uuid": "1a16e631-906d-48e0-9e24-fa7eb593cc0a", "stdout": "\r\nPLAY RECAP *********************************************************************\r\n192.168.121.158 : ok=2 changed=0 unreachable=0 failed=0 \r\n192.168.121.181 : ok=2 changed=0 unreachable=0 failed=0 \r\n192.168.121.206 : ok=2 changed=0 unreachable=0 failed=0 \r\nlocalhost : ok=31 changed=8 unreachable=0 failed=0 \r\n", "counter": 240, "pid": 10948, "created": "2019-05-28T22:26:09.264012", "end_line": 308, "runner_ident": "8c093db0-8197-11e9-b07a-2016b900e38f", "start_line": 301, "event_data": {"ignored": 0, "skipped": {"localhost": 37}, "ok": {"192.168.121.181": 2, "192.168.121.158": 2, "192.168.121.206": 2, "localhost": 31}, "artifact_data": {}, "rescued": 0, "changed": {"localhost": 8}, "pid": 10948, "dark": {}, "playbook_uuid": "a12ec40e-bce9-4bc9-b09e-2d8f76a5be02", "playbook": "shrink-osd.yml", "failures": {}, "processed": {"192.168.121.181": 1, "192.168.121.158": 1, "192.168.121.206": 1, "localhost": 1}}, "parent_uuid": "a12ec40e-bce9-4bc9-b09e-2d8f76a5be02"}}
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
Output form Ansible orchestrator
|
2019-01-29 14:51:16 +00:00
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
..
|
|
|
|
|
Blink Device Lights
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
ceph orchestrator device ident-on <host> <devname>
|
|
|
|
|
ceph orchestrator device ident-off <host> <devname>
|
|
|
|
|
ceph orchestrator device fault-on <host> <devname>
|
|
|
|
|
ceph orchestrator device fault-off <host> <devname>
|
|
|
|
|
|
2018-12-19 14:17:55 +00:00
|
|
|
|
ceph orchestrator osd ident-on {primary,journal,db,wal,all} <osd-id>
|
|
|
|
|
ceph orchestrator osd ident-off {primary,journal,db,wal,all} <osd-id>
|
|
|
|
|
ceph orchestrator osd fault-on {primary,journal,db,wal,all} <osd-id>
|
|
|
|
|
ceph orchestrator osd fault-off {primary,journal,db,wal,all} <osd-id>
|
|
|
|
|
|
|
|
|
|
Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of
|
|
|
|
|
bluestore and ``all`` stands for all devices associated with the osd
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
Monitor and manager management
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
Creates or removes MONs or MGRs from the cluster. Orchestrator may return an
|
|
|
|
|
error if it doesn't know how to do this transition.
|
|
|
|
|
|
|
|
|
|
Update the number of monitor nodes::
|
|
|
|
|
|
|
|
|
|
ceph orchestrator mon update <num> [host, host:network...]
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-03-26 20:52:12 +00:00
|
|
|
|
Each host can optionally specify a network for the monitor to listen on.
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
Update the number of manager nodes::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 17:31:59 +00:00
|
|
|
|
ceph orchestrator mgr update <num> [host...]
|
|
|
|
|
|
|
|
|
|
..
|
2018-11-15 15:29:03 +00:00
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
The host lists are the new full list of mon/mgr hosts
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
specifying hosts is optional for some orchestrator modules
|
|
|
|
|
and mandatory for others (e.g. Ansible).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Service Status
|
2019-01-29 14:51:16 +00:00
|
|
|
|
~~~~~~~~~~~~~~
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
2018-10-30 18:28:02 +00:00
|
|
|
|
Print a list of services known to the orchestrator. The list can be limited to
|
|
|
|
|
services on a particular host with the optional --host parameter and/or
|
|
|
|
|
services of a particular type via optional --type parameter
|
|
|
|
|
(mon, osd, mgr, mds, rgw):
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
2019-07-15 08:52:12 +00:00
|
|
|
|
ceph orchestrator service ls [--host host] [--svc_type type] [--refresh]
|
2018-10-30 18:28:02 +00:00
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
Discover the status of a particular service::
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
2019-07-15 08:52:12 +00:00
|
|
|
|
ceph orchestrator service ls --svc_type type --svc_id <name> [--refresh]
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
|
|
|
|
|
2018-11-15 15:29:03 +00:00
|
|
|
|
Query the status of a particular service instance (mon, osd, mds, rgw). For OSDs
|
2019-09-09 19:36:04 +00:00
|
|
|
|
the id is the numeric OSD ID, for MDS services it is the file system name::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
|
|
|
|
ceph orchestrator service-instance status <type> <instance-name> [--refresh]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI)
|
2019-01-29 14:51:16 +00:00
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
2018-11-15 15:29:03 +00:00
|
|
|
|
The orchestrator is not responsible for configuring the services. Please look into the corresponding
|
|
|
|
|
documentation for details.
|
|
|
|
|
|
|
|
|
|
The ``name`` parameter is an identifier of the group of instances:
|
2019-03-22 13:11:05 +00:00
|
|
|
|
|
2019-09-09 19:36:04 +00:00
|
|
|
|
* a CephFS file system for a group of MDS daemons,
|
2018-11-15 15:29:03 +00:00
|
|
|
|
* a zone name for a group of RGWs
|
|
|
|
|
|
|
|
|
|
Sizing: the ``size`` parameter gives the number of daemons in the cluster
|
2019-09-09 19:36:04 +00:00
|
|
|
|
(e.g. the number of MDS daemons for a particular CephFS file system).
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
Creating/growing/shrinking/removing services::
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 11:56:01 +00:00
|
|
|
|
ceph orchestrator {mds,rgw} update <name> <size> [host…]
|
|
|
|
|
ceph orchestrator {mds,rgw} add <name>
|
2018-11-15 17:22:38 +00:00
|
|
|
|
ceph orchestrator nfs update <name> <size> [host…]
|
|
|
|
|
ceph orchestrator nfs add <name> <pool> [--namespace=<namespace>]
|
2019-01-29 14:51:16 +00:00
|
|
|
|
ceph orchestrator {mds,rgw,nfs} rm <name>
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
2018-12-20 11:56:01 +00:00
|
|
|
|
e.g., ``ceph orchestrator mds update myfs 3 host1 host2 host3``
|
2018-11-15 15:29:03 +00:00
|
|
|
|
|
|
|
|
|
Start/stop/reload::
|
|
|
|
|
|
|
|
|
|
ceph orchestrator service {stop,start,reload} <type> <name>
|
|
|
|
|
|
|
|
|
|
ceph orchestrator service-instance {start,stop,reload} <type> <instance-name>
|
2018-08-02 14:32:38 +00:00
|
|
|
|
|
|
|
|
|
|
2019-01-29 14:51:16 +00:00
|
|
|
|
Current Implementation Status
|
|
|
|
|
=============================
|
|
|
|
|
|
|
|
|
|
This is an overview of the current implementation status of the orchestrators.
|
|
|
|
|
|
|
|
|
|
=================================== ========= ====== ========= =====
|
|
|
|
|
Command Ansible Rook DeepSea SSH
|
|
|
|
|
=================================== ========= ====== ========= =====
|
2019-04-11 08:51:50 +00:00
|
|
|
|
host add ✔️ ⚪ ⚪ ✔️
|
2019-09-10 06:48:59 +00:00
|
|
|
|
host ls ✔️ ✔️ ⚪ ✔️
|
2019-04-11 08:51:50 +00:00
|
|
|
|
host rm ✔️ ⚪ ⚪ ✔️
|
2019-02-19 19:28:50 +00:00
|
|
|
|
mgr update ⚪ ⚪ ⚪ ✔️
|
2019-04-08 09:24:16 +00:00
|
|
|
|
mon update ⚪ ✔️ ⚪ ✔️
|
2019-07-15 08:52:12 +00:00
|
|
|
|
osd create ✔️ ✔️ ⚪ ✔️
|
2019-01-29 14:51:16 +00:00
|
|
|
|
osd device {ident,fault}-{on,off} ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
osd rm ✔️ ⚪ ⚪ ⚪
|
|
|
|
|
device {ident,fault}-(on,off} ⚪ ⚪ ⚪ ⚪
|
2019-02-19 19:28:50 +00:00
|
|
|
|
device ls ✔️ ✔️ ✔️ ✔️
|
2019-01-29 14:51:16 +00:00
|
|
|
|
service ls ⚪ ✔️ ✔️ ⚪
|
|
|
|
|
service-instance status ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
iscsi {stop,start,reload} ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
iscsi add ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
iscsi rm ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
iscsi update ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
mds {stop,start,reload} ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
mds add ⚪ ✔️ ⚪ ⚪
|
|
|
|
|
mds rm ⚪ ✔️ ⚪ ⚪
|
|
|
|
|
mds update ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
nfs {stop,start,reload} ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
nfs add ⚪ ✔️ ⚪ ⚪
|
|
|
|
|
nfs rm ⚪ ✔️ ⚪ ⚪
|
2019-09-10 06:48:59 +00:00
|
|
|
|
nfs update ⚪ ✔️ ⚪ ⚪
|
2019-01-29 14:51:16 +00:00
|
|
|
|
rbd-mirror {stop,start,reload} ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
rbd-mirror add ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
rbd-mirror rm ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
rbd-mirror update ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
rgw {stop,start,reload} ⚪ ⚪ ⚪ ⚪
|
2019-09-10 06:48:59 +00:00
|
|
|
|
rgw add ✔️ ✔️ ⚪ ⚪
|
|
|
|
|
rgw rm ✔️ ✔️ ⚪ ⚪
|
2019-01-29 14:51:16 +00:00
|
|
|
|
rgw update ⚪ ⚪ ⚪ ⚪
|
|
|
|
|
=================================== ========= ====== ========= =====
|
|
|
|
|
|
|
|
|
|
where
|
|
|
|
|
|
|
|
|
|
* ⚪ = not yet implemented
|
|
|
|
|
* ❌ = not applicable
|
|
|
|
|
* ✔ = implemented
|