.. _orchestrator-cli-module: ================ Orchestrator CLI ================ This module provides a command line interface (CLI) to orchestrator modules (ceph-mgr modules which interface with external orchestation services) As the orchestrator CLI unifies different external orchestrators, a common nomenclature for the orchestrator module is needed. +--------------------------------------+---------------------------------------+ | host | hostname (not DNS name) of the | | | physical host. Not the podname, | | | container name, or hostname inside | | | the container. | +--------------------------------------+---------------------------------------+ | service type | The type of the service. e.g., nfs, | | | mds, osd, mon, rgw, mgr, iscsi | +--------------------------------------+---------------------------------------+ | service | A logical service, Typically | | | comprised of multiple service | | | instances on multiple hosts for HA | | | | | | * ``fs_name`` for mds type | | | * ``rgw_zone`` for rgw type | | | * ``ganesha_cluster_id`` for nfs type | +--------------------------------------+---------------------------------------+ | service instance | A single instance of a service. | | |  Usually a daemon, but maybe not | | | (e.g., might be a kernel service | | | like LIO or knfsd or whatever) | | | | | | This identifier should | | | uniquely identify the instance | +--------------------------------------+---------------------------------------+ | daemon | A running process on a host; use | | | “service instance” instead | +--------------------------------------+---------------------------------------+ The relation between the names is the following: * a service belongs to a service type * a service instance belongs to a service type * a service instance belongs to a single service group Configuration ============= You can select the orchestrator module to use with the ``set backend`` command:: ceph orchestrator set backend For example, to enable the Rook orchestrator module and use it with the CLI:: ceph mgr module enable rook ceph orchestrator set backend rook You can then check backend is properly configured:: ceph orchestrator status Usage ===== .. warning:: The orchestrator CLI is unfinished and work in progress. Some commands will not exist, or return a different result. .. note:: Orchestrator modules may only implement a subset of the commands listed below. Also, the implementation of the commands are orchestrator module dependent and will differ between implementations. Status ~~~~~~ :: ceph orchestrator status Show current orchestrator mode and high-level status (whether the module able to talk to it) Also show any in-progress actions. Host Management ~~~~~~~~~~~~~~~ List hosts associated with the cluster:: ceph orchestrator host ls Add and remove hosts:: ceph orchestrator host add ceph orchestrator host rm OSD Management ~~~~~~~~~~~~~~ List Devices ^^^^^^^^^^^^ Print a list of discovered devices, grouped by node and optionally filtered to a particular node: :: ceph orchestrator device ls [--host=...] [--refresh] Example:: # ceph orchestrator device ls Host 192.168.121.206: Device Path Type Size Rotates Available Model /dev/sdb hdd 50.0G True True ATA/QEMU HARDDISK /dev/sda hdd 50.0G True False ATA/QEMU HARDDISK Host 192.168.121.181: Device Path Type Size Rotates Available Model /dev/sdb hdd 50.0G True True ATA/QEMU HARDDISK /dev/sda hdd 50.0G True False ATA/QEMU HARDDISK .. note:: Output form Ansible orchestrator Create OSDs ^^^^^^^^^^^ Create OSDs on a group of devices on a single host:: ceph orchestrator osd create : ceph orchestrator osd create -i The output of ``osd create`` is not specified and may vary between orchestrator backends. Where ``drive.group.json`` is a JSON file containing the fields defined in :class:`orchestrator.DriveGroupSpec` Example:: # ceph orchestrator osd create 192.168.121.206:/dev/sdc {"status": "OK", "msg": "", "data": {"event": "playbook_on_stats", "uuid": "7082f3ba-f5b7-4b7c-9477-e74ca918afcb", "stdout": "\r\nPLAY RECAP *********************************************************************\r\n192.168.121.206 : ok=96 changed=3 unreachable=0 failed=0 \r\n", "counter": 932, "pid": 10294, "created": "2019-05-28T22:22:58.527821", "end_line": 1170, "runner_ident": "083cad3c-8197-11e9-b07a-2016b900e38f", "start_line": 1166, "event_data": {"ignored": 0, "skipped": {"192.168.121.206": 186}, "ok": {"192.168.121.206": 96}, "artifact_data": {}, "rescued": 0, "changed": {"192.168.121.206": 3}, "pid": 10294, "dark": {}, "playbook_uuid": "409364a6-9d49-4e44-8b7b-c28e5b3adf89", "playbook": "add-osd.yml", "failures": {}, "processed": {"192.168.121.206": 1}}, "parent_uuid": "409364a6-9d49-4e44-8b7b-c28e5b3adf89"}} .. note:: Output form Ansible orchestrator Decommission an OSD ^^^^^^^^^^^^^^^^^^^ :: ceph orchestrator osd rm  [osd-id...] Removes one or more OSDs from the cluster and the host, if the OSDs are marked as ``destroyed``. Example:: # ceph orchestrator osd rm 4 {"status": "OK", "msg": "", "data": {"event": "playbook_on_stats", "uuid": "1a16e631-906d-48e0-9e24-fa7eb593cc0a", "stdout": "\r\nPLAY RECAP *********************************************************************\r\n192.168.121.158 : ok=2 changed=0 unreachable=0 failed=0 \r\n192.168.121.181 : ok=2 changed=0 unreachable=0 failed=0 \r\n192.168.121.206 : ok=2 changed=0 unreachable=0 failed=0 \r\nlocalhost : ok=31 changed=8 unreachable=0 failed=0 \r\n", "counter": 240, "pid": 10948, "created": "2019-05-28T22:26:09.264012", "end_line": 308, "runner_ident": "8c093db0-8197-11e9-b07a-2016b900e38f", "start_line": 301, "event_data": {"ignored": 0, "skipped": {"localhost": 37}, "ok": {"192.168.121.181": 2, "192.168.121.158": 2, "192.168.121.206": 2, "localhost": 31}, "artifact_data": {}, "rescued": 0, "changed": {"localhost": 8}, "pid": 10948, "dark": {}, "playbook_uuid": "a12ec40e-bce9-4bc9-b09e-2d8f76a5be02", "playbook": "shrink-osd.yml", "failures": {}, "processed": {"192.168.121.181": 1, "192.168.121.158": 1, "192.168.121.206": 1, "localhost": 1}}, "parent_uuid": "a12ec40e-bce9-4bc9-b09e-2d8f76a5be02"}} .. note:: Output form Ansible orchestrator .. Blink Device Lights ^^^^^^^^^^^^^^^^^^^ :: ceph orchestrator device ident-on ceph orchestrator device ident-off ceph orchestrator device fault-on ceph orchestrator device fault-off ceph orchestrator osd ident-on {primary,journal,db,wal,all} ceph orchestrator osd ident-off {primary,journal,db,wal,all} ceph orchestrator osd fault-on {primary,journal,db,wal,all} ceph orchestrator osd fault-off {primary,journal,db,wal,all} Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of bluestore and ``all`` stands for all devices associated with the osd Monitor and manager management ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Creates or removes MONs or MGRs from the cluster. Orchestrator may return an error if it doesn't know how to do this transition. Update the number of monitor nodes:: ceph orchestrator mon update [host, host:network...] Each host can optionally specify a network for the monitor to listen on. Update the number of manager nodes:: ceph orchestrator mgr update [host...] .. .. note:: The host lists are the new full list of mon/mgr hosts .. note:: specifying hosts is optional for some orchestrator modules and mandatory for others (e.g. Ansible). Service Status ~~~~~~~~~~~~~~ Print a list of services known to the orchestrator. The list can be limited to services on a particular host with the optional --host parameter and/or services of a particular type via optional --type parameter (mon, osd, mgr, mds, rgw): :: ceph orchestrator service ls [--host host] [--svc_type type] [--refresh|--no-cache] Discover the status of a particular service:: ceph orchestrator service status  [--refresh] Query the status of a particular service instance (mon, osd, mds, rgw). For OSDs the id is the numeric OSD ID, for MDS services it is the filesystem name:: ceph orchestrator service-instance status  [--refresh] Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The orchestrator is not responsible for configuring the services. Please look into the corresponding documentation for details. The ``name`` parameter is an identifier of the group of instances: * a CephFS filesystem for a group of MDS daemons, * a zone name for a group of RGWs Sizing: the ``size`` parameter gives the number of daemons in the cluster (e.g. the number of MDS daemons for a particular CephFS filesystem). Creating/growing/shrinking/removing services:: ceph orchestrator {mds,rgw} update [host…] ceph orchestrator {mds,rgw} add ceph orchestrator nfs update [host…] ceph orchestrator nfs add [--namespace=] ceph orchestrator {mds,rgw,nfs} rm e.g., ``ceph orchestrator mds update myfs 3 host1 host2 host3`` Start/stop/reload:: ceph orchestrator service {stop,start,reload} ceph orchestrator service-instance {start,stop,reload} Current Implementation Status ============================= This is an overview of the current implementation status of the orchestrators. =================================== ========= ====== ========= ===== Command Ansible Rook DeepSea SSH =================================== ========= ====== ========= ===== host add ✔️ ⚪ ⚪ ✔️ host ls ✔️ ⚪ ⚪ ✔️ host rm ✔️ ⚪ ⚪ ✔️ mgr update ⚪ ⚪ ⚪ ✔️ mon update ⚪ ✔️ ⚪ ✔️ osd create ✔️ ✔️ ⚪ ✔️ osd device {ident,fault}-{on,off} ⚪ ⚪ ⚪ ⚪ osd rm ✔️ ⚪ ⚪ ⚪ device {ident,fault}-(on,off} ⚪ ⚪ ⚪ ⚪ device ls ✔️ ✔️ ✔️ ✔️ service ls ⚪ ✔️ ✔️ ⚪ service status ⚪ ✔️ ✔️ ⚪ service-instance status ⚪ ⚪ ⚪ ⚪ iscsi {stop,start,reload} ⚪ ⚪ ⚪ ⚪ iscsi add ⚪ ⚪ ⚪ ⚪ iscsi rm ⚪ ⚪ ⚪ ⚪ iscsi update ⚪ ⚪ ⚪ ⚪ mds {stop,start,reload} ⚪ ⚪ ⚪ ⚪ mds add ⚪ ✔️ ⚪ ⚪ mds rm ⚪ ✔️ ⚪ ⚪ mds update ⚪ ⚪ ⚪ ⚪ nfs {stop,start,reload} ⚪ ⚪ ⚪ ⚪ nfs add ⚪ ✔️ ⚪ ⚪ nfs rm ⚪ ✔️ ⚪ ⚪ nfs update ⚪ ⚪ ⚪ ⚪ rbd-mirror {stop,start,reload} ⚪ ⚪ ⚪ ⚪ rbd-mirror add ⚪ ⚪ ⚪ ⚪ rbd-mirror rm ⚪ ⚪ ⚪ ⚪ rbd-mirror update ⚪ ⚪ ⚪ ⚪ rgw {stop,start,reload} ⚪ ⚪ ⚪ ⚪ rgw add ⚪ ✔️ ⚪ ⚪ rgw rm ⚪ ✔️ ⚪ ⚪ rgw update ⚪ ⚪ ⚪ ⚪ =================================== ========= ====== ========= ===== where * ⚪ = not yet implemented * ❌ = not applicable * ✔ = implemented