diff --git a/doc/cluster-ops/control.rst b/doc/cluster-ops/control.rst new file mode 100644 index 00000000000..90ac4b09757 --- /dev/null +++ b/doc/cluster-ops/control.rst @@ -0,0 +1,390 @@ +.. index:: control, commands + +================== + Control Commands +================== + + +Monitor Commands +================ + +Monitor commands are issued using the ceph utility:: + + ceph [-m monhost] {command} + +The command is usually (though not always) of the form:: + + ceph {subsystem} {command} + + +System Commands +=============== + +Execute the following to display the current status of the cluster. :: + + ceph -s + ceph status + +Execute the following to display a running summary of the status of the cluster, +and major events. :: + + ceph -w + +Execute the following to show the monitor quorum, including which monitors are +participating and which one is the leader. :: + + ceph quorum_status + +Execute the following to query the status of a single monitor, including whether +or not it is in the quorum. :: + + ceph [-m monhost] mon_status + + +Authentication Subsystem +======================== + +To add a keyring for an OSD, execute the following:: + + ceph auth add {osd} {--in-file|-i} {path-to-osd-keyring} + +To list the cluster's keys and their capabilities, execute the following:: + + ceph auth list + + +Placement Group Subsystem +========================= + +To display the statistics for all placement groups, execute the following:: + + ceph -- pg dump [--format {format}] + +The valid formats are ``plain`` (default) and ``json``. + +To display the statistics for all placement groups stuck in a specified state, +execute the following:: + + ceph -- pg dump_stuck inactive|unclean|stale [--format {format}] [-t|--threshold {seconds}] + + +``--format`` may be ``plain`` (default) or ``json`` + +``--threshold`` defines how many seconds "stuck" is (default: 300) + +**Inactive** Placement groups cannot process reads or writes because they are waiting for an OSD +with the most up-to-date data to come back. + +**Unclean** Placement groups contain objects that are not replicated the desired number +of times. They should be recovering. + +**Stale** Placement groups are in an unknown state - the OSDs that host them have not +reported to the monitor cluster in a while (configured by +``mon_osd_report_timeout``). + +Revert "lost" objects to their prior state, either a previous version +or delete them if they were just created. :: + + ceph pg {pgid} mark_unfound_lost revert + + +OSD Subsystem +============= + +Query osd subsystem status. :: + + ceph osd stat + +Write a copy of the most recent osd map to a file. See +`osdmaptool`_. :: + + ceph osd getmap -o file + +.. _osdmaptool: ../man/8/osdmaptool + +Write a copy of the crush map from the most recent osd map to +file. :: + + ceph osd getcrushmap -o file + +The foregoing functionally equivalent to :: + + ceph osd getmap -o /tmp/osdmap + osdmaptool /tmp/osdmap --export-crush file + +Dump the OSD map. Valid formats for ``-f`` are ``plain`` and ``json``. If no +``--format`` option is given, the OSD map is dumped as plain text. :: + + ceph osd dump [--format {format}] + +Dump the OSD map as a tree with one line per OSD containing weight +and state. :: + + ceph osd tree [--format {format}] + +Add or move a new item (OSD) with the given id/name/weight at the specified +location. :: + + ceph osd crush set {id} {name} {weight} [{loc1} [{loc2} ...]] + +Remove an existing item from the CRUSH map. :: + + ceph osd crush remove {id} + +Move an existing bucket from one position in the hierarchy to another. :: + + ceph osd crush move {id} {loc1} [{loc2} ...] + +Set the weight of the item given by ``{name}`` to ``{weight}``. :: + + ceph osd crush reweight {name} {weight} + +Create a cluster snapshot. :: + + ceph osd cluster_snap {name} + +Mark an OSD as lost. This may result in permanent data loss. Use with caution. :: + + ceph osd lost [--yes-i-really-mean-it] + +Create a new OSD. If no ID is given, a new ID is automatically selected +if possible. :: + + ceph osd create [{id}] + +Remove the given OSD(s). :: + + ceph osd rm [{id}...] + +Query the current max_osd parameter in the osd map. :: + + ceph osd getmaxosd + +Import the given OSD map. Note that this can be a bit dangerous, +since the OSD map includes dynamic state about which OSDs are current +on or offline; only do this if you've just modified a (very) recent +copy of the map. :: + + ceph osd setmap -i file + +Import the given crush map. :: + + ceph osd setcrushmap -i file + +Set the ``max_osd`` parameter in the OSD map. This is necessary when +expanding the storage cluster. :: + + ceph osd setmaxosd + +Mark OSD ``{osd-num}`` down. :: + + ceph osd down {osd-num} + +Mark OSD ``{osd-num}`` out of the distribution (i.e. allocated no data). :: + + ceph osd out {osd-num} + +Mark ``{osd-num}`` in the distribution (i.e. allocated data). :: + + ceph osd in {osd-num} + +List classes that are loaded in the ceph cluster. :: + + ceph class list + +Set or clear the pause flags in the OSD map. If set, no IO requests +will be sent to any OSD. Clearing the flags via unpause results in +resending pending requests. :: + + ceph osd pause + ceph osd unpause + +Set the weight of ``{osd-num}`` to ``{weight}``. Two OSDs with the same weight will receive +roughly the same number of I/O requests and store approximately the +same amount of data. :: + + ceph osd reweight {osd-num} {weight} + +Reweights all the OSDs by reducing the weight of OSDs which are +heavily overused. By default it will adjust the weights downward on +OSDs which have 120% of the average utilization, but if you include +threshold it will use that percentage instead. :: + + ceph osd reweight-by-utilization [threshold] + +Adds/removes the address to/from the blacklist. When adding an address, +you can specify how long it should be blacklisted in seconds; otherwise, +it will default to 1 hour. A blacklisted address is prevented from +connecting to any OSD. Blacklisting is most often used to prevent a +lagging metadata server from making bad changes to data on the OSDs. + +These commands are mostly only useful for failure testing, as +blacklists are normally maintained automatically and shouldn't need +manual intervention. :: + + ceph osd blacklist add ADDRESS[:source_port] [TIME] + ceph osd blacklist rm ADDRESS[:source_port] + +Creates/deletes a snapshot of a pool. :: + + ceph osd pool mksnap {pool-name} {snap-name} + ceph osd pool rmsnap {pool-name} {snap-name} + +Creates/deletes/renames a storage pool. :: + + ceph osd pool create {pool-name} [pg_num [pgp_num]] + ceph osd pool delete {pool-name} + ceph osd pool rename {old-name} {new-name} + +Changes a pool setting. :: + + ceph osd pool set {pool-name} {field} {value} + +Valid fields are: + + * ``size``: Sets the number of copies of data in the pool. + * ``crash_replay_interval``: The number of seconds to allow + clients to replay acknowledged but uncommited requests. + * ``pg_num``: The placement group number. + * ``pgp_num``: Effective number when calculating pg placement. + * ``crush_ruleset``: rule number for mapping placement. + +Get the value of a pool setting. :: + + ceph osd pool get {pool-name} {field} + +Valid fields are: + + * ``pg_num``: The placement group number. + * ``pgp_num``: Effective number of placement groups when calculating placement. + * ``lpg_num``: The number of local placement groups. + * ``lpgp_num``: The number used for placing the local placement groups. + + +Sends a scrub command to OSD ``{osd-num}``. To send the command to all OSDs, use ``*``. :: + + ceph osd scrub {osd-num} + +Sends a repair command to osdN. To send the command to all osds, use ``*``. :: + + ceph osd repair N + +Runs a simple throughput benchmark against osdN, writing ``TOTAL_BYTES`` +in write requests of ``BYTES_PER_WRITE`` each. By default, the test +writes 1 GB in total in 4-MB increments. :: + + ceph osd tell N bench [BYTES_PER_WRITE] [TOTAL_BYTES] + + +MDS Subsystem +------------- + +Change configuration parameters on a running mds. :: + + ceph mds tell {mds-id} injectargs '--{switch} {value} [--{switch} {value}]' + +Example:: + + ceph mds tell 0 injectargs '--debug_ms 1 --debug_mds 10' + +Enables debug messages. :: + + ceph mds stat + +Displays the status of all metadata servers. + +.. todo:: ``ceph mds`` subcommands missing docs: set_max_mds, dump, getmap, stop, setmap + + +Mon Subsystem +------------- + +Show monitor stats:: + + ceph mon stat + + 2011-12-14 10:40:59.044395 mon {- [mon,stat] + 2011-12-14 10:40:59.057111 mon.1 -} 'e3: 5 mons at {a=10.1.2.3:6789/0,b=10.1.2.4:6789/0,c=10.1.2.5:6789/0,d=10.1.2.6:6789/0,e=10.1.2.7:6789/0}, election epoch 16, quorum 0,1,2,3' (0) + +The ``quorum`` list at the end lists monitor nodes that are part of the current quorum. + +This is also available more directly:: + + $ ./ceph quorum_status + + 2011-12-14 10:44:20.417705 mon {- [quorum_status] + 2011-12-14 10:44:20.431890 mon.0 -} + +.. code-block:: javascript + + '{ "election_epoch": 10, + "quorum": [ + 0, + 1, + 2], + "monmap": { "epoch": 1, + "fsid": "444b489c-4f16-4b75-83f0-cb8097468898", + "modified": "2011-12-12 13:28:27.505520", + "created": "2011-12-12 13:28:27.505520", + "mons": [ + { "rank": 0, + "name": "a", + "addr": "127.0.0.1:6789\/0"}, + { "rank": 1, + "name": "b", + "addr": "127.0.0.1:6790\/0"}, + { "rank": 2, + "name": "c", + "addr": "127.0.0.1:6791\/0"}]}}' (0) + +The above will block until a quorum is reached. + +For a status of just the monitor you connect to (use ``-m HOST:PORT`` +to select):: + + ceph mon_status + + + 2011-12-14 10:45:30.644414 mon {- [mon_status] + 2011-12-14 10:45:30.644632 mon.0 -} + +.. code-block:: javascript + + '{ "name": "a", + "rank": 0, + "state": "leader", + "election_epoch": 10, + "quorum": [ + 0, + 1, + 2], + "outside_quorum": [], + "monmap": { "epoch": 1, + "fsid": "444b489c-4f16-4b75-83f0-cb8097468898", + "modified": "2011-12-12 13:28:27.505520", + "created": "2011-12-12 13:28:27.505520", + "mons": [ + { "rank": 0, + "name": "a", + "addr": "127.0.0.1:6789\/0"}, + { "rank": 1, + "name": "b", + "addr": "127.0.0.1:6790\/0"}, + { "rank": 2, + "name": "c", + "addr": "127.0.0.1:6791\/0"}]}}' (0) + +A dump of the monitor state:: + + ceph mon dump + + 2011-12-14 10:43:08.015333 mon {- [mon,dump] + 2011-12-14 10:43:08.015567 mon.0 -} 'dumped monmap epoch 1' (0) + epoch 1 + fsid 444b489c-4f16-4b75-83f0-cb8097468898 + last_changed 2011-12-12 13:28:27.505520 + created 2011-12-12 13:28:27.505520 + 0: 127.0.0.1:6789/0 mon.a + 1: 127.0.0.1:6790/0 mon.b + 2: 127.0.0.1:6791/0 mon.c +