diff --git a/doc/ops/index.rst b/doc/ops/index.rst deleted file mode 100644 index aa8d6a92a21..00000000000 --- a/doc/ops/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -============ - Operations -============ - -.. =========================== -.. This Document is deprecated -.. See ceph/doc/cluster-ops -.. =========================== - -.. toctree:: - - manage/index - radosgw diff --git a/doc/ops/manage/crush.rst b/doc/ops/manage/crush.rst deleted file mode 100644 index 5e277aea240..00000000000 --- a/doc/ops/manage/crush.rst +++ /dev/null @@ -1,199 +0,0 @@ -.. _adjusting-crush: - -========================= - Adjusting the CRUSH map -========================= - -There are a few ways to adjust the crush map: - -* online, by issuing commands to the monitor -* offline, by extracting the current map to a file, modifying it, and then reinjecting a new map - -For offline changes, some can be made directly with ``crushtool``, and -others require you to decompile the file to text form, manually edit -it, and then recompile. - - -Adding a new device (OSD) to the map -==================================== - -.. _adjusting-crush-set: - -Adding new devices or moving existing devices to new positions in the -CRUSH hierarchy can be done via the monitor. The general form is:: - - $ ceph osd crush set [ [ ...]] - -where - - * ``id`` is the numeric device id (the OSD id) - * ``name`` is an alphanumeric name. By convention Ceph uses - ``osd.$id``. - * ``weight`` is a floating point weight value controlling how much - data the device will be allocated. A decent convention is to make - this the number of TB the device will store. - * ``loc`` is a list of ``what=where`` pairs indicating where in the - CRUSH hierarchy the device will be stored. By default, the - hierarchy (the ``what``s) includes ``pool`` (the ``default`` pool - is normally the root of the hierarchy), ``rack``, and ``host``. - At least one of these location specifiers has to refer to an - existing point in the hierarchy, and only the lowest (most - specific) match counts. Beneath that point, any intervening - branches will be created as needed. Specifying the complete - location is always sufficient, and also safe in that existing - branches (and devices) won't be moved around. - -For example, if the OSD id is ``123``, we want a weight of ``1.0`` and -the device is on host ``hostfoo`` and rack ``rackbar``:: - - $ ceph osd crush set 123 1.0 root=default rack=rackbar host=hostfoo - -will add it to the hierarchy, or move it from its previous position. -(Note: for clusters created prior to v0.52, replace ``root=default`` -with ``pool=default``.) The rack ``rackbar`` and host ``hostfoo`` -will be added as needed, as long as the pool ``default`` exists (as it -does in the default Ceph CRUSH map generated during cluster creation). - -Note that if I later add another device in the same host but specify a -different pool or rack:: - - $ ceph osd crush set 124 1.0 root=nondefault rack=weirdrack host=hostfoo - -the device will still be placed in host ``hostfoo`` at its current -location (rack ``rackbar`` and pool ``default``). - - -Moving a bucket to a different position in the hierarchy -======================================================== - -To move an existing bucket to a different position in the hierarchy, -identify the bucket to move by name and specify the new location in -the same fashion as with ``osd crush set ...``:: - - $ ceph osd crush move [ [ ...]] - -where - - * ``name`` is the name of the bucket to move. - * ``loc`` is a list of ``what=where`` pairs indicating where the bucket should - be moved. - - -Adjusting the CRUSH weight -========================== - -You can adjust the CRUSH weight for a device with:: - - $ ceph osd crush reweight osd.123 2.0 - -Removing a device -================= - -You can remove a device from the crush map with:: - - $ ceph osd crush remove osd.123 - -Tunables -======== - -There are several magic numbers that were used in the original CRUSH -implementation that have proven to be poor choices. To support -the transition away from them, newer versions of CRUSH (starting with -the v0.48 argonaut series) allow the values to be adjusted or tuned. - -Clusters running recent Ceph releases support using the tunable values -in the CRUSH maps. However, older clients and daemons will not correctly interact -with clusters using the "tuned" CRUSH maps. To detect this situation, -there is now a feature bit ``CRUSH_TUNABLES`` (value 0x40000) to -reflect support for tunables. - -If the OSDMap currently used by the ``ceph-mon`` or ``ceph-osd`` -daemon has non-legacy values, it will require the ``CRUSH_TUNABLES`` -feature bit from clients and daemons who connect to it. This means -that old clients will not be able to connect. - -At some future point in time, newly created clusters will have -improved default values for the tunables. This is a matter of waiting -until the support has been present in the Linux kernel clients long -enough to make this a painless transition for most users. - -Impact of legacy values -~~~~~~~~~~~~~~~~~~~~~~~ - -The legacy values result in several misbehaviors: - - * For hiearchies with a small number of devices in the leaf buckets, - some PGs map to fewer than the desired number of replicas. This - commonly happens for hiearchies with "host" nodes with a small - number (1-3) of OSDs nested beneath each one. - - * For large clusters, some small percentages of PGs map to less than - the desired number of OSDs. This is more prevalent when there are - several layers of the hierarchy (e.g., row, rack, host, osd). - - * When some OSDs are marked out, the data tends to get redistributed - to nearby OSDs instead of across the entire hierarchy. - -Which client versions support tunables -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - * argonaut series, v0.48.1 or later - * v0.49 or later - * Linux kernel version v3.5 or later (for the file system and RBD kernel clients) - -A few important points -~~~~~~~~~~~~~~~~~~~~~~ - - * Adjusting these values will result in the shift of some PGs between - storage nodes. If the Ceph cluster is already storing a lot of - data, be prepared for some fraction of the data to move. - * The ``ceph-osd`` and ``ceph-mon`` daemons will start requiring the - ``CRUSH_TUNABLES`` feature of new connections as soon as they get - the updated map. However, already-connected clients are - effectively grandfathered in, and will misbehave if they do not - support the new feature. - * If the CRUSH tunables are set to non-legacy values and then later - changed back to the defult values, ``ceph-osd`` daemons will not be - required to support the feature. However, the OSD peering process - requires examining and understanding old maps. Therefore, you - should not run old (pre-v0.48) versions of the ``ceph-osd`` daemon - if the cluster has previosly used non-legacy CRUSH values, even if - the latest version of the map has been switched back to using the - legacy defaults. - -Tuning CRUSH -~~~~~~~~~~~~ - -If you can ensure that all clients are running recent code, you can -adjust the tunables by extracting the CRUSH map, modifying the values, -and reinjecting it into the cluster. - -* Extract the latest CRUSH map:: - - ceph osd getcrushmap -o /tmp/crush - -* Adjust tunables. These values appear to offer the best behavior - for both large and small clusters we tested with. You will need to - additionally specify the ``--enable-unsafe-tunables`` argument to - ``crushtool`` for this to work. Please use this option with - extreme care.:: - - crushtool -i /tmp/crush --set-choose-local-tries 0 --set-choose-local-fallback-tries 0 --set-choose-total-tries 50 -o /tmp/crush.new - -* Reinject modified map:: - - ceph osd setcrushmap -i /tmp/crush.new - -Legacy values -~~~~~~~~~~~~~ - -For reference, the legacy values for the CRUSH tunables can be set -with:: - - crushtool -i /tmp/crush --set-choose-local-tries 2 --set-choose-local-fallback-tries 5 --set-choose-total-tries 19 -o /tmp/crush.legacy - -Again, the special ``--enable-unsafe-tunables`` option is required. -Further, as noted above, be careful running old versions of the -``ceph-osd`` daemon after reverting to legacy values as the feature -bit is not perfectly enforced. - diff --git a/doc/ops/manage/failures/index.rst b/doc/ops/manage/failures/index.rst deleted file mode 100644 index 51813af0439..00000000000 --- a/doc/ops/manage/failures/index.rst +++ /dev/null @@ -1,36 +0,0 @@ -.. _failures-osd: - -========================== - Recovering from failures -========================== - -The current health of the Ceph cluster, as known by the monitors, can -be checked with the ``ceph health`` command. If all is well, you get:: - - $ ceph health - HEALTH_OK - -If there are problems, you will see something like:: - - $ ceph health - HEALTH_WARN short summary of problem(s) - -or:: - - $ ceph health - HEALTH_ERROR short summary of very serious problem(s) - -To get more detail:: - - $ ceph health detail - HEALTH_WARN short description of problem - - one problem - another problem - yet another problem - ... - -.. toctree:: - - radosgw - diff --git a/doc/ops/manage/failures/radosgw.rst b/doc/ops/manage/failures/radosgw.rst deleted file mode 100644 index 0dffc36fb60..00000000000 --- a/doc/ops/manage/failures/radosgw.rst +++ /dev/null @@ -1,104 +0,0 @@ -==================================== - Recovering from ``radosgw`` failure -==================================== - - -HTTP Request Errors -=================== - -Examining the access and error logs for the web server itself is -probably the first step in identifying what is going on. If there is -a 500 error, that usually indicates a problem communicating with the -``radosgw`` daemon. Ensure the daemon is running, its socket path is -configured, and that the web server is looking for it in the proper -location. - - -Crashed ``radosgw`` process -=========================== - -If the ``radosgw`` process dies, you will normally see a 500 error -from the web server (apache, nginx, etc.). In that situation, simply -restarting radosgw will restore service. - -To diagnose the cause of the crash, check the log in ``/var/log/ceph`` -and/or the core file (if one was generated). - - -Blocked ``radosgw`` Requests -============================ - -If some (or all) radosgw requests appear to be blocked, you can get -some insight into the internal state of the ``radosgw`` daemon via -its admin socket. By default, there will be a socket configured to -reside in ``/var/run/ceph``, and the daemon can be queried with:: - - $ ceph --admin-daemon /var/run/ceph/client.rgw help - help list available commands - objecter_requests show in-progress osd requests - perfcounters_dump dump perfcounters value - perfcounters_schema dump perfcounters schema - version get protocol version - -Of particular interest:: - - $ ceph --admin-daemon /var/run/ceph/client.rgw objecter_requests - ... - -will dump information about current in-progress requests with the -RADOS cluster. This allows one to identify if any requests are blocked -by a non-responsive ceph-osd. For example, one might see:: - - { "ops": [ - { "tid": 1858, - "pg": "2.d2041a48", - "osd": 1, - "last_sent": "2012-03-08 14:56:37.949872", - "attempts": 1, - "object_id": "fatty_25647_object1857", - "object_locator": "@2", - "snapid": "head", - "snap_context": "0=[]", - "mtime": "2012-03-08 14:56:37.949813", - "osd_ops": [ - "write 0~4096"]}, - { "tid": 1873, - "pg": "2.695e9f8e", - "osd": 1, - "last_sent": "2012-03-08 14:56:37.970615", - "attempts": 1, - "object_id": "fatty_25647_object1872", - "object_locator": "@2", - "snapid": "head", - "snap_context": "0=[]", - "mtime": "2012-03-08 14:56:37.970555", - "osd_ops": [ - "write 0~4096"]}], - "linger_ops": [], - "pool_ops": [], - "pool_stat_ops": [], - "statfs_ops": []} - -In this dump, two requests are in progress. The ``last_sent`` field is -the time the RADOS request was sent. If this is a while ago, it suggests -that the OSD is not responding. For example, for request 1858, you could -check the OSD status with:: - - $ ceph pg map 2.d2041a48 - osdmap e9 pg 2.d2041a48 (2.0) -> up [1,0] acting [1,0] - -This tells us to look at ``osd.1``, the primary copy for this PG:: - - $ ceph --admin-daemon /var/run/ceph/osd.1.asok - { "num_ops": 651, - "ops": [ - { "description": "osd_op(client.4124.0:1858 fatty_25647_object1857 [write 0~4096] 2.d2041a48)", - "received_at": "1331247573.344650", - "age": "25.606449", - "flag_point": "waiting for sub ops", - "client_info": { "client": "client.4124", - "tid": 1858}}, - ... - -The ``flag_point`` field indicates that the OSD is currently waiting -for replicas to respond, in this case ``osd.0``. diff --git a/doc/ops/manage/index.rst b/doc/ops/manage/index.rst deleted file mode 100644 index 9ab218f9aea..00000000000 --- a/doc/ops/manage/index.rst +++ /dev/null @@ -1,8 +0,0 @@ -========================= - Managing a Ceph cluster -========================= - -.. toctree:: - - failures/index - crush diff --git a/doc/ops/radosgw.rst b/doc/ops/radosgw.rst deleted file mode 100644 index e61a88da187..00000000000 --- a/doc/ops/radosgw.rst +++ /dev/null @@ -1,228 +0,0 @@ -.. index:: RADOS Gateway, radosgw - -========================================= - Radosgw installation and administration -========================================= - -RADOS Gateway (radosgw or rgw) provides a RESTful API to the object -store. It interfaces with a web server via FastCGI, and with RADOS via -libradospp. - -Configuring Ceph for RADOS Gateway ----------------------------------- - -In order for a host to act as a RADOS gateway, you must add a -``[client.radosgw.]`` section to your Ceph configuration file -(typically ``/etc/ceph/ceph.conf``): - -:: - - [client.radosgw.gateway] - host = gateway - rgw socket path = /tmp/radosgw.sock - -``host`` is the name of the host running radosgw. ``keyring`` points -to the keyring file for Cephx authentication. ``rgw socket path`` is -the location of the UNIX socket which radosgw binds to. - -If your Ceph cluster has Cephx authentication enabled (highly -recommended) you also need to add the following option to tell radosgw -where it finds its authentication key: - -:: - - [client.radosgw.gateway] - keyring = /etc/ceph/keyring.radosgw.gateway - - -Creating authentication credentials ------------------------------------ - -To allow radosgw to sucessfully authenticate with the Ceph cluster, -use the ``ceph-authtool`` command to create a key and set its -capabilities: - -:: - - ceph-authtool -C -n client.radosgw.gateway \ - --gen-key /etc/ceph/keyring.radosgw.gateway - ceph-authtool -n client.radosgw.gateway \ - --cap mon 'allow r' --cap osd 'allow rwx' --cap mds 'allow' \ - /etc/ceph/keyring.radosgw.gateway - -Finally, add this key to the authentication entries: - -:: - - ceph auth add client.radosgw.gateway \ - --in-file=/etc/ceph/keyring.radosgw.gateway - - -Configuring the web server for radosgw --------------------------------------- - - -The radosgw FastCGI wrapper ---------------------------- - -A wrapper script, customarily named ``radosgw.cgi`` needs to go into -your preferred location -- typically your web server root directory. - -:: - - #!/bin/sh - exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway - - -The ``-c`` option may be omitted if your Ceph configuration file -resides in its default location ((``/etc/ceph/ceph.conf``)). The -``-n`` option identifies the ``client`` section in the configuration -file that radosgw should parse -- if omitted, this would default to -``client.admin``. - -Configuring Apache for radosgw ------------------------------- - -The recommended way of deploying radosgw is with Apache and -``mod_fastcgi``. Ensure that both ``mod_fastcgi`` and ``mod_rewrite`` -are enabled in your Apache configuration. Set the -``FastCGIExternalServer`` option to point to the radosgw FastCGI -wrapper. - -:: - - - FastCgiExternalServer /var/www/radosgw.fcgi -socket /tmp/radosgw.sock - - - -Then, create a virtual host configuration as follows: - -:: - - - ServerName radosgw.example.com - ServerAlias rgw.example.com - ServerAdmin webmaster@example.com - DocumentRoot /var/www - - - RewriteEngine On - RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] - - - - - Options +ExecCGI - AllowOverride All - SetHandler fastcgi-script - Order allow,deny - Allow from all - AuthBasicAuthoritative Off - - - - AllowEncodedSlashes On - ServerSignature Off - - - -Starting the daemons --------------------- - -For the gateway to become operational, start both the radosgw daemon -and your web server: - -:: - - service radosgw start - service apache start - - -Creating users --------------- - -In order to be able to use the RESTful API, create a user with the -``radosgw-admin`` utility: - -:: - - $ radosgw-admin user create --uid=johndoe --display-name="John Doe" --email=john@example.com - { "user_id": "johndoe", - "rados_uid": 0, - "display_name": "John Doe", - "email": "john@example.com", - "suspended": 0, - "subusers": [], - "keys": [ - { "user": "johndoe", - "access_key": "QFAMEDSJP5DEKJO0DDXY", - "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}], - "swift_keys": []} - -Note that creating a user also creates an ``access_key`` and -``secret_key`` entry for use with any S3 API-compatible client. - - -Enabling Swift access ---------------------- - -Allowing access to the object store with Swift (OpenStack Object -Storage) compatible clients requires an additional step, the creation -of a subuser and a Swift access key. - -:: - - # radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift --access=full - { "user_id": "johndoe", - "rados_uid": 0, - "display_name": "John Doe", - "email": "john@example.com", - "suspended": 0, - "subusers": [ - { "id": "johndoe:swift", - "permissions": "full-control"}], - "keys": [ - { "user": "johndoe", - "access_key": "QFAMEDSJP5DEKJO0DDXY", - "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}], - "swift_keys": []} - - # radosgw-admin key create --subuser=johndoe:swift --key-type=swift - { "user_id": "johndoe", - "rados_uid": 0, - "display_name": "John Doe", - "email": "john@example.com", - "suspended": 0, - "subusers": [ - { "id": "johndoe:swift", - "permissions": "full-control"}], - "keys": [ - { "user": "johndoe", - "access_key": "QFAMEDSJP5DEKJO0DDXY", - "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}], - "swift_keys": [ - { "user": "johndoe:swift", - "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]} - -With this configuration, you are able to use any Swift client to -connect to and use radosgw. As an example, you might use the ``swift`` -command-line client utility that ships with the OpenStack Object -Storage packages. - -:: - - $ swift -V 1.0 -A http://radosgw.example.com/auth \ - -U johndoe:swift -K E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 \ - post test - $ swift -V 1.0 -A http://radosgw.example.com/auth \ - -U johndoe:swift -K E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 \ - upload test myfile - -Note that the radosgw ``user:subuser`` tuple maps to the -``tenant:user`` tuple expected by Swift. - -Note also that the radosgw Swift authentication service only supports -built-in Swift authentication (``-V 1.0``) at this point. There is -currently no way to make radosgw authenticate users via OpenStack -Identity Service (Keystone).