mirror of
https://github.com/ceph/ceph
synced 2024-12-30 07:23:11 +00:00
a31b9e9c75
Signed-off-by: John Wilkins <john.wilkins@inktank.com>
433 lines
15 KiB
ReStructuredText
433 lines
15 KiB
ReStructuredText
=================================
|
|
Configuring Ceph Object Gateway
|
|
=================================
|
|
|
|
Configuring a Ceph Object Gateway requires a running Ceph Storage Cluster,
|
|
and an Apache web server with the FastCGI module.
|
|
|
|
The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a
|
|
Ceph Storage Cluster client, it requires:
|
|
|
|
- A name for the gateway instance. We use ``gateway`` in this guide.
|
|
- A storage cluster user name with appropriate permissions in a keyring.
|
|
- Pools to store its data.
|
|
- A data directory for the gateway instance.
|
|
- An instance entry in the Ceph Configuration file.
|
|
- A configuration file for the web server to interact with FastCGI.
|
|
|
|
|
|
Create a User and Keyring
|
|
=========================
|
|
|
|
Each instance must have a user name and key to communicate with a Ceph Storage
|
|
Cluster. In the following steps, we use an admin node to create a keyring.
|
|
Then, we create a client user name and key. Next, we add the
|
|
key to the Ceph Storage Cluster. Finally, we distribute the key ring to
|
|
the node containing the gateway instance.
|
|
|
|
.. topic:: Monitor Key CAPS
|
|
|
|
When you provide CAPS to the key, you MUST provide read capability.
|
|
However, you have the option of providing write capability for the monitor.
|
|
This is an important choice. If you provide write capability to the key,
|
|
the Ceph Object Gateway will have the ability to create pools automatically;
|
|
however, it will create pools with either the default number of placement
|
|
groups (not ideal) or the number of placement groups you specified in your
|
|
Ceph configuration file. If you allow the Ceph Object Gateway to create
|
|
pools automatically, ensure that you have reasonable defaults for the number
|
|
of placement groups first. See `Pool Configuration`_ for details.
|
|
|
|
|
|
See the `Cephx Guide`_ for additional details on Ceph authentication.
|
|
|
|
#. Create a keyring for the gateway. ::
|
|
|
|
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
|
|
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
#. Generate a Ceph Object Gateway user name and key for each instance. For
|
|
exemplary purposes, we will use the name ``gateway`` after ``client.radosgw``::
|
|
|
|
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key
|
|
|
|
|
|
#. Add capabilities to the key. See `Configuration Reference - Pools`_ for details
|
|
on the effect of write permissions for the monitor and creating pools. ::
|
|
|
|
sudo ceph-authtool -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
#. Once you have created a keyring and key to enable the Ceph Object Gateway
|
|
with access to the Ceph Storage Cluster, add the key to your
|
|
Ceph Storage Cluster. For example::
|
|
|
|
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
#. Distribute the keyring to the node with the gateway instance. ::
|
|
|
|
sudo scp /etc/ceph/ceph.client.radosgw.keyring ceph@{hostname}:/home/ceph
|
|
ssh {hostname}
|
|
sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
|
|
Create Pools
|
|
============
|
|
|
|
Ceph Object Gateways require Ceph Storage Cluster pools to store specific
|
|
gateway data. If the user you created has permissions, the gateway
|
|
will create the pools automatically. However, you should ensure that you have
|
|
set an appropriate default number of placement groups per pool into your Ceph
|
|
configuration file.
|
|
|
|
When configuring a gateway with the default region and zone, the naming
|
|
convention for pools typically omits region and zone naming, but you can use any
|
|
naming convention you prefer. For example:
|
|
|
|
|
|
- ``.rgw``
|
|
- ``.rgw.root``
|
|
- ``.rgw.control``
|
|
- ``.rgw.gc``
|
|
- ``.rgw.buckets``
|
|
- ``.rgw.buckets.index``
|
|
- ``.log``
|
|
- ``.intent-log``
|
|
- ``.usage``
|
|
- ``.users``
|
|
- ``.users.email``
|
|
- ``.users.swift``
|
|
- ``.users.uid``
|
|
|
|
|
|
See `Configuration Reference - Pools`_ for details on the default pools for
|
|
gateways. See `Pools`_ for details on creating pools. Execute the following
|
|
to create a pool::
|
|
|
|
ceph osd pool create {poolname} {pg-num} {pgp-num}
|
|
|
|
|
|
.. tip:: When adding a large number of pools, it may take some time for your
|
|
cluster to return to a ``active + clean`` state.
|
|
|
|
When you have completed this step, execute the following to ensure that
|
|
you have created all of the foregoing pools::
|
|
|
|
rados lspools
|
|
|
|
|
|
Create Data Directory
|
|
=====================
|
|
|
|
Deployment scripts may not create the default Ceph Object Gateway data
|
|
directory. Create data directories for each instance of a ``radosgw`` daemon
|
|
(if you haven't done so already). The ``host`` variables in the Ceph
|
|
configuration file determine which host runs each instance of a ``radosgw``
|
|
daemon. The typical form specifies the ``radosgw`` daemon, the cluster name and
|
|
the daemon ID. ::
|
|
|
|
sudo mkdir -p /var/lib/ceph/radosgw/{$cluster}-{$id}
|
|
|
|
Using the exemplary ``ceph.conf`` settings above, you would execute the following::
|
|
|
|
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway
|
|
|
|
|
|
|
|
Add a Gateway Configuration to Ceph
|
|
===================================
|
|
|
|
Add the Ceph Object Gateway configuration to your Ceph Configuration file. The
|
|
Ceph Object Gateway configuration requires you to identify the Ceph Object
|
|
Gateway instance. Then, you must specify the host name where you installed the
|
|
Ceph Object Gateway daemon, a keyring (for use with cephx), the socket path for
|
|
FastCGI and a log file. For example::
|
|
|
|
[client.radosgw.{instance-name}]
|
|
host = {host-name}
|
|
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
|
rgw socket path = /var/run/ceph/ceph.radosgw.{instance-name}.fastcgi.sock
|
|
log file = /var/log/ceph/client.radosgw.{instance-name}.log
|
|
|
|
The ``[client.radosgw.*]`` portion of the gateway instance identifies this
|
|
portion of the Ceph configuration file as configuring a Ceph Storage Cluster
|
|
client where the client type is a Ceph Object Gateway (i.e., ``radosgw``). The
|
|
instance name follows. For example::
|
|
|
|
[client.radosgw.gateway]
|
|
host = ceph-gateway
|
|
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
|
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
|
|
log file = /var/log/ceph/client.radosgw.gateway.log
|
|
|
|
.. note:: The ``host`` must be your machine hostname, not the FQDN. Make sure
|
|
that the name you use for the FastCGI socket is not the same as the one
|
|
used for the object gateway, which is
|
|
``ceph-client.radosgw.{instance-name}.asok`` by default. You must use the
|
|
same name in your S3 FastCGI file too. See `Add a Ceph Object Gateway
|
|
Script`_ for details.
|
|
|
|
|
|
Configuring Operations Logging
|
|
------------------------------
|
|
|
|
In early releases of Ceph (v0.66 and earlier), the Ceph Object Gateway will log
|
|
every successful operation in the Ceph Object Gateway backend by default. This
|
|
means that every request, whether it is a read request or a write request will
|
|
generate a gateway operation that writes data. This does not come without cost,
|
|
and may affect overall performance. Turning off logging completely can be done
|
|
by adding the following config option to the Ceph configuration file::
|
|
|
|
rgw enable ops log = false
|
|
|
|
Another way to reduce the logging load is to send operations logging data to a
|
|
UNIX domain socket, instead of writing it to the Ceph Object Gateway backend::
|
|
|
|
rgw ops log rados = false
|
|
rgw enable ops log = true
|
|
rgw ops log socket path = <path to socket>
|
|
|
|
When specifying a UNIX domain socket, it is also possible to specify the maximum
|
|
amount of memory that will be used to keep the data backlog::
|
|
|
|
rgw ops log data backlog = <size in bytes>
|
|
|
|
Any backlogged data in excess to the specified size will be lost, so the socket
|
|
needs to be read constantly.
|
|
|
|
|
|
Enabling Subdomain S3 Calls
|
|
---------------------------
|
|
|
|
To use a Ceph Object Gateway with subdomain S3 calls (e.g.,
|
|
``http://bucketname.hostname``), you must add the Ceph Object Gateway DNS name
|
|
under the ``[client.radosgw.gateway]`` section of your Ceph configuration file::
|
|
|
|
[client.radosgw.gateway]
|
|
...
|
|
rgw dns name = {hostname}
|
|
|
|
You should also consider installing a DNS server such as `Dnsmasq`_ on your
|
|
client machine(s) when using ``http://{bucketname}.{hostname}`` syntax. The
|
|
``dnsmasq.conf`` file should include the following settings::
|
|
|
|
address=/{hostname}/{host-ip-address}
|
|
listen-address={client-loopback-ip}
|
|
|
|
Then, add the ``{client-loopback-ip}`` IP address as the first DNS nameserver
|
|
on client the machine(s).
|
|
|
|
See `Add Wildcard to DNS`_ for details.
|
|
|
|
|
|
Redeploy Ceph Configuration
|
|
---------------------------
|
|
|
|
To use ``ceph-deploy`` to push a new copy of the configuration file to the hosts
|
|
in your cluster, execute the following::
|
|
|
|
ceph-deploy config push {host-name [host-name]...}
|
|
|
|
|
|
Add a Ceph Object Gateway Script
|
|
================================
|
|
|
|
Add a ``s3gw.fcgi`` file (use the same name referenced in the first line
|
|
of ``rgw.conf``). For Debian/Ubuntu distributions, save the file to the
|
|
``/var/www`` directory. For CentOS/RHEL distributions, save the file to the
|
|
``/var/www/html`` directory. Assuming a cluster named ``ceph`` (default),
|
|
and the user created in previous steps, the contents of the file should
|
|
include::
|
|
|
|
#!/bin/sh
|
|
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
|
|
|
|
Ensure that you apply execute permissions to ``s3gw.fcgi``. ::
|
|
|
|
sudo chmod +x s3gw.fcgi
|
|
|
|
On some distributions, you must also change the ownership to ``apache``. ::
|
|
|
|
sudo chown apache:apache s3gw.fcgi
|
|
|
|
|
|
Create a Gateway Configuration
|
|
==============================
|
|
|
|
On the host where you installed the Ceph Object Gateway, create an ``rgw.conf``
|
|
file. For Debian/Ubuntu systems, place the file in the
|
|
``/etc/apache2/sites-available`` directory. For CentOS/RHEL systems, place the
|
|
file in the ``/etc/httpd/conf.d`` directory.
|
|
|
|
We recommend deploying FastCGI as an external server, because allowing Apache to
|
|
manage FastCGI sometimes introduces high latency. To manage FastCGI as an
|
|
external server, use the ``FastCgiExternalServer`` directive. See
|
|
`FastCgiExternalServer`_ for details on this directive. See `Module
|
|
mod_fastcgi`_ for general details. See `Apache Virtual Host documentation`_ for
|
|
details on ``<VirtualHost>`` format and settings. See `<IfModule> Directive`_
|
|
for additional details.
|
|
|
|
Ceph Object Gateway requires a rewrite rule for the Amazon S3-compatible
|
|
interface. It's required for passing in the ``HTTP_AUTHORIZATION env`` for S3,
|
|
which is filtered out by Apache. The rewrite rule is not necessary for the
|
|
OpenStack Swift-compatible interface.
|
|
|
|
You should configure Apache to allow encoded slashes, provide paths for log
|
|
files and to turn off server signatures. See below for an exemplary embodiment
|
|
of a gateway configuration for Debian/Ubuntu and CentOS/RHEL.
|
|
|
|
.. rubric:: Debian/Ubuntu
|
|
|
|
.. literalinclude:: rgw-debian.conf
|
|
:language: ini
|
|
|
|
.. rubric:: CentOS/RHEL
|
|
|
|
.. literalinclude:: rgw-centos.conf
|
|
:language: ini
|
|
|
|
|
|
#. Replace the ``/{path}/{socket-name}`` entry with path to the socket and
|
|
the socket name. For example,
|
|
``/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock``. Ensure that you use the
|
|
same path and socket name in your ``ceph.conf`` entry.
|
|
|
|
#. Replace the ``{fqdn}`` entry with the fully-qualified domain name of the
|
|
server.
|
|
|
|
#. Replace the ``{email.address}`` entry with the email address for the
|
|
server administrator.
|
|
|
|
#. Add a ``ServerAlias`` if you wish to use S3-style subdomains
|
|
(of course you do).
|
|
|
|
#. Save the configuration to a file (e.g., ``rgw.conf``).
|
|
|
|
Finally, if you enabled SSL, make sure that you set the port to your SSL port
|
|
(usually 443) and your configuration file includes the following::
|
|
|
|
SSLEngine on
|
|
SSLCertificateFile /etc/apache2/ssl/apache.crt
|
|
SSLCertificateKeyFile /etc/apache2/ssl/apache.key
|
|
SetEnv SERVER_PORT_SECURE 443
|
|
|
|
|
|
.. _Module mod_fastcgi: http://www.fastcgi.com/drupal/node/25
|
|
.. _FastCgiExternalServer: http://www.fastcgi.com/drupal/node/25#FastCgiExternalServer
|
|
.. _Apache Virtual Host documentation: http://httpd.apache.org/docs/2.2/vhosts/
|
|
.. _<IfModule> Directive: http://httpd.apache.org/docs/2.2/mod/core.html#ifmodule
|
|
|
|
|
|
.. important:: If you are using CentOS, RHEL or a similar distribution, make
|
|
sure that ``FastCgiWrapper`` is turned ``off`` in
|
|
``/etc/httpd/conf.d/fastcgi.conf``. It is usually ``on`` by default.
|
|
|
|
For Debian/Ubuntu distributions, enable the site for ``rgw.conf``. ::
|
|
|
|
sudo a2ensite rgw.conf
|
|
|
|
Then, disable the default site. ::
|
|
|
|
sudo a2dissite default
|
|
|
|
|
|
Adjust Path Ownership/Permissions
|
|
=================================
|
|
|
|
On some distributions, you must change ownership for ``/var/log/httpd`` or
|
|
``/var/log/apache2`` and ``/var/run/ceph`` to ensure that Apache has permissions
|
|
to create a socket or log file. ::
|
|
|
|
sudo chown apache:apache /path/to/file
|
|
|
|
On some systems, you may need to set SELinux to ``Permissive``. If you are
|
|
unable to communicate with the gateway after attempting to start it, try
|
|
executing::
|
|
|
|
getenforce
|
|
|
|
If the result is ``1`` or ``Enforcing``, execute::
|
|
|
|
sudo setenforce 0
|
|
|
|
Then, restart Apache and the gateway daemon to see if that resolves the issue.
|
|
If it does, you can configure your system to disable SELinux.
|
|
|
|
|
|
Restart Services and Start the Gateway
|
|
======================================
|
|
|
|
To ensure that all components have reloaded their configurations, we recommend
|
|
restarting your ``ceph`` and ``apache`` services. Then, start up the
|
|
``radosgw`` service.
|
|
|
|
For the Ceph Storage Cluster, see `Operating a Cluster`_ for details. Some
|
|
versions of Ceph use different methods for starting and stopping clusters.
|
|
|
|
|
|
Restart Apache
|
|
--------------
|
|
|
|
On Debian/Ubuntu systems, use ``apache2``. For example::
|
|
|
|
sudo service apache2 restart
|
|
sudo /etc/init.d/apache2 restart
|
|
|
|
On CentOS/RHEL systems, use ``httpd``. For example::
|
|
|
|
sudo /etc/init.d/httpd restart
|
|
|
|
|
|
Start the Gateway
|
|
-----------------
|
|
|
|
On Debian/Ubuntu systems, use ``radosgw``. For example::
|
|
|
|
sudo /etc/init.d/radosgw start
|
|
|
|
On CentOS/RHEL systems, use ``ceph-radosgw``. For example::
|
|
|
|
sudo /etc/init.d/ceph-radosgw start
|
|
|
|
|
|
Verify the Runtime
|
|
------------------
|
|
|
|
Once the service is up and running, you can make an anonymous GET request to see
|
|
if the gateway returns a response. A simple HTTP request to the domain name
|
|
should return the following:
|
|
|
|
.. code-block:: xml
|
|
|
|
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Owner>
|
|
<ID>anonymous</ID>
|
|
<DisplayName/>
|
|
</Owner>
|
|
<Buckets/>
|
|
</ListAllMyBucketsResult>
|
|
|
|
|
|
If you receive an error, check your settings and try again. See
|
|
`Adjust Path Ownership/Permissions`_ for details.
|
|
|
|
Using The Gateway
|
|
=================
|
|
|
|
To use the REST interfaces, first create an initial Ceph Object Gateway user for
|
|
the S3 interface. Then, create a subuser for the swift interface. See the `Admin
|
|
Guide`_ for details.
|
|
|
|
|
|
|
|
.. _Dnsmasq: https://help.ubuntu.com/community/Dnsmasq
|
|
.. _Configuration Reference - Pools: ../config-ref#pools
|
|
.. _Pool Configuration: ../../rados/configuration/pool-pg-config-ref/
|
|
.. _Pools: ../../rados/operations/pools
|
|
.. _Cephx Guide: ../../rados/operations/authentication/#cephx-guide
|
|
.. _Operating a Cluster: ../../rados/rados/operations/operating
|
|
.. _Admin Guide: ../admin
|
|
.. _Add Wildcard to DNS: ../../install/install-ceph-gateway#add-wildcard-to-dns
|