mirror of
https://github.com/ceph/ceph
synced 2024-12-28 22:43:29 +00:00
d788bae60d
This file documents how to configure RGW to use Apache/FastCGI, so rename the file and modify the title and intro to make that clear. Also, add a note that CGI can pose a security risk - e.g. http://httpoxy.org Signed-off-by: Nathan Cutler <ncutler@suse.com>
596 lines
18 KiB
ReStructuredText
596 lines
18 KiB
ReStructuredText
=====================================================
|
||
Configuring Ceph Object Gateway with Apache/FastCGI
|
||
=====================================================
|
||
|
||
Configuring a Ceph Object Gateway requires a running Ceph Storage Cluster.
|
||
Since it contains an embedded webserver (civetweb), the Ceph Object Gateway
|
||
does not require an external web server, but it can be configured to use
|
||
Apache with the FastCGI module.
|
||
|
||
.. note:: CGI can pose a security risk.
|
||
|
||
The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a
|
||
Ceph Storage Cluster client, it requires:
|
||
|
||
- A name for the gateway instance. We use ``gateway`` in this guide.
|
||
- A storage cluster user name with appropriate permissions in a keyring.
|
||
- Pools to store its data.
|
||
- A data directory for the gateway instance.
|
||
- An instance entry in the Ceph Configuration file.
|
||
- A configuration file for the web server to interact with FastCGI.
|
||
|
||
|
||
Create a User and Keyring
|
||
=========================
|
||
|
||
Each instance must have a user name and key to communicate with a Ceph Storage
|
||
Cluster. In the following steps, we use an admin node to create a keyring.
|
||
Then, we create a client user name and key. Next, we add the
|
||
key to the Ceph Storage Cluster. Finally, we distribute the key ring to
|
||
the node containing the gateway instance.
|
||
|
||
.. topic:: Monitor Key CAPS
|
||
|
||
When you provide CAPS to the key, you MUST provide read capability.
|
||
However, you have the option of providing write capability for the monitor.
|
||
This is an important choice. If you provide write capability to the key,
|
||
the Ceph Object Gateway will have the ability to create pools automatically;
|
||
however, it will create pools with either the default number of placement
|
||
groups (not ideal) or the number of placement groups you specified in your
|
||
Ceph configuration file. If you allow the Ceph Object Gateway to create
|
||
pools automatically, ensure that you have reasonable defaults for the number
|
||
of placement groups first. See `Pool Configuration`_ for details.
|
||
|
||
|
||
See `User Management`_ for additional details on Ceph authentication.
|
||
|
||
#. Generate a Ceph Object Gateway user name and key for each instance. For
|
||
exemplary purposes, we will use the name ``gateway`` after ``client.radosgw``::
|
||
|
||
sudo ceph auth get-or-create client.radosgw.gateway osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.radosgw.keyring
|
||
|
||
#. Distribute the keyring to the node with the gateway instance. ::
|
||
|
||
sudo scp /etc/ceph/ceph.client.radosgw.keyring ceph@{hostname}:/home/ceph
|
||
ssh {hostname}
|
||
sudo mv ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring
|
||
|
||
|
||
.. note:: The 2nd step is optional if ``admin node`` is the ``gateway host``.
|
||
|
||
Create Pools
|
||
============
|
||
|
||
Ceph Object Gateways require Ceph Storage Cluster pools to store specific
|
||
gateway data. If the user you created has permissions, the gateway
|
||
will create the pools automatically. However, you should ensure that you have
|
||
set an appropriate default number of placement groups per pool into your Ceph
|
||
configuration file.
|
||
|
||
.. note:: Ceph Object Gateways have multiple pools, so don't make the number of
|
||
PGs too high considering all of the pools assigned to the same CRUSH
|
||
hierarchy, or performance may suffer.
|
||
|
||
When configuring a gateway with the default region and zone, the naming
|
||
convention for pools typically omits region and zone naming, but you can use any
|
||
naming convention you prefer. For example:
|
||
|
||
|
||
- ``.rgw.root``
|
||
- ``.rgw.control``
|
||
- ``.rgw.gc``
|
||
- ``.rgw.buckets``
|
||
- ``.rgw.buckets.index``
|
||
- ``.rgw.buckets.extra``
|
||
- ``.log``
|
||
- ``.intent-log``
|
||
- ``.usage``
|
||
- ``.users``
|
||
- ``.users.email``
|
||
- ``.users.swift``
|
||
- ``.users.uid``
|
||
|
||
|
||
See `Configuration Reference - Pools`_ for details on the default pools for
|
||
gateways. See `Pools`_ for details on creating pools. As already said, if
|
||
write permission is given, Ceph Object Gateway will create pools automatically.
|
||
To create a pool manually, execute the following::
|
||
|
||
ceph osd pool create {poolname} {pg-num} {pgp-num} {replicated | erasure} [{erasure-code-profile}] {ruleset-name} {ruleset-number}
|
||
|
||
|
||
.. tip:: Ceph supports multiple CRUSH hierarchies and CRUSH rulesets, enabling
|
||
great flexibility in the way you configure your gateway. Pools such as
|
||
``rgw.buckets.index`` may benefit from a pool of SSDs for fast performance.
|
||
Backing storage may benefit from the increased economy of erasure-coded
|
||
storage, and/or the improved performance from cache tiering.
|
||
|
||
When you have completed this step, execute the following to ensure that
|
||
you have created all of the foregoing pools::
|
||
|
||
rados lspools
|
||
|
||
|
||
Add a Gateway Configuration to Ceph
|
||
===================================
|
||
|
||
Add the Ceph Object Gateway configuration to your Ceph Configuration file in
|
||
``admin node``. The Ceph Object Gateway configuration requires you to
|
||
identify the Ceph Object Gateway instance. Then, you must specify the host name
|
||
where you installed the Ceph Object Gateway daemon, a keyring (for use with
|
||
cephx), the socket path for FastCGI and a log file.
|
||
|
||
For distros with Apache 2.2 and early versions of Apache 2.4 (RHEL 6, Ubuntu
|
||
12.04, 14.04 etc), append the following configuration to ``/etc/ceph/ceph.conf``
|
||
in your ``admin node``::
|
||
|
||
[client.radosgw.gateway]
|
||
host = {hostname}
|
||
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
||
rgw socket path = ""
|
||
log file = /var/log/radosgw/client.radosgw.gateway.log
|
||
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
|
||
rgw print continue = false
|
||
|
||
|
||
.. note:: Apache 2.2 and early versions of Apache 2.4 do not use Unix Domain
|
||
Sockets but use localhost TCP.
|
||
|
||
For distros with Apache 2.4.9 or later (RHEL 7, CentOS 7 etc), append the
|
||
following configuration to ``/etc/ceph/ceph.conf`` in your ``admin node``::
|
||
|
||
[client.radosgw.gateway]
|
||
host = {hostname}
|
||
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
||
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
|
||
log file = /var/log/radosgw/client.radosgw.gateway.log
|
||
rgw print continue = false
|
||
|
||
|
||
.. note:: ``Apache 2.4.9`` supports Unix Domain Socket (UDS) but as
|
||
``Ubuntu 14.04`` ships with ``Apache 2.4.7`` it doesn't have UDS support and
|
||
has to be configured for use with localhost TCP. A bug has been filed for
|
||
backporting UDS support in ``Apache 2.4.7`` for ``Ubuntu 14.04``.
|
||
See: `Backport support for UDS in Ubuntu Trusty`_
|
||
|
||
Here, ``{hostname}`` is the short hostname (output of command ``hostname -s``)
|
||
of the node that is going to provide the gateway service i.e., the
|
||
``gateway host``.
|
||
|
||
The ``[client.radosgw.gateway]`` portion of the gateway instance identifies this
|
||
portion of the Ceph configuration file as configuring a Ceph Storage Cluster
|
||
client where the client type is a Ceph Object Gateway (i.e., ``radosgw``).
|
||
|
||
|
||
.. note:: The last line in the configuration i.e., ``rgw print continue = false``
|
||
is added to avoid issues with ``PUT`` operations.
|
||
|
||
Once you finish the setup procedure, if you encounter issues with your
|
||
configuration, you can add debugging to the ``[global]`` section of your Ceph
|
||
configuration file and restart the gateway to help troubleshoot any
|
||
configuration issues. For example::
|
||
|
||
[global]
|
||
#append the following in the global section.
|
||
debug ms = 1
|
||
debug rgw = 20
|
||
|
||
|
||
Distribute updated Ceph configuration file
|
||
==========================================
|
||
|
||
The updated Ceph configuration file needs to be distributed to all Ceph cluster
|
||
nodes from the ``admin node``.
|
||
|
||
It involves the following steps:
|
||
|
||
#. Pull the updated ``ceph.conf`` from ``/etc/ceph/`` to the root directory of
|
||
the cluster in admin node (e.g. ``my-cluster`` directory). The contents of
|
||
``ceph.conf`` in ``my-cluster`` will get overwritten. To do so, execute the
|
||
following::
|
||
|
||
ceph-deploy --overwrite-conf config pull {hostname}
|
||
|
||
Here, ``{hostname}`` is the short hostname of the Ceph admin node.
|
||
|
||
#. Push the updated ``ceph.conf`` file from the admin node to all other nodes in
|
||
the cluster including the ``gateway host``::
|
||
|
||
ceph-deploy --overwrite-conf config push [HOST] [HOST...]
|
||
|
||
Give the hostnames of the other Ceph nodes in place of ``[HOST] [HOST...]``.
|
||
|
||
|
||
Copy ceph.client.admin.keyring from admin node to gateway host
|
||
==============================================================
|
||
|
||
As the ``gateway host`` can be a different node that is not part of the cluster,
|
||
the ``ceph.client.admin.keyring`` needs to be copied from the ``admin node`` to
|
||
the ``gateway host``. To do so, execute the following on ``admin node``::
|
||
|
||
sudo scp /etc/ceph/ceph.client.admin.keyring ceph@{hostname}:/home/ceph
|
||
ssh {hostname}
|
||
sudo mv ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring
|
||
|
||
|
||
.. note:: The above step need not be executed if ``admin node`` is the
|
||
``gateway host``.
|
||
|
||
|
||
Create Data Directory
|
||
=====================
|
||
|
||
Deployment scripts may not create the default Ceph Object Gateway data
|
||
directory. Create data directories for each instance of a ``radosgw``
|
||
daemon (if you haven't done so already). The ``host`` variables in the
|
||
Ceph configuration file determine which host runs each instance of a
|
||
``radosgw`` daemon. The typical form specifies the ``radosgw`` daemon,
|
||
the cluster name and the daemon ID.
|
||
|
||
To create the directory on the ``gateway host``, execute the following::
|
||
|
||
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway
|
||
|
||
|
||
Adjust Socket Directory Permissions
|
||
===================================
|
||
|
||
On some distros, the ``radosgw`` daemon runs as the unprivileged ``apache``
|
||
UID, and this UID must have write access to the location where it will write
|
||
its socket file.
|
||
|
||
To grant permissions to the default socket location, execute the following on
|
||
the ``gateway host``::
|
||
|
||
sudo chown apache:apache /var/run/ceph
|
||
|
||
|
||
Change Log File Owner
|
||
=====================
|
||
|
||
On some distros, the ``radosgw`` daemon runs as the unprivileged ``apache`` UID,
|
||
but the ``root`` user owns the log file by default. You must change it to the
|
||
``apache`` user so that Apache can populate the log file. To do so, execute
|
||
the following::
|
||
|
||
sudo chown apache:apache /var/log/radosgw/client.radosgw.gateway.log
|
||
|
||
|
||
Start radosgw service
|
||
=====================
|
||
|
||
The Ceph Object gateway daemon needs to be started. To do so, execute the
|
||
following on the ``gateway host``:
|
||
|
||
On Debian-based distros::
|
||
|
||
sudo /etc/init.d/radosgw start
|
||
|
||
On RPM-based distros::
|
||
|
||
sudo /etc/init.d/ceph-radosgw start
|
||
|
||
|
||
Create a Gateway Configuration file
|
||
===================================
|
||
|
||
On the host where you installed the Ceph Object Gateway i.e., ``gateway host``,
|
||
create an ``rgw.conf`` file. Place the file in ``/etc/apache2/conf-available``
|
||
directory for ``Debian-based`` distros and in ``/etc/httpd/conf.d`` directory
|
||
for ``RPM-based`` distros. It is a Apache configuration file which is needed
|
||
for the ``radosgw`` service. This file must be readable by the web server.
|
||
|
||
Execute the following steps:
|
||
|
||
#. Create the file:
|
||
|
||
For Debian-based distros, execute::
|
||
|
||
sudo vi /etc/apache2/conf-available/rgw.conf
|
||
|
||
For RPM-based distros, execute::
|
||
|
||
sudo vi /etc/httpd/conf.d/rgw.conf
|
||
|
||
#. For distros with Apache 2.2 and early versions of Apache 2.4 that use
|
||
localhost TCP and do not support Unix Domain Socket, add the following
|
||
contents to the file::
|
||
|
||
<VirtualHost *:80>
|
||
ServerName localhost
|
||
DocumentRoot /var/www/html
|
||
|
||
ErrorLog /var/log/httpd/rgw_error.log
|
||
CustomLog /var/log/httpd/rgw_access.log combined
|
||
|
||
# LogLevel debug
|
||
|
||
RewriteEngine On
|
||
|
||
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
|
||
|
||
SetEnv proxy-nokeepalive 1
|
||
|
||
ProxyPass / fcgi://localhost:9000/
|
||
|
||
</VirtualHost>
|
||
|
||
.. note:: For Debian-based distros replace ``/var/log/httpd/``
|
||
with ``/var/log/apache2``.
|
||
|
||
#. For distros with Apache 2.4.9 or later that support Unix Domain Socket,
|
||
add the following contents to the file::
|
||
|
||
<VirtualHost *:80>
|
||
ServerName localhost
|
||
DocumentRoot /var/www/html
|
||
|
||
ErrorLog /var/log/httpd/rgw_error.log
|
||
CustomLog /var/log/httpd/rgw_access.log combined
|
||
|
||
# LogLevel debug
|
||
|
||
RewriteEngine On
|
||
|
||
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
|
||
|
||
SetEnv proxy-nokeepalive 1
|
||
|
||
ProxyPass / unix:///var/run/ceph/ceph.radosgw.gateway.fastcgi.sock|fcgi://localhost:9000/
|
||
|
||
</VirtualHost>
|
||
|
||
|
||
Restart Apache
|
||
==============
|
||
|
||
The Apache service needs to be restarted to accept the new configuration.
|
||
|
||
For Debian-based distros, run::
|
||
|
||
sudo service apache2 restart
|
||
|
||
For RPM-based distros, run::
|
||
|
||
sudo service httpd restart
|
||
|
||
Or::
|
||
|
||
sudo systemctl restart httpd
|
||
|
||
|
||
Using The Gateway
|
||
=================
|
||
|
||
To use the REST interfaces, first create an initial Ceph Object Gateway
|
||
user for the S3 interface. Then, create a subuser for the Swift interface.
|
||
See the `Admin Guide`_ for more details on user management.
|
||
|
||
Create a radosgw user for S3 access
|
||
------------------------------------
|
||
|
||
A ``radosgw`` user needs to be created and granted access. The command
|
||
``man radosgw-admin`` will provide information on additional command options.
|
||
|
||
To create the user, execute the following on the ``gateway host``::
|
||
|
||
sudo radosgw-admin user create --uid="testuser" --display-name="First User"
|
||
|
||
The output of the command will be something like the following::
|
||
|
||
{"user_id": "testuser",
|
||
"display_name": "First User",
|
||
"email": "",
|
||
"suspended": 0,
|
||
"max_buckets": 1000,
|
||
"auid": 0,
|
||
"subusers": [],
|
||
"keys": [
|
||
{ "user": "testuser",
|
||
"access_key": "I0PJDPCIYZ665MW88W9R",
|
||
"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"}],
|
||
"swift_keys": [],
|
||
"caps": [],
|
||
"op_mask": "read, write, delete",
|
||
"default_placement": "",
|
||
"placement_tags": [],
|
||
"bucket_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"user_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"temp_url_keys": []}
|
||
|
||
|
||
.. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are
|
||
needed for access validation.
|
||
|
||
Create a Swift user
|
||
-------------------
|
||
|
||
A Swift subuser needs to be created if this kind of access is needed. Creating
|
||
a Swift user is a two step process. The first step is to create the user.
|
||
The second is to create the secret key.
|
||
|
||
Execute the following steps on the ``gateway host``:
|
||
|
||
Create the Swift user::
|
||
|
||
sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
|
||
|
||
The output will be something like the following::
|
||
|
||
{ "user_id": "testuser",
|
||
"display_name": "First User",
|
||
"email": "",
|
||
"suspended": 0,
|
||
"max_buckets": 1000,
|
||
"auid": 0,
|
||
"subusers": [
|
||
{ "id": "testuser:swift",
|
||
"permissions": "full-control"}],
|
||
"keys": [
|
||
{ "user": "testuser:swift",
|
||
"access_key": "3Y1LNW4Q6X0Y53A52DET",
|
||
"secret_key": ""},
|
||
{ "user": "testuser",
|
||
"access_key": "I0PJDPCIYZ665MW88W9R",
|
||
"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"}],
|
||
"swift_keys": [],
|
||
"caps": [],
|
||
"op_mask": "read, write, delete",
|
||
"default_placement": "",
|
||
"placement_tags": [],
|
||
"bucket_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"user_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"temp_url_keys": []}
|
||
|
||
Create the secret key::
|
||
|
||
sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
|
||
|
||
The output will be something like the following::
|
||
|
||
{ "user_id": "testuser",
|
||
"display_name": "First User",
|
||
"email": "",
|
||
"suspended": 0,
|
||
"max_buckets": 1000,
|
||
"auid": 0,
|
||
"subusers": [
|
||
{ "id": "testuser:swift",
|
||
"permissions": "full-control"}],
|
||
"keys": [
|
||
{ "user": "testuser:swift",
|
||
"access_key": "3Y1LNW4Q6X0Y53A52DET",
|
||
"secret_key": ""},
|
||
{ "user": "testuser",
|
||
"access_key": "I0PJDPCIYZ665MW88W9R",
|
||
"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"}],
|
||
"swift_keys": [
|
||
{ "user": "testuser:swift",
|
||
"secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA"}],
|
||
"caps": [],
|
||
"op_mask": "read, write, delete",
|
||
"default_placement": "",
|
||
"placement_tags": [],
|
||
"bucket_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"user_quota": { "enabled": false,
|
||
"max_size_kb": -1,
|
||
"max_objects": -1},
|
||
"temp_url_keys": []}
|
||
|
||
Access Verification
|
||
===================
|
||
|
||
You then need to verify if the created users are able to access the gateway.
|
||
|
||
Test S3 access
|
||
--------------
|
||
|
||
You need to write and run a Python test script for verifying S3 access. The S3
|
||
access test script will connect to the ``radosgw``, create a new bucket and list
|
||
all buckets. The values for ``aws_access_key_id`` and ``aws_secret_access_key``
|
||
are taken from the values of ``access_key`` and ``secret_key`` returned by the
|
||
``radosgw_admin`` command.
|
||
|
||
Execute the following steps:
|
||
|
||
#. You will need to install the ``python-boto`` package.
|
||
|
||
For Debian-based distros, run::
|
||
|
||
sudo apt-get install python-boto
|
||
|
||
For RPM-based distros, run::
|
||
|
||
sudo yum install python-boto
|
||
|
||
#. Create the Python script::
|
||
|
||
vi s3test.py
|
||
|
||
#. Add the following contents to the file::
|
||
|
||
import boto
|
||
import boto.s3.connection
|
||
access_key = 'I0PJDPCIYZ665MW88W9R'
|
||
secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA'
|
||
conn = boto.connect_s3(
|
||
aws_access_key_id = access_key,
|
||
aws_secret_access_key = secret_key,
|
||
host = '{hostname}',
|
||
is_secure=False,
|
||
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
|
||
)
|
||
bucket = conn.create_bucket('my-new-bucket')
|
||
for bucket in conn.get_all_buckets():
|
||
print "{name}\t{created}".format(
|
||
name = bucket.name,
|
||
created = bucket.creation_date,
|
||
)
|
||
|
||
Replace ``{hostname}`` with the hostname of the host where you have
|
||
configured the gateway service i.e., the ``gateway host``.
|
||
|
||
#. Run the script::
|
||
|
||
python s3test.py
|
||
|
||
The output will be something like the following::
|
||
|
||
my-new-bucket 2015-02-16T17:09:10.000Z
|
||
|
||
Test swift access
|
||
-----------------
|
||
|
||
Swift access can be verified via the ``swift`` command line client. The command
|
||
``man swift`` will provide more information on available command line options.
|
||
|
||
To install ``swift`` client, execute the following:
|
||
|
||
For Debian-based distros::
|
||
|
||
sudo apt-get install python-setuptools
|
||
sudo easy_install pip
|
||
sudo pip install --upgrade setuptools
|
||
sudo pip install --upgrade python-swiftclient
|
||
|
||
For RPM-based distros::
|
||
|
||
sudo yum install python-setuptools
|
||
sudo easy_install pip
|
||
sudo pip install --upgrade setuptools
|
||
sudo pip install --upgrade python-swiftclient
|
||
|
||
To test swift access, execute the following::
|
||
|
||
swift -A http://{IP ADDRESS}/auth/1.0 -U testuser:swift -K ‘{swift_secret_key}’ list
|
||
|
||
Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and
|
||
``{swift_secret_key}`` with its value from the output of
|
||
``radosgw-admin key create`` command executed for the ``swift`` user.
|
||
|
||
For example::
|
||
|
||
swift -A http://10.19.143.116/auth/1.0 -U testuser:swift -K ‘244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA’ list
|
||
|
||
The output should be::
|
||
|
||
my-new-bucket
|
||
|
||
|
||
.. _Configuration Reference - Pools: ../config-ref#pools
|
||
.. _Pool Configuration: ../../rados/configuration/pool-pg-config-ref/
|
||
.. _Pools: ../../rados/operations/pools
|
||
.. _User Management: ../../rados/operations/user-management
|
||
.. _Backport support for UDS in Ubuntu Trusty: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1411030
|
||
.. _Admin Guide: ../admin
|