mirror of https://github.com/ceph/ceph
820 lines
28 KiB
ReStructuredText
820 lines
28 KiB
ReStructuredText
================================
|
|
Configuring Federated Gateways
|
|
================================
|
|
|
|
.. versionadded:: 0.67 Dumpling
|
|
|
|
In Ceph version 0.67 Dumpling and beyond, you may configure each :term:`Ceph
|
|
Object Gateway` to participate in a federated architecture, with multiple
|
|
regions, and with multiple zones for a region.
|
|
|
|
- **Region**: A region represents a *logical* geographic area and contains one
|
|
or more zones. A cluster with multiple regions must specify a master region.
|
|
|
|
- **Zone**: A zone is a *logical* grouping of one or more Ceph Object Gateway
|
|
instance(s). A region has a master zone that processes client requests.
|
|
|
|
.. important:: Only write objects to the master zone in a region. You may read
|
|
objects from secondary zones. Currently, the Gateway does not prevent you
|
|
from writing to a secondary zone, but **DON'T DO IT**.
|
|
|
|
|
|
Background
|
|
==========
|
|
|
|
When you deploy a :term:`Ceph Object Store` service that spans geographical
|
|
locales, configuring Ceph Object Gateway regions and metadata synchronization
|
|
agents enables the service to maintain a global namespace, even though Ceph
|
|
Object Gateway instances run in different geographic locales and potentially on
|
|
different Ceph Storage Clusters. When you separate one or more Ceph Object
|
|
Gateway instances within a region into separate logical containers to maintain
|
|
an extra copy (or copies) of the data, configuring Ceph Object Gateway zones and
|
|
data synchronization agents enables the service to maintain one or more
|
|
copy(ies) of the master zone's data. Extra copies of the data are important for
|
|
failover, backup and disaster recovery.
|
|
|
|
You may deploy a single Ceph Storage Cluster with a federated architecture if
|
|
you have low latency network connections (this isn't recommended). You may also
|
|
deploy one Ceph Storage Cluster per region with a separate set of pools for
|
|
each zone (typical). You may also deploy a separate Ceph Storage Cluster for
|
|
each zone if your requirements and resources warrant this level of redundancy.
|
|
|
|
|
|
About this Guide
|
|
================
|
|
|
|
In the following sections, we will demonstrate how to configure a federated
|
|
cluster in two logical steps:
|
|
|
|
- **Configure a Master Region:** This section of the guide describes how to
|
|
set up a region with multiple zones, and how to synchronize data between the
|
|
master zone and the secondary zone(s) within the master region.
|
|
|
|
- **Configure a Secondary Region:** This section of the guide describes how
|
|
to repeat the section on setting up a master region and multiple zones so
|
|
that you have two regions with intra-zone synchronization in each region.
|
|
Finally, you will learn how to set up a metadata synchronization agent so
|
|
that you can maintain a global namespace for the regions in your cluster.
|
|
|
|
|
|
|
|
Configure a Master Region
|
|
=========================
|
|
|
|
This section provides an exemplary procedure for setting up a region, and two
|
|
zones within the region. The cluster will comprise two gateway daemon
|
|
instances--one per zone. This region will serve as the master region.
|
|
|
|
|
|
Naming for the Master Region
|
|
----------------------------
|
|
|
|
Before configuring the cluster, defining region, zone and instance names will
|
|
help you manage your cluster. Let's assume the region represents the United
|
|
States, and we refer to it by its standard abbreviation.
|
|
|
|
- United States: ``us``
|
|
|
|
Let's assume the zones represent the Eastern and Western United States. For
|
|
continuity, our naming convention will use ``{region name}-{zone name}`` format,
|
|
but you can use any naming convention you prefer.
|
|
|
|
- United States, East Region: ``us-east``
|
|
- United States, West Region: ``us-west``
|
|
|
|
Finally, let's assume that zones may have more than one Ceph Object Gateway
|
|
instance per zone. For continuity, our naming convention will use
|
|
``{region name}-{zone name}-{instance}`` format, but you can use any naming
|
|
convention you prefer.
|
|
|
|
|
|
- United States Region, Master Zone, Instance 1: ``us-east-1``
|
|
- United States Region, Secondary Zone, Instance 1: ``us-west-1``
|
|
|
|
|
|
Create Pools
|
|
------------
|
|
|
|
You may have a Ceph Storage Cluster for the entire region or a Ceph Storage
|
|
Cluster for each zone.
|
|
|
|
For continuity, our naming convention will use ``{region name}-{zone name}``
|
|
format prepended to the pool name, but you can use any naming convention you
|
|
prefer. For example:
|
|
|
|
|
|
- ``.us.rgw.root``
|
|
|
|
- ``.us-east.domain.rgw``
|
|
- ``.us-east.rgw.root``
|
|
- ``.us-east.rgw.control``
|
|
- ``.us-east.rgw.gc``
|
|
- ``.us-east.rgw.buckets.index``
|
|
- ``.us-east.rgw.buckets``
|
|
- ``.us-east.log``
|
|
- ``.us-east.intent-log``
|
|
- ``.us-east.usage``
|
|
- ``.us-east.users``
|
|
- ``.us-east.users.email``
|
|
- ``.us-east.users.swift``
|
|
- ``.us-east.users.uid``
|
|
|
|
- ``.us-west.domain.rgw``
|
|
- ``.us-west.rgw.root``
|
|
- ``.us-west.rgw.control``
|
|
- ``.us-west.rgw.gc``
|
|
- ``.us-west.rgw.buckets.index``
|
|
- ``.us-west.rgw.buckets``
|
|
- ``.us-west.log``
|
|
- ``.us-west.intent-log``
|
|
- ``.us-west.usage``
|
|
- ``.us-west.users``
|
|
- ``.us-west.users.email``
|
|
- ``.us-west.users.swift``
|
|
- ``.us-west.users.uid``
|
|
|
|
See `Configuration Reference - Pools`_ for details on the default pools for
|
|
gateways. See `Pools`_ for details on creating pools. Execute the following
|
|
to create a pool::
|
|
|
|
ceph osd pool create {poolname} {pg-num} {pgp-num}
|
|
|
|
|
|
.. tip:: When adding a large number of pools, it may take some time for your
|
|
cluster to return to a ``active + clean`` state.
|
|
|
|
.. topic:: CRUSH Maps
|
|
|
|
When deploying a Ceph Storage Cluster for the entire region, consider
|
|
using a CRUSH rule for the zone such that you do NOT have overlapping
|
|
failure domains. See `CRUSH Map`_ for details.
|
|
|
|
When you have completed this step, execute the following to ensure that
|
|
you have created all of the foregoing pools::
|
|
|
|
rados lspools
|
|
|
|
|
|
Create a Keyring
|
|
----------------
|
|
|
|
Each instance must have a user name and key to communicate with a Ceph Storage
|
|
Cluster. In the following steps, we use an admin node to create a keyring.
|
|
Then, we create a client user name and key for each instance. Next, we add the
|
|
keys to the Ceph Storage Cluster(s). Finally, we distribute the key ring to
|
|
each node containing an instance.
|
|
|
|
#. Create a keyring. ::
|
|
|
|
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
|
|
sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
#. Generate a Ceph Object Gateway user name and key for each instance. ::
|
|
|
|
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-east-1 --gen-key
|
|
sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-west-1 --gen-key
|
|
|
|
|
|
#. Add capabilities to each key. See `Configuration Reference - Pools`_ for details
|
|
on the effect of write permissions for the monitor and creating pools. ::
|
|
|
|
sudo ceph-authtool -n client.radosgw.us-east-1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
|
|
sudo ceph-authtool -n client.radosgw.us-west-1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
#. Once you have created a keyring and key to enable the Ceph Object Gateway
|
|
with access to the Ceph Storage Cluster, add each key as an entry to your
|
|
Ceph Storage Cluster(s). For example::
|
|
|
|
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring
|
|
sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring
|
|
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four users **after**
|
|
you create the master region and the secondary region.
|
|
|
|
|
|
Install Apache/FastCGI
|
|
----------------------
|
|
|
|
For each :term:`Ceph Node` that runs a :term:`Ceph Object Gateway` daemon
|
|
instance, you must install Apache, FastCGI, the Ceph Object Gateway daemon
|
|
(``radosgw``) and the Ceph Object Gateway Sync Agent (``radosgw-agent``).
|
|
See `Install Ceph Object Gateway`_ for details.
|
|
|
|
|
|
Create Data Directories
|
|
-----------------------
|
|
|
|
Create data directories for each daemon instance on their respective
|
|
hosts. ::
|
|
|
|
ssh {us-east-1}
|
|
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.us-east-1
|
|
|
|
ssh {us-west-1}
|
|
sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.us-west-1
|
|
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four data directories
|
|
**after** you create the master region and the secondary region.
|
|
|
|
|
|
Create a Gateway Configuration
|
|
------------------------------
|
|
|
|
For each instance, create an Ceph Object Gateway configuration file under the
|
|
``/etc/apache2/sites-available`` directory on the host(s) where you installed
|
|
the Ceph Object Gateway daemon(s). See below for an exemplary embodiment of a
|
|
gateway configuration as discussed in the following text.
|
|
|
|
.. literalinclude:: rgw.conf
|
|
:language: ini
|
|
|
|
#. Replace the ``/{path}/{socket-name}`` entry with path to the socket and
|
|
the socket name. For example,
|
|
``/var/run/ceph/client.radosgw.us-east-1.sock``. Ensure that you use the
|
|
same path and socket name in your ``ceph.conf`` entry.
|
|
|
|
#. Replace the ``{fqdn}`` entry with the fully-qualified domain name of the
|
|
server.
|
|
|
|
#. Replace the ``{email.address}`` entry with the email address for the
|
|
server administrator.
|
|
|
|
#. Add a ``ServerAlias`` if you wish to use S3-style subdomains
|
|
(of course you do).
|
|
|
|
#. Save the configuration to a file (e.g., ``rgw-us-east.conf``).
|
|
|
|
Repeat the process for the secondary zone (e.g., ``rgw-us-west.conf``).
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four gateway
|
|
configuration files on the respective nodes **after**
|
|
you create the master region and the secondary region.
|
|
|
|
|
|
Finally, if you enabled SSL, make sure that you set the port to your SSL port
|
|
(usually 443) and your configuration file includes the following::
|
|
|
|
SSLEngine on
|
|
SSLCertificateFile /etc/apache2/ssl/apache.crt
|
|
SSLCertificateKeyFile /etc/apache2/ssl/apache.key
|
|
SetEnv SERVER_PORT_SECURE 443
|
|
|
|
|
|
Enable the Configuration
|
|
------------------------
|
|
|
|
For each instance, enable the gateway configuration and disable the
|
|
default site.
|
|
|
|
#. Enable the site for the gateway configuration. ::
|
|
|
|
sudo a2ensite {rgw-conf-filename}
|
|
|
|
#. Disable the default site. ::
|
|
|
|
sudo a2dissite default
|
|
|
|
.. note:: Failure to disable the default site can lead to problems.
|
|
|
|
|
|
Add a FastCGI Script
|
|
--------------------
|
|
|
|
FastCGI requires a script for each Ceph Object Gateway instance to
|
|
enable the S3-compatible interface. To create the script, execute
|
|
the following procedures.
|
|
|
|
|
|
#. Go to the ``/var/www`` directory. ::
|
|
|
|
cd /var/www
|
|
|
|
#. Open an editor with the file name ``s3gw.fcgi``. **Note:** The configuration
|
|
file specifies this filename. ::
|
|
|
|
sudo vim s3gw.fcgi
|
|
|
|
#. Add a shell script with ``exec`` and the path to the gateway binary,
|
|
the path to the Ceph configuration file, and the user name (``-n``;
|
|
the same user name created in step 2 of `Create a Keyring`_.
|
|
Copy the following into the editor. ::
|
|
|
|
#!/bin/sh
|
|
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.{ID}
|
|
|
|
For example::
|
|
|
|
#!/bin/sh
|
|
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.us-east-1
|
|
|
|
#. Save the file.
|
|
|
|
#. Change the permissions on the file so that it is executable. ::
|
|
|
|
sudo chmod +x s3gw.fcgi
|
|
|
|
|
|
Repeat the process for the secondary zone.
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four FastCGI scripts
|
|
**after** you create the master region and the secondary region.
|
|
|
|
|
|
|
|
Add Instances to Ceph Config File
|
|
---------------------------------
|
|
|
|
On an admin node, add an entry for each instance in the Ceph configuration file
|
|
for your Ceph Storage Cluster(s). For example::
|
|
|
|
...
|
|
|
|
[client.radosgw.us-east-1]
|
|
rgw region = us
|
|
rgw region root pool = .us.rgw.root
|
|
rgw zone = us-east
|
|
rgw zone root pool = .us-east.rgw.root
|
|
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
|
rgw dns name = {hostname}
|
|
rgw socket path = /var/run/ceph/$name.sock
|
|
host = {host-name}
|
|
|
|
[client.radosgw.us-west-1]
|
|
rgw region = us
|
|
rgw region root pool = .us.rgw.root
|
|
rgw zone = us-west
|
|
rgw zone root pool = .us-west.rgw.root
|
|
keyring = /etc/ceph/ceph.client.radosgw.keyring
|
|
rgw dns name = {hostname}
|
|
rgw socket path = /var/run/ceph/$name.sock
|
|
host = {host-name}
|
|
|
|
|
|
Then, update each :term:`Ceph Node` with the updated Ceph configuration
|
|
file. For example::
|
|
|
|
ceph-deploy --overwrite-conf config push {node1} {node2} {nodex}
|
|
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us`` with ``eu`` for the name, region, pools and zones.
|
|
You will have a total of four entries **after**
|
|
you create the master region and the secondary region.
|
|
|
|
|
|
|
|
Create a Region
|
|
---------------
|
|
|
|
#. Configure a region infile called ``us.json`` for the ``us`` region.
|
|
|
|
Copy the contents of the following example to a text editor. Set
|
|
``is_master`` to ``true``. Replace ``{fqdn}`` with the fully-qualified
|
|
domain name of the endpoint. It will specify a master zone as ``us-east``
|
|
and list it in the ``zones`` list along with the ``us-west`` zone.
|
|
See `Configuration Reference - Regions`_ for details.::
|
|
|
|
{ "name": "us",
|
|
"api_name": "us",
|
|
"is_master": "true",
|
|
"endpoints": [
|
|
"http:\/\/{fqdn}:80\/"],
|
|
"master_zone": "us-east",
|
|
"zones": [
|
|
{ "name": "us-east",
|
|
"endpoints": [
|
|
"http:\/\/{fqdn}:80\/"],
|
|
"log_meta": "true",
|
|
"log_data": "true"},
|
|
{ "name": "us-west",
|
|
"endpoints": [
|
|
"http:\/\/{fqdn}:80\/"],
|
|
"log_meta": "true",
|
|
"log_data": "true"}],
|
|
"placement_targets": [
|
|
{
|
|
"name": "default-placement",
|
|
"tags": []
|
|
}
|
|
],
|
|
"default_placement": "default-placement"}
|
|
|
|
|
|
#. Create the ``us`` region using the ``us.json`` infile you just
|
|
created. ::
|
|
|
|
radosgw-admin region set --infile us.json --name client.radosgw.us-east-1
|
|
|
|
#. Delete the default region (if it exists). ::
|
|
|
|
rados -p .us.rgw.root rm region_info.default
|
|
|
|
#. Set the ``us`` region as the default region. ::
|
|
|
|
radosgw-admin region default --rgw-region=us --name client.radosgw.us-east-1
|
|
|
|
Only one region can be the default region for a cluster.
|
|
|
|
#. Update the region map. ::
|
|
|
|
radosgw-admin regionmap update --name client.radosgw.us-east-1
|
|
|
|
|
|
If you use different Ceph Storage Cluster instances for regions, you should
|
|
repeat steps 2, 4 and 5 in by executing them with ``--name
|
|
client.radosgw-us-west-1``. You may also export the region map from the initial
|
|
gateway instance and import it followed by updating the region map.
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us`` with ``eu``. You will have a total of two regions **after**
|
|
you create the master region and the secondary region.
|
|
|
|
|
|
|
|
Create Zones
|
|
------------
|
|
|
|
#. Configure a zone infile called ``us-east.json`` for the
|
|
``us-east`` zone.
|
|
|
|
Copy the contents of the following example to a text editor.
|
|
This configuration uses pool names prepended with the region name and
|
|
zone name. See `Configuration Reference - Pools`_ for additional details on
|
|
gateway pools. See `Configuration Reference - Zones`_ for additional
|
|
details on zones. ::
|
|
|
|
{ "domain_root": ".us-east.domain.rgw",
|
|
"control_pool": ".us-east.rgw.control",
|
|
"gc_pool": ".us-east.rgw.gc",
|
|
"log_pool": ".us-east.log",
|
|
"intent_log_pool": ".us-east.intent-log",
|
|
"usage_log_pool": ".us-east.usage",
|
|
"user_keys_pool": ".us-east.users",
|
|
"user_email_pool": ".us-east.users.email",
|
|
"user_swift_pool": ".us-east.users.swift",
|
|
"user_uid_pool": ".us-east.users.uid",
|
|
"system_key": { "access_key": "", "secret_key": ""},
|
|
"placement_pools": [
|
|
{ "key": "default-placement",
|
|
"val": { "index_pool": ".us-east.rgw.buckets.index",
|
|
"data_pool": ".us-east.rgw.buckets"}
|
|
}
|
|
]
|
|
}
|
|
|
|
|
|
#. Add the ``us-east`` zone using the ``us-east.json`` infile you
|
|
just created in both the east and west pools by specifying their respective
|
|
user names (i.e., ``--name``). ::
|
|
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1
|
|
|
|
Repeat step 1 to create a zone infile for ``us-west``. Then add the zone
|
|
using the ``us-west.json`` infile in both the east and west pools by
|
|
specifying their respective user names (i.e., ``--name``). ::
|
|
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1
|
|
|
|
|
|
#. Delete the default zone (if it exists). ::
|
|
|
|
rados -p .rgw.root rm zone_info.default
|
|
|
|
|
|
#. Update the region map. ::
|
|
|
|
radosgw-admin regionmap update --name client.radosgw.us-east-1
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four zones **after**
|
|
you create the master zone and the secondary zone in each region.
|
|
|
|
|
|
Create Zone Users
|
|
-----------------
|
|
|
|
Ceph Object Gateway stores zone users in the zone pools. So you must create zone
|
|
users after configuring the zones. Copy the ``access_key`` and ``secret_key``
|
|
fields for each user so you can update your zone configuration once you complete
|
|
this step. ::
|
|
|
|
radosgw-admin user create --uid="us-east" --display-name="Region-US Zone-East" --name client.radosgw.us-east-1 --system
|
|
radosgw-admin user create --uid="us-west" --display-name="Region-US Zone-West" --name client.radosgw.us-west-1 --system
|
|
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four zone users
|
|
**after** you create the master region and the secondary region and their
|
|
zones. These users are different from the users created in `Create a
|
|
Keyring`_.
|
|
|
|
|
|
Update Zone Configurations
|
|
--------------------------
|
|
|
|
You must update the zone configuration with zone users so that
|
|
the synchronization agents can authenticate with the zones.
|
|
|
|
#. Open your ``us-east.json`` zone configuration file and paste the contents of
|
|
the ``access_key`` and ``secret_key`` fields from the step of creating
|
|
zone users into the ``system_key`` field of your zone configuration
|
|
infile. ::
|
|
|
|
{ "domain_root": ".us-east.domain.rgw",
|
|
"control_pool": ".us-east.rgw.control",
|
|
"gc_pool": ".us-east.rgw.gc",
|
|
"log_pool": ".us-east.log",
|
|
"intent_log_pool": ".us-east.intent-log",
|
|
"usage_log_pool": ".us-east.usage",
|
|
"user_keys_pool": ".us-east.users",
|
|
"user_email_pool": ".us-east.users.email",
|
|
"user_swift_pool": ".us-east.users.swift",
|
|
"user_uid_pool": ".us-east.users.uid",
|
|
"system_key": {
|
|
"access_key": "{paste-access_key-here}",
|
|
"secret_key": "{paste-secret_key-here}"
|
|
},
|
|
"placement_pools": [
|
|
{ "key": "default-placement",
|
|
"val": { "index_pool": ".us-east.rgw.buckets.index",
|
|
"data_pool": ".us-east.rgw.buckets"}
|
|
}
|
|
]
|
|
}
|
|
|
|
#. Save the ``us-east.json`` file. Then, update your zone configuration. ::
|
|
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1
|
|
|
|
#. Repeat step 1 to update the zone infile for ``us-west``. Then, update
|
|
your zone configuration. ::
|
|
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1
|
|
|
|
|
|
.. note:: When you use this procedure to configure the secondary region,
|
|
replace ``us-`` with ``eu-``. You will have a total of four zones **after**
|
|
you create the master zone and the secondary zone in each region.
|
|
|
|
|
|
Restart Services
|
|
----------------
|
|
|
|
Once you have redeployed your Ceph configuration files, we recommend restarting
|
|
your Ceph Storage Cluster(s) and Apache instances.
|
|
|
|
For Ubuntu, use the following on each :term:`Ceph Node`::
|
|
|
|
sudo restart ceph-all
|
|
|
|
For Red Hat/CentOS, use the following::
|
|
|
|
sudo /etc/init.d/ceph restart
|
|
|
|
To ensure that all components have reloaded their configurations, for each
|
|
gateway instance we recommend restarting the ``apache2`` service. For example::
|
|
|
|
sudo service apache2 restart
|
|
|
|
|
|
Start Gateway Instances
|
|
-----------------------
|
|
|
|
Start up the ``radosgw`` service. ::
|
|
|
|
sudo /etc/init.d/radosgw start
|
|
|
|
If you are running multiple instances on the same host, you must specify the
|
|
user name. ::
|
|
|
|
sudo /etc/init.d/radosgw start --name client.radosgw.us-east-1
|
|
|
|
|
|
Open a browser and check the endpoints for each zone. A simple HTTP request
|
|
to the domain name should return the following:
|
|
|
|
.. code-block:: xml
|
|
|
|
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
<Owner>
|
|
<ID>anonymous</ID>
|
|
<DisplayName/>
|
|
</Owner>
|
|
<Buckets/>
|
|
</ListAllMyBucketsResult>
|
|
|
|
|
|
Configure a Secondary Region
|
|
============================
|
|
|
|
This section provides an exemplary procedure for setting up a cluster with
|
|
multiple regions. Configuring a cluster that spans regions requires maintaining
|
|
a global namespace, so that there are no namespace clashes among object names
|
|
stored across in different regions.
|
|
|
|
This section extends the procedure in `Configure a Master Region`_, but
|
|
changes the region name and modifies a few procedures. See the following
|
|
sections for details.
|
|
|
|
|
|
Naming for the Secondary Region
|
|
-------------------------------
|
|
|
|
Before configuring the cluster, defining region, zone and instance names will
|
|
help you manage your cluster. Let's assume the region represents the European
|
|
Union, and we refer to it by its standard abbreviation.
|
|
|
|
- European Union: ``eu``
|
|
|
|
Let's assume the zones represent the Eastern and Western European Union. For
|
|
continuity, our naming convention will use ``{region name}-{zone name}``
|
|
format, but you can use any naming convention you prefer.
|
|
|
|
- European Union, East Region: ``eu-east``
|
|
- European Union, West Region: ``eu-west``
|
|
|
|
Finally, let's assume that zones may have more than one Ceph Object Gateway
|
|
instance per zone. For continuity, our naming convention will use
|
|
``{region name}-{zone name}-{instance}`` format, but you can use any naming
|
|
convention you prefer.
|
|
|
|
- European Union Region, Master Zone, Instance 1: ``eu-east-1``
|
|
- European Union Region, Secondary Zone, Instance 1: ``eu-west-1``
|
|
|
|
|
|
Configuring a Secondary Region
|
|
------------------------------
|
|
|
|
Repeat the exemplary procedure of `Configure a Master Region`_
|
|
with the following differences:
|
|
|
|
#. Use `Naming for the Secondary Region`_ in lieu of `Naming for
|
|
the Master Region`_.
|
|
|
|
#. `Create Pools`_ using ``eu`` instead of ``us``.
|
|
|
|
#. `Create a Keyring`_ and the corresponding keys using ``eu`` instead of
|
|
``us``. You may use the same keyring if you desire, but ensure that you
|
|
create the keys on the Ceph Storage Cluster for that region (or region
|
|
and zone).
|
|
|
|
#. `Install Apache/FastCGI`_.
|
|
|
|
#. `Create Data Directories`_ using ``eu`` instead of ``us``.
|
|
|
|
#. `Create a Gateway Configuration`_ using ``eu`` instead of ``us`` for
|
|
the socket names.
|
|
|
|
#. `Enable the Configuration`_.
|
|
|
|
#. `Add a FastCGI Script`_ using ``eu`` instead of ``us`` for the user names.
|
|
|
|
#. `Add Instances to Ceph Config File`_ using ``eu`` instead of ``us`` for the
|
|
pool names.
|
|
|
|
#. `Create a Region`_ using ``eu`` instead of ``us``. Set ``is_master`` to
|
|
``false``. For consistency, create the master region in the secondary region
|
|
too. ::
|
|
|
|
radosgw-admin region set --infile us.json --name client.radosgw.eu-east-1
|
|
|
|
#. `Create Zones`_ using ``eu`` instead of ``us``. Ensure that you update the
|
|
user name (i.e., ``--name``) so that you create the zones in the correct
|
|
cluster.
|
|
|
|
#. `Update Zone Configurations`_ using ``eu`` instead of ``us``.
|
|
|
|
#. Create zones from master region in the secondary region. ::
|
|
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.eu-east-1
|
|
radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.eu-west-1
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.eu-east-1
|
|
radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.eu-west-1
|
|
|
|
#. Create zones from secondary region in the master region. ::
|
|
|
|
radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.us-west-1
|
|
radosgw-admin zone set --rgw-zone=eu-west --infile eu-west.json --name client.radosgw.us-east-1
|
|
radosgw-admin zone set --rgw-zone=eu-west --infile eu-west.json --name client.radosgw.us-west-1
|
|
|
|
#. `Restart Services`_.
|
|
|
|
#. `Start Gateway Instances`_.
|
|
|
|
|
|
Multi-Site Data Replication
|
|
===========================
|
|
|
|
The data synchronization agent replicates the data of a master zone to a
|
|
secondary zone. The master zone of a region is the source for the secondary zone
|
|
of the region and it gets selected automatically.
|
|
|
|
.. image:: ../images/zone-sync.png
|
|
|
|
To configure the synchronization agent, retrieve the access key and secret for
|
|
the source and destination, and the destination URL and port.
|
|
|
|
You may use ``radosgw-admin zone list`` to get a list of zone names. You
|
|
may use ``radosgw-admin zone get`` to identify the key and secret for the
|
|
zone. You may refer to the gateway configuration file you created under
|
|
`Create a Gateway Configuration`_ to identify the port number.
|
|
|
|
You only need the hostname and port for a single instance (assuming all
|
|
gateway instances in a region/zone access the same Ceph Storage Cluster).
|
|
Specify these values in a configuration file
|
|
(e.g., ``cluster-data-sync.conf``), and include a ``log_file`` name.
|
|
|
|
|
|
For example:
|
|
|
|
.. code-block:: ini
|
|
|
|
src_access_key: {source-access-key}
|
|
src_secret_key: {source-secret-key}
|
|
destination: https://zone-name.fqdn.com:port
|
|
dest_access_key: {destination-access-key}
|
|
dest_secret_key: {destination-secret-key}
|
|
log_file: {log.filename}
|
|
|
|
A concrete example may look like this:
|
|
|
|
.. code-block:: ini
|
|
|
|
src_access_key: DG8RE354EFPZBICHIAF0
|
|
src_secret_key: i3U0HiRP8CXaBWrcF8bbh6CbsxGYuPPwRkixfFSb
|
|
destination: https://us-west.storage.net:80
|
|
dest_access_key: U60RFI6B08F32T2PD30G
|
|
dest_secret_key: W3HuUor7Gl1Ee93pA2pq2wFk1JMQ7hTrSDecYExl
|
|
log_file: /var/log/radosgw/radosgw-sync-us-east-west.log
|
|
|
|
To activate the data synchronization agent, open a terminal and
|
|
execute the following::
|
|
|
|
radosgw-agent -c region-data-sync.conf
|
|
|
|
When the synchronization agent is running, you should see output
|
|
indicating that the agent is synchronizing shards of data. ::
|
|
|
|
INFO:radosgw_agent.sync:Starting incremental sync
|
|
INFO:radosgw_agent.worker:17910 is processing shard number 0
|
|
INFO:radosgw_agent.worker:shard 0 has 0 entries after ''
|
|
INFO:radosgw_agent.worker:finished processing shard 0
|
|
INFO:radosgw_agent.worker:17910 is processing shard number 1
|
|
INFO:radosgw_agent.sync:1/64 shards processed
|
|
INFO:radosgw_agent.worker:shard 1 has 0 entries after ''
|
|
INFO:radosgw_agent.worker:finished processing shard 1
|
|
INFO:radosgw_agent.sync:2/64 shards processed
|
|
...
|
|
|
|
.. note:: You must have an agent for each source-destination pair.
|
|
|
|
|
|
Inter-Region Metadata Replication
|
|
=================================
|
|
|
|
The data synchronization agent replicates the metadata of master zone in the
|
|
master region to a master zone in a secondary region. Metadata consists of
|
|
gateway users and buckets, but not the objects within the buckets--ensuring a
|
|
unified namespace across the cluster. The master zone of the master region is
|
|
the source for the master zone of the secondary region and it gets selected
|
|
automatically.
|
|
|
|
.. image:: ../images/region-sync.png
|
|
:align: center
|
|
|
|
Follow the same steps in `Multi-Site Data Replication`_ by specifying the master
|
|
zone of the master region as the source zone and the master zone of the
|
|
secondary region as the secondary zone. When activating the ``radosgw-agent``,
|
|
specify ``--metadata-only`` so that it only copies metadata. For example::
|
|
|
|
radosgw-agent -c inter-region-data-sync.conf --metadata-only
|
|
|
|
Once you have completed the foregoing procedure, you should have a cluster
|
|
consisting of a master region (``us``) and a secondary region (``eu``) where
|
|
there is a unified namespace between the two regions.
|
|
|
|
|
|
|
|
.. _CRUSH Map: ../../rados/operations/crush-map
|
|
.. _Install Ceph Object Gateway: ../../install/install-ceph-gateway
|
|
.. _Cephx Administration: ../../rados/operations/authentication/#cephx-administration
|
|
.. _Ceph configuration file: ../../rados/configuration/ceph-conf
|
|
.. _Configuration Reference - Pools: ../config-ref#pools
|
|
.. _Configuration Reference - Regions: ../config-ref#regions
|
|
.. _Configuration Reference - Zones: ../config-ref#zones
|
|
.. _Pools: ../../rados/operations/pools
|
|
.. _Simple Configuration: ../config
|