=========== RGW Service =========== .. _cephadm-deploy-rgw: Deploy RGWs =========== Cephadm deploys radosgw as a collection of daemons that manage a particular *realm* and *zone*. (For more information about realms and zones, see :ref:`multisite`.) Note that with cephadm, radosgw daemons are configured via the monitor configuration database instead of via a `ceph.conf` or the command line. If that configuration isn't already in place (usually in the ``client.rgw..`` section), then the radosgw daemons will start up with default settings (e.g., binding to port 80). To deploy a set of radosgw daemons for a particular realm and zone, run the following command: .. prompt:: bash # ceph orch apply rgw ** ** --placement="** [** ...]" For example, to deploy 2 rgw daemons serving the *myorg* realm and the *us-east-1* zone on *myhost1* and *myhost2*: .. prompt:: bash # ceph orch apply rgw myorg us-east-1 --placement="2 myhost1 myhost2" Cephadm will wait for a healthy cluster and automatically create the supplied realm and zone if they do not exist before deploying the rgw daemon(s) Alternatively, the realm, zonegroup, and zone can be manually created using ``radosgw-admin`` commands: .. prompt:: bash # radosgw-admin realm create --rgw-realm= --default .. prompt:: bash # radosgw-admin zonegroup create --rgw-zonegroup= --master --default .. prompt:: bash # radosgw-admin zone create --rgw-zonegroup= --rgw-zone= --master --default .. prompt:: bash # radosgw-admin period update --rgw-realm= --commit See :ref:`orchestrator-cli-placement-spec` for details of the placement specification. .. _orchestrator-haproxy-service-spec: High availability service for RGW ================================= This service allows the user to create a high avalilability RGW service providing a mimimun set of configuration options. The orchestrator will deploy and configure automatically several HAProxy and Keepalived containers to assure the continuity of the RGW service while the Ceph cluster will have at least 1 RGW daemon running. The next image explains graphically how this service works: .. image:: ../images/HAProxy_for_RGW.svg There are N hosts where the HA RGW service is deployed. This means that we have an HAProxy and a keeplived daemon running in each of this hosts. Keepalived is used to provide a "virtual IP" binded to the hosts. All RGW clients use this "virtual IP" to connect with the RGW Service. Each keeplived daemon is checking each few seconds what is the status of the HAProxy daemon running in the same host. Also it is aware that the "master" keepalived daemon will be running without problems. If the "master" keepalived daemon or the Active HAproxy is not responding, one of the keeplived daemons running in backup mode will be elected as master, and the "virtual ip" will be moved to that node. The active HAProxy also acts like a load balancer, distributing all RGW requests between all the RGW daemons available. **Prerequisites:** * At least two RGW daemons running in the Ceph cluster * Operating system prerequisites: In order for the Keepalived service to forward network packets properly to the real servers, each router node must have IP forwarding turned on in the kernel. So it will be needed to set this system option:: net.ipv4.ip_forward = 1 Load balancing in HAProxy and Keepalived at the same time also requires the ability to bind to an IP address that are nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to an IP that is not local for failover. So it will be needed to set this system option:: net.ipv4.ip_nonlocal_bind = 1 Be sure to set properly these two options in the file ``/etc/sysctl.conf`` in order to persist this values even if the hosts are restarted. These configuration changes must be applied in all the hosts where the HAProxy for RGW service is going to be deployed. **Deploy of the high availability service for RGW** Use the command:: ceph orch apply -i **Service specification file:** It is a yaml format file with the following properties: .. code-block:: yaml service_type: ha-rgw service_id: haproxy_for_rgw placement: hosts: - host1 - host2 - host3 spec: virtual_ip_interface: # ex: eth0 virtual_ip_address: / # ex: 192.168.20.1/24 frontend_port: # ex: 8080 ha_proxy_port: # ex: 1967 ha_proxy_stats_enabled: # ex: true ha_proxy_stats_user: # ex: admin ha_proxy_stats_password: # ex: true ha_proxy_enable_prometheus_exporter: # ex: true ha_proxy_monitor_uri: # ex: /haproxy_health keepalived_user: # ex: admin keepalived_password: # ex: admin ha_proxy_frontend_ssl_certificate: ex: [ "-----BEGIN CERTIFICATE-----", "MIIDZTCCAk2gAwIBAgIUClb9dnseOsgJWAfhPQvrZw2MP2kwDQYJKoZIhvcNAQEL", .... "-----END CERTIFICATE-----", "-----BEGIN PRIVATE KEY-----", .... "sCHaZTUevxb4h6dCEk1XdPr2O2GdjV0uQ++9bKahAy357ELT3zPE8yYqw7aUCyBO", "aW5DSCo8DgfNOgycVL/rqcrc", "-----END PRIVATE KEY-----" ] ha_proxy_frontend_ssl_port: # ex: 8090 ha_proxy_ssl_dh_param: # ex: 1024 ha_proxy_ssl_ciphers: # ex: ECDH+AESGCM:!MD5 ha_proxy_ssl_options: # ex: no-sslv3 haproxy_container_image: # ex: haproxy:2.4-dev3-alpine keepalived_container_image: # ex: arcts/keepalived:1.2.2 where the properties of this service specification are: * ``service_type`` Mandatory and set to "ha-rgw" * ``service_id`` The name of the service. * ``placement hosts`` The hosts where it is desired to run the HA daemons. An HAProxy and a Keepalived containers will be deployed in these hosts. The RGW daemons can run in other different hosts or not. * ``virtual_ip_interface`` The physical network interface where the virtual ip will be binded * ``virtual_ip_address`` The virtual IP ( and network ) where the HA RGW service will be available. All your RGW clients must point to this IP in order to use the HA RGW service . * ``frontend_port`` The port used to access the HA RGW service * ``ha_proxy_port`` The port used by HAProxy containers * ``ha_proxy_stats_enabled`` If it is desired to enable the statistics URL in HAProxy daemons * ``ha_proxy_stats_user`` User needed to access the HAProxy statistics URL * ``ha_proxy_stats_password`` The password needed to access the HAProxy statistics URL * ``ha_proxy_enable_prometheus_exporter`` If it is desired to enable the Promethes exporter in HAProxy. This will allow to consume RGW Service metrics from Grafana. * ``ha_proxy_monitor_uri``: To set the API endpoint where the health of HAProxy daemon is provided * ``keepalived_user`` User needed to access keepalived daemons * ``keepalived_password``: The password needed to access keepalived daemons * ``ha_proxy_frontend_ssl_certificate``: SSl certificate. You must paste the content of your .pem file * ``ha_proxy_frontend_ssl_port``: The https port used by HAProxy containers * ``ha_proxy_ssl_dh_param``: Value used for the `tune.ssl.default-dh-param` setting in the HAProxy config file * ``ha_proxy_ssl_ciphers``: Value used for the `ssl-default-bind-ciphers` setting in HAProxy config file. * ``ha_proxy_ssl_options``: Value used for the `ssl-default-bind-options` setting in HAProxy config file. * ``haproxy_container_image``: HAProxy image location used to pull the image * ``keepalived_container_image``: Keepalived image location used to pull the image **Useful hints for the RGW Service:** * Good to have at least 3 RGW daemons * Use at least 3 hosts for the HAProxy for RGW service * In each host an HAProxy and a Keepalived daemon will be deployed. These daemons can be managed as systemd services