mirror of
https://github.com/ceph/ceph
synced 2024-12-19 01:46:00 +00:00
522c184f42
Fixes: https://tracker.ceph.com/issues/52942 Signed-off-by: Sage Weil <sage@newdream.net>
577 lines
19 KiB
ReStructuredText
577 lines
19 KiB
ReStructuredText
.. _mgr-nfs:
|
|
|
|
=============================
|
|
CephFS & RGW Exports over NFS
|
|
=============================
|
|
|
|
CephFS namespaces and RGW buckets can be exported over NFS protocol
|
|
using the `NFS-Ganesha NFS server`_.
|
|
|
|
The ``nfs`` manager module provides a general interface for managing
|
|
NFS exports of either CephFS directories or RGW buckets. Exports can
|
|
be managed either via the CLI ``ceph nfs export ...`` commands
|
|
or via the dashboard.
|
|
|
|
The deployment of the nfs-ganesha daemons can also be managed
|
|
automatically if either the :ref:`cephadm` or :ref:`mgr-rook`
|
|
orchestrators are enabled. If neither are in use (e.g., Ceph is
|
|
deployed via an external orchestrator like Ansible or Puppet), the
|
|
nfs-ganesha daemons must be manually deployed; for more information,
|
|
see :ref:`nfs-ganesha-config`.
|
|
|
|
.. note:: Starting with Ceph Pacific, the ``nfs`` mgr module must be enabled.
|
|
|
|
NFS Cluster management
|
|
======================
|
|
|
|
Create NFS Ganesha Cluster
|
|
--------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster create <cluster_id> [<placement>] [--port <port>] [--ingress --virtual-ip <ip>]
|
|
|
|
This creates a common recovery pool for all NFS Ganesha daemons, new user based on
|
|
``cluster_id``, and a common NFS Ganesha config RADOS object.
|
|
|
|
.. note:: Since this command also brings up NFS Ganesha daemons using a ceph-mgr
|
|
orchestrator module (see :doc:`/mgr/orchestrator`) such as cephadm or rook, at
|
|
least one such module must be enabled for it to work.
|
|
|
|
Currently, NFS Ganesha daemon deployed by cephadm listens on the standard
|
|
port. So only one daemon will be deployed on a host.
|
|
|
|
``<cluster_id>`` is an arbitrary string by which this NFS Ganesha cluster will be
|
|
known (e.g., ``mynfs``).
|
|
|
|
``<placement>`` is an optional string signifying which hosts should have NFS Ganesha
|
|
daemon containers running on them and, optionally, the total number of NFS
|
|
Ganesha daemons on the cluster (should you want to have more than one NFS Ganesha
|
|
daemon running per node). For example, the following placement string means
|
|
"deploy NFS Ganesha daemons on nodes host1 and host2 (one daemon per host)::
|
|
|
|
"host1,host2"
|
|
|
|
and this placement specification says to deploy single NFS Ganesha daemon each
|
|
on nodes host1 and host2 (for a total of two NFS Ganesha daemons in the
|
|
cluster)::
|
|
|
|
"2 host1,host2"
|
|
|
|
NFS can be deployed on a port other than 2049 (the default) with ``--port <port>``.
|
|
|
|
To deploy NFS with a high-availability front-end (virtual IP and load balancer), add the
|
|
``--ingress`` flag and specify a virtual IP address. This will deploy a combination
|
|
of keepalived and haproxy to provide an high-availability NFS frontend for the NFS
|
|
service.
|
|
|
|
.. note:: The ingress implementation is not yet complete. Enabling
|
|
ingress will deploy multiple ganesha instances and balance
|
|
load across them, but a host failure will not immediately
|
|
cause cephadm to deploy a replacement daemon before the NFS
|
|
grace period expires. This high-availability functionality
|
|
is expected to be completed by the Quincy release (March
|
|
2022).
|
|
|
|
For more details, refer :ref:`orchestrator-cli-placement-spec` but keep
|
|
in mind that specifying the placement via a YAML file is not supported.
|
|
|
|
Ingress
|
|
-------
|
|
|
|
The core *nfs* service will deploy one or more nfs-ganesha daemons,
|
|
each of which will provide a working NFS endpoint. The IP for each
|
|
NFS endpoint will depend on which host the nfs-ganesha daemons are
|
|
deployed. By default, daemons are placed semi-randomly, but users can
|
|
also explicitly control where daemons are placed; see
|
|
:ref:`orchestrator-cli-placement-spec`.
|
|
|
|
When a cluster is created with ``--ingress``, an *ingress* service is
|
|
additionally deployed to provide load balancing and high-availability
|
|
for the NFS servers. A virtual IP is used to provide a known, stable
|
|
NFS endpoint that all clients can use to mount. Ceph will take care
|
|
of the details of NFS redirecting traffic on the virtual IP to the
|
|
appropriate backend NFS servers, and redeploying NFS servers when they
|
|
fail.
|
|
|
|
Enabling ingress via the ``ceph nfs cluster create`` command deploys a
|
|
simple ingress configuration with the most common configuration
|
|
options. Ingress can also be added to an existing NFS service (e.g.,
|
|
one created without the ``--ingress`` flag), and the basic NFS service can
|
|
also be modified after the fact to include non-default options, by modifying
|
|
the services directly. For more information, see :ref:`cephadm-ha-nfs`.
|
|
|
|
Show NFS Cluster IP(s)
|
|
----------------------
|
|
|
|
To examine an NFS cluster's IP endpoints, including the IPs for the individual NFS
|
|
daemons, and the virtual IP (if any) for the ingress service,
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster info [<cluster_id>]
|
|
|
|
.. note:: This will not work with the rook backend. Instead, expose the port with
|
|
the kubectl patch command and fetch the port details with kubectl get services
|
|
command::
|
|
|
|
$ kubectl patch service -n rook-ceph -p '{"spec":{"type": "NodePort"}}' rook-ceph-nfs-<cluster-name>-<node-id>
|
|
$ kubectl get services -n rook-ceph rook-ceph-nfs-<cluster-name>-<node-id>
|
|
|
|
|
|
Delete NFS Ganesha Cluster
|
|
--------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster rm <cluster_id>
|
|
|
|
This deletes the deployed cluster.
|
|
|
|
Updating an NFS Cluster
|
|
-----------------------
|
|
|
|
In order to modify cluster parameters (like the port or placement), you need to
|
|
use the orchestrator interface to update the NFS service spec. The safest way to do
|
|
that is to export the current spec, modify it, and then re-apply it. For example,
|
|
to modify the ``nfs.foo`` service,
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph orch ls --service-name nfs.foo --export > nfs.foo.yaml
|
|
$ vi nfs.foo.yaml
|
|
$ ceph orch apply -i nfs.foo.yaml
|
|
|
|
For more information about the NFS service spec, see :ref:`deploy-cephadm-nfs-ganesha`.
|
|
|
|
List NFS Ganesha Clusters
|
|
-------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster ls
|
|
|
|
This lists deployed clusters.
|
|
|
|
.. _nfs-cluster-set:
|
|
|
|
Set Customized NFS Ganesha Configuration
|
|
----------------------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster config set <cluster_id> -i <config_file>
|
|
|
|
With this the nfs cluster will use the specified config and it will have
|
|
precedence over default config blocks.
|
|
|
|
Example use cases include:
|
|
|
|
#. Changing log level. The logging level can be adjusted with the following config
|
|
fragment::
|
|
|
|
LOG {
|
|
COMPONENTS {
|
|
ALL = FULL_DEBUG;
|
|
}
|
|
}
|
|
|
|
#. Adding custom export block.
|
|
|
|
The following sample block creates a single export. This export will not be
|
|
managed by `ceph nfs export` interface::
|
|
|
|
EXPORT {
|
|
Export_Id = 100;
|
|
Transports = TCP;
|
|
Path = /;
|
|
Pseudo = /ceph/;
|
|
Protocols = 4;
|
|
Access_Type = RW;
|
|
Attr_Expiration_Time = 0;
|
|
Squash = None;
|
|
FSAL {
|
|
Name = CEPH;
|
|
Filesystem = "filesystem name";
|
|
User_Id = "user id";
|
|
Secret_Access_Key = "secret key";
|
|
}
|
|
}
|
|
|
|
.. note:: User specified in FSAL block should have proper caps for NFS-Ganesha
|
|
daemons to access ceph cluster. User can be created in following way using
|
|
`auth get-or-create`::
|
|
|
|
# ceph auth get-or-create client.<user_id> mon 'allow r' osd 'allow rw pool=.nfs namespace=<nfs_cluster_name>, allow rw tag cephfs data=<fs_name>' mds 'allow rw path=<export_path>'
|
|
|
|
View Customized NFS Ganesha Configuration
|
|
-----------------------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster config get <cluster_id>
|
|
|
|
This will output the user defined configuration (if any).
|
|
|
|
Reset NFS Ganesha Configuration
|
|
-------------------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs cluster config reset <cluster_id>
|
|
|
|
This removes the user defined configuration.
|
|
|
|
.. note:: With a rook deployment, ganesha pods must be explicitly restarted
|
|
for the new config blocks to be effective.
|
|
|
|
|
|
Export Management
|
|
=================
|
|
|
|
.. warning:: Currently, the nfs interface is not integrated with dashboard. Both
|
|
dashboard and nfs interface have different export requirements and
|
|
create exports differently. Management of dashboard created exports is not
|
|
supported.
|
|
|
|
Create CephFS Export
|
|
--------------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs export create cephfs <fsname> <cluster_id> <pseudo_path> [--readonly] [--path=/path/in/cephfs] [--client_addr <value>...] [--squash <value>]
|
|
|
|
This creates export RADOS objects containing the export block, where
|
|
|
|
``<fsname>`` is the name of the FS volume used by the NFS Ganesha cluster
|
|
that will serve this export.
|
|
|
|
``<cluster_id>`` is the NFS Ganesha cluster ID.
|
|
|
|
``<pseudo_path>`` is the export position within the NFS v4 Pseudo Filesystem where the export will be available on the server. It must be an absolute path and unique.
|
|
|
|
``<path>`` is the path within cephfs. Valid path should be given and default
|
|
path is '/'. It need not be unique. Subvolume path can be fetched using:
|
|
|
|
.. code::
|
|
|
|
$ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]
|
|
|
|
``<client_addr>`` is the list of client address for which these export
|
|
permissions will be applicable. By default all clients can access the export
|
|
according to specified export permissions. See the `NFS-Ganesha Export Sample`_
|
|
for permissible values.
|
|
|
|
``<squash>`` defines the kind of user id squashing to be performed. The default
|
|
value is `no_root_squash`. See the `NFS-Ganesha Export Sample`_ for
|
|
permissible values.
|
|
|
|
.. note:: Export creation is supported only for NFS Ganesha clusters deployed using nfs interface.
|
|
|
|
Create RGW Export
|
|
-----------------
|
|
|
|
To export a bucket
|
|
|
|
.. code::
|
|
|
|
$ ceph nfs export create rgw <bucket_name> <cluster_id> <pseudo_path> [--readonly] [--client_addr <value>...] [--squash <value>]
|
|
|
|
For example, to export *mybucket* via NFS cluster *mynfs* at the pseudo-path */bucketdata* to any host in the ``192.168.10.0/24`` network
|
|
|
|
.. code::
|
|
|
|
$ ceph nfs export create rgw mybucket mynfs /bucketdata --client_addr 192.168.10.0/24
|
|
|
|
.. note:: Export creation is supported only for NFS Ganesha clusters deployed using nfs interface.
|
|
|
|
``<bucket_name>`` is the name of the bucket that will be exported.
|
|
|
|
.. note:: Currently, if multi-site RGW is enabled, Ceph can only export RGW buckets in the default realm.
|
|
|
|
``<cluster_id>`` is the NFS Ganesha cluster ID.
|
|
|
|
``<pseudo_path>`` is the export position within the NFS v4 Pseudo Filesystem where the export will be available on the server. It must be an absolute path and unique.
|
|
|
|
``<client_addr>`` is the list of client address for which these export
|
|
permissions will be applicable. By default all clients can access the export
|
|
according to specified export permissions. See the `NFS-Ganesha Export Sample`_
|
|
for permissible values.
|
|
|
|
``<squash>`` defines the kind of user id squashing to be performed. The default
|
|
value is `no_root_squash`. See the `NFS-Ganesha Export Sample`_ for
|
|
permissible values.
|
|
|
|
Delete Export
|
|
-------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs export rm <cluster_id> <pseudo_path>
|
|
|
|
This deletes an export in an NFS Ganesha cluster, where:
|
|
|
|
``<cluster_id>`` is the NFS Ganesha cluster ID.
|
|
|
|
``<pseudo_path>`` is the pseudo root path (must be an absolute path).
|
|
|
|
List Exports
|
|
------------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs export ls <cluster_id> [--detailed]
|
|
|
|
It lists exports for a cluster, where:
|
|
|
|
``<cluster_id>`` is the NFS Ganesha cluster ID.
|
|
|
|
With the ``--detailed`` option enabled it shows entire export block.
|
|
|
|
Get Export
|
|
----------
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph nfs export info <cluster_id> <pseudo_path>
|
|
|
|
This displays export block for a cluster based on pseudo root name,
|
|
where:
|
|
|
|
``<cluster_id>`` is the NFS Ganesha cluster ID.
|
|
|
|
``<pseudo_path>`` is the pseudo root path (must be an absolute path).
|
|
|
|
|
|
Create or update export via JSON specification
|
|
----------------------------------------------
|
|
|
|
An existing export can be dumped in JSON format with:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph nfs export info *<cluster_id>* *<pseudo_path>*
|
|
|
|
An export can be created or modified by importing a JSON description in the
|
|
same format:
|
|
|
|
.. prompt:: bash #
|
|
|
|
ceph nfs export apply *<cluster_id>* -i <json_file>
|
|
|
|
For example,::
|
|
|
|
$ ceph nfs export info mynfs /cephfs > update_cephfs_export.json
|
|
$ cat update_cephfs_export.json
|
|
{
|
|
"export_id": 1,
|
|
"path": "/",
|
|
"cluster_id": "mynfs",
|
|
"pseudo": "/cephfs",
|
|
"access_type": "RW",
|
|
"squash": "no_root_squash",
|
|
"security_label": true,
|
|
"protocols": [
|
|
4
|
|
],
|
|
"transports": [
|
|
"TCP"
|
|
],
|
|
"fsal": {
|
|
"name": "CEPH",
|
|
"user_id": "nfs.mynfs.1",
|
|
"fs_name": "a",
|
|
"sec_label_xattr": ""
|
|
},
|
|
"clients": []
|
|
}
|
|
|
|
The imported JSON can be a single dict describing a single export, or a JSON list
|
|
containing multiple export dicts.
|
|
|
|
The exported JSON can be modified and then reapplied. Below, *pseudo*
|
|
and *access_type* are modified. When modifying an export, the
|
|
provided JSON should fully describe the new state of the export (just
|
|
as when creating a new export), with the exception of the
|
|
authentication credentials, which will be carried over from the
|
|
previous state of the export where possible.
|
|
|
|
::
|
|
|
|
$ ceph nfs export apply mynfs -i update_cephfs_export.json
|
|
$ cat update_cephfs_export.json
|
|
{
|
|
"export_id": 1,
|
|
"path": "/",
|
|
"cluster_id": "mynfs",
|
|
"pseudo": "/cephfs_testing",
|
|
"access_type": "RO",
|
|
"squash": "no_root_squash",
|
|
"security_label": true,
|
|
"protocols": [
|
|
4
|
|
],
|
|
"transports": [
|
|
"TCP"
|
|
],
|
|
"fsal": {
|
|
"name": "CEPH",
|
|
"user_id": "nfs.mynfs.1",
|
|
"fs_name": "a",
|
|
"sec_label_xattr": ""
|
|
},
|
|
"clients": []
|
|
}
|
|
|
|
An export can also be created or updated by injecting a Ganesha NFS EXPORT config
|
|
fragment. For example,::
|
|
|
|
$ ceph nfs export apply mynfs -i update_cephfs_export.conf
|
|
$ cat update_cephfs_export.conf
|
|
EXPORT {
|
|
FSAL {
|
|
name = "CEPH";
|
|
filesystem = "a";
|
|
}
|
|
export_id = 1;
|
|
path = "/";
|
|
pseudo = "/a";
|
|
access_type = "RW";
|
|
squash = "none";
|
|
attr_expiration_time = 0;
|
|
security_label = true;
|
|
protocols = 4;
|
|
transports = "TCP";
|
|
}
|
|
|
|
|
|
Mounting
|
|
========
|
|
|
|
After the exports are successfully created and NFS Ganesha daemons are
|
|
deployed, exports can be mounted with:
|
|
|
|
.. code:: bash
|
|
|
|
$ mount -t nfs <ganesha-host-name>:<pseudo_path> <mount-point>
|
|
|
|
For example, if the NFS cluster was created with ``--ingress --virtual-ip 192.168.10.10``
|
|
and the export's pseudo-path was ``/foo``, the export can be mounted at ``/mnt`` with:
|
|
|
|
.. code:: bash
|
|
|
|
$ mount -t nfs 192.168.10.10:/foo /mnt
|
|
|
|
If the NFS service is running on a non-standard port number:
|
|
|
|
.. code:: bash
|
|
|
|
$ mount -t nfs -o port=<ganesha-port> <ganesha-host-name>:<ganesha-pseudo_path> <mount-point>
|
|
|
|
.. note:: Only NFS v4.0+ is supported.
|
|
|
|
Troubleshooting
|
|
===============
|
|
|
|
Checking NFS-Ganesha logs with
|
|
|
|
1) ``cephadm``: The NFS daemons can be listed with:
|
|
|
|
.. code:: bash
|
|
|
|
$ ceph orch ps --daemon-type nfs
|
|
|
|
You can via the logs for a specific daemon (e.g., ``nfs.mynfs.0.0.myhost.xkfzal``) on
|
|
the relevant host with:
|
|
|
|
.. code:: bash
|
|
|
|
# cephadm logs --fsid <fsid> --name nfs.mynfs.0.0.myhost.xkfzal
|
|
|
|
2) ``rook``:
|
|
|
|
.. code:: bash
|
|
|
|
$ kubectl logs -n rook-ceph rook-ceph-nfs-<cluster_id>-<node_id> nfs-ganesha
|
|
|
|
The NFS log level can be adjusted using `nfs cluster config set` command (see :ref:`nfs-cluster-set`).
|
|
|
|
|
|
.. _nfs-ganesha-config:
|
|
|
|
|
|
Manual Ganesha deployment
|
|
=========================
|
|
|
|
It may be possible to deploy and manage the NFS ganesha daemons manually
|
|
instead of allowing cephadm or rook to do so.
|
|
|
|
.. note:: Manual configuration is not tested or fully documented; your
|
|
mileage may vary. If you make this work, please help us by
|
|
updating this documentation.
|
|
|
|
Known issues
|
|
------------
|
|
|
|
* The ``mgr/nfs`` module enumerates NFS clusters via the orchestrator API; if NFS is
|
|
not managed by the orchestrator (e.g., cephadm or rook) then this will not work. It
|
|
may be possible to create the cluster, mark the cephadm service as 'unmanaged', but this
|
|
is awkward and not ideal.
|
|
|
|
Requirements
|
|
------------
|
|
|
|
The following packages are required to enable CephFS and RGW exports with nfs-ganesha:
|
|
|
|
- ``nfs-ganesha``, ``nfs-ganesha-ceph``, ``nfs-ganesha-rados-grace`` and
|
|
``nfs-ganesha-rados-urls`` packages (version 3.3 and above)
|
|
|
|
Ganesha Configuration Hierarchy
|
|
-------------------------------
|
|
|
|
Cephadm and rook start each nfs-ganesha daemon with a minimal
|
|
`bootstrap` configuration file that pulls from a shared `common`
|
|
configuration stored in the ``.nfs`` RADOS pool and watches the common
|
|
config for changes. Each export is written to a separate RADOS object
|
|
that is referenced by URL from the common config.
|
|
|
|
.. ditaa::
|
|
|
|
rados://$pool/$namespace/export-$i rados://$pool/$namespace/userconf-nfs.$cluster_id
|
|
(export config) (user config)
|
|
|
|
+----------+ +----------+ +----------+ +---------------------------+
|
|
| | | | | | | |
|
|
| export-1 | | export-2 | | export-3 | | userconf-nfs.$cluster_id |
|
|
| | | | | | | |
|
|
+----+-----+ +----+-----+ +-----+----+ +-------------+-------------+
|
|
^ ^ ^ ^
|
|
| | | |
|
|
+--------------------------------+-------------------------+
|
|
%url |
|
|
|
|
|
+--------+--------+
|
|
| | rados://$pool/$namespace/conf-nfs.$svc
|
|
| conf+nfs.$svc | (common config)
|
|
| |
|
|
+--------+--------+
|
|
^
|
|
|
|
|
watch_url |
|
|
+----------------------------------------------+
|
|
| | |
|
|
| | | RADOS
|
|
+----------------------------------------------------------------------------------+
|
|
| | | CONTAINER
|
|
watch_url | watch_url | watch_url |
|
|
| | |
|
|
+--------+-------+ +--------+-------+ +-------+--------+
|
|
| | | | | | /etc/ganesha/ganesha.conf
|
|
| nfs.$svc.a | | nfs.$svc.b | | nfs.$svc.c | (bootstrap config)
|
|
| | | | | |
|
|
+----------------+ +----------------+ +----------------+
|
|
|
|
|
|
.. _NFS-Ganesha NFS Server: https://github.com/nfs-ganesha/nfs-ganesha/wiki
|
|
.. _NFS-Ganesha Export Sample: https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/config_samples/export.txt
|