mirror of https://github.com/ceph/ceph
365 lines
13 KiB
ReStructuredText
365 lines
13 KiB
ReStructuredText
==============================
|
|
Block Devices and Kubernetes
|
|
==============================
|
|
|
|
You may use Ceph Block Device images with Kubernetes v1.13 and later through
|
|
`ceph-csi`_, which dynamically provisions RBD images to back Kubernetes
|
|
`volumes`_ and maps these RBD images as block devices (optionally mounting
|
|
a file system contained within the image) on worker nodes running
|
|
`pods`_ that reference an RBD-backed volume. Ceph stripes block device images as
|
|
objects across the cluster, which means that large Ceph Block Device images have
|
|
better performance than a standalone server!
|
|
|
|
To use Ceph Block Devices with Kubernetes v1.13 and higher, you must install
|
|
and configure ``ceph-csi`` within your Kubernetes environment. The following
|
|
diagram depicts the Kubernetes/Ceph technology stack.
|
|
|
|
.. ditaa::
|
|
+---------------------------------------------------+
|
|
| Kubernetes |
|
|
+---------------------------------------------------+
|
|
| ceph--csi |
|
|
+------------------------+--------------------------+
|
|
|
|
|
| configures
|
|
v
|
|
+------------------------+ +------------------------+
|
|
| | | rbd--nbd |
|
|
| Kernel Modules | +------------------------+
|
|
| | | librbd |
|
|
+------------------------+-+------------------------+
|
|
| RADOS Protocol |
|
|
+------------------------+-+------------------------+
|
|
| OSDs | | Monitors |
|
|
+------------------------+ +------------------------+
|
|
|
|
|
|
.. important::
|
|
``ceph-csi`` uses the RBD kernel modules by default which may not support all
|
|
Ceph `CRUSH tunables`_ or `RBD image features`_.
|
|
|
|
Create a Pool
|
|
=============
|
|
|
|
By default, Ceph block devices use the ``rbd`` pool. Create a pool for
|
|
Kubernetes volume storage. Ensure your Ceph cluster is running, then create
|
|
the pool. ::
|
|
|
|
$ ceph osd pool create kubernetes
|
|
|
|
See `Create a Pool`_ for details on specifying the number of placement groups
|
|
for your pools, and `Placement Groups`_ for details on the number of placement
|
|
groups you should set for your pools.
|
|
|
|
A newly created pool must be initialized prior to use. Use the ``rbd`` tool
|
|
to initialize the pool::
|
|
|
|
$ rbd pool init kubernetes
|
|
|
|
Configure ceph-csi
|
|
==================
|
|
|
|
Setup Ceph Client Authentication
|
|
--------------------------------
|
|
|
|
Create a new user for Kubernetes and `ceph-csi`. Execute the following and
|
|
record the generated key::
|
|
|
|
$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
|
|
[client.kubernetes]
|
|
key = AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
|
|
|
|
Generate `ceph-csi` `ConfigMap`
|
|
-------------------------------
|
|
|
|
The `ceph-csi` requires a `ConfigMap` object stored in Kubernetes to define the
|
|
the Ceph monitor addresses for the Ceph cluster. Collect both the Ceph cluster
|
|
unique `fsid` and the monitor addresses::
|
|
|
|
$ ceph mon dump
|
|
<...>
|
|
fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
|
|
<...>
|
|
0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
|
|
1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
|
|
2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
|
|
|
|
.. note::
|
|
``ceph-csi`` currently only supports the `legacy V1 protocol`_.
|
|
|
|
Generate a `csi-config-map.yaml` file similar to the example below, substituting
|
|
the `fsid` for "clusterID", and the monitor addresses for "monitors"::
|
|
|
|
$ cat <<EOF > csi-config-map.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
data:
|
|
config.json: |-
|
|
[
|
|
{
|
|
"clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
|
|
"monitors": [
|
|
"192.168.1.1:6789",
|
|
"192.168.1.2:6789",
|
|
"192.168.1.3:6789"
|
|
]
|
|
}
|
|
]
|
|
metadata:
|
|
name: ceph-csi-config
|
|
EOF
|
|
|
|
Once generated, store the new `ConfigMap` object in Kubernetes::
|
|
|
|
$ kubectl apply -f csi-config-map.yaml
|
|
|
|
Recent versions of `ceph-csi` also require an additional `ConfigMap` object to
|
|
define Key Management Service (KMS) provider details. If KMS isn't set up, put
|
|
an empty configuration in a `csi-kms-config-map.yaml` file or refer to examples
|
|
at https://github.com/ceph/ceph-csi/tree/master/examples/kms::
|
|
|
|
$ cat <<EOF > csi-kms-config-map.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
data:
|
|
config.json: |-
|
|
{}
|
|
metadata:
|
|
name: ceph-csi-encryption-kms-config
|
|
EOF
|
|
|
|
Once generated, store the new `ConfigMap` object in Kubernetes::
|
|
|
|
$ kubectl apply -f csi-kms-config-map.yaml
|
|
|
|
Recent versions of `ceph-csi` also require yet another `ConfigMap` object
|
|
to define Ceph configuration to add to ceph.conf file inside CSI containers::
|
|
|
|
$ cat <<EOF > ceph-config-map.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
data:
|
|
ceph.conf: |
|
|
[global]
|
|
auth_cluster_required = cephx
|
|
auth_service_required = cephx
|
|
auth_client_required = cephx
|
|
# keyring is a required key and its value should be empty
|
|
keyring: |
|
|
metadata:
|
|
name: ceph-config
|
|
EOF
|
|
|
|
Once generated, store the new `ConfigMap` object in Kubernetes::
|
|
|
|
$ kubectl apply -f ceph-config-map.yaml
|
|
|
|
Generate `ceph-csi` cephx `Secret`
|
|
----------------------------------
|
|
|
|
`ceph-csi` requires the cephx credentials for communicating with the Ceph
|
|
cluster. Generate a `csi-rbd-secret.yaml` file similar to the example below,
|
|
using the newly created Kubernetes user id and cephx key::
|
|
|
|
$ cat <<EOF > csi-rbd-secret.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: csi-rbd-secret
|
|
namespace: default
|
|
stringData:
|
|
userID: kubernetes
|
|
userKey: AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
|
|
EOF
|
|
|
|
Once generated, store the new `Secret` object in Kubernetes::
|
|
|
|
$ kubectl apply -f csi-rbd-secret.yaml
|
|
|
|
Configure `ceph-csi` Plugins
|
|
----------------------------
|
|
|
|
Create the required `ServiceAccount` and RBAC `ClusterRole`/`ClusterRoleBinding`
|
|
Kubernetes objects. These objects do not necessarily need to be customized for
|
|
your Kubernetes environment and therefore can be used as-is from the `ceph-csi`
|
|
deployment YAMLs::
|
|
|
|
$ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
|
|
$ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
|
|
|
|
Finally, create the `ceph-csi` provisioner and node plugins. With the
|
|
possible exception of the `ceph-csi` container release version, these objects do
|
|
not necessarily need to be customized for your Kubernetes environment and
|
|
therefore can be used as-is from the `ceph-csi` deployment YAMLs::
|
|
|
|
$ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
|
|
$ kubectl apply -f csi-rbdplugin-provisioner.yaml
|
|
$ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
|
|
$ kubectl apply -f csi-rbdplugin.yaml
|
|
|
|
.. important::
|
|
The provisioner and node plugin YAMLs will, by default, pull the development
|
|
release of the `ceph-csi` container (quay.io/cephcsi/cephcsi:canary).
|
|
The YAMLs should be updated to use a release version container for
|
|
production workloads.
|
|
|
|
Using Ceph Block Devices
|
|
========================
|
|
|
|
Create a `StorageClass`
|
|
-----------------------
|
|
|
|
The Kubernetes `StorageClass` defines a class of storage. Multiple `StorageClass`
|
|
objects can be created to map to different quality-of-service levels (i.e. NVMe
|
|
vs HDD-based pools) and features.
|
|
|
|
For example, to create a `ceph-csi` `StorageClass` that maps to the `kubernetes`
|
|
pool created above, the following YAML file can be used after ensuring that the
|
|
"clusterID" property matches your Ceph cluster's `fsid`::
|
|
|
|
$ cat <<EOF > csi-rbd-sc.yaml
|
|
---
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: csi-rbd-sc
|
|
provisioner: rbd.csi.ceph.com
|
|
parameters:
|
|
clusterID: b9127830-b0cc-4e34-aa47-9d1a2e9949a8
|
|
pool: kubernetes
|
|
imageFeatures: layering
|
|
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
|
|
csi.storage.k8s.io/provisioner-secret-namespace: default
|
|
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
|
|
csi.storage.k8s.io/controller-expand-secret-namespace: default
|
|
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
|
|
csi.storage.k8s.io/node-stage-secret-namespace: default
|
|
reclaimPolicy: Delete
|
|
allowVolumeExpansion: true
|
|
mountOptions:
|
|
- discard
|
|
EOF
|
|
$ kubectl apply -f csi-rbd-sc.yaml
|
|
|
|
Note that in Kubernetes v1.14 and v1.15 volume expansion feature was in alpha
|
|
status and required enabling `ExpandCSIVolumes` feature gate.
|
|
|
|
Create a `PersistentVolumeClaim`
|
|
--------------------------------
|
|
|
|
A `PersistentVolumeClaim` is a request for abstract storage resources by a user.
|
|
The `PersistentVolumeClaim` would then be associated to a `Pod` resource to
|
|
provision a `PersistentVolume`, which would be backed by a Ceph block image.
|
|
An optional `volumeMode` can be included to select between a mounted file system
|
|
(default) or raw block device-based volume.
|
|
|
|
Using `ceph-csi`, specifying `Filesystem` for `volumeMode` can support both
|
|
`ReadWriteOnce` and `ReadOnlyMany` `accessMode` claims, and specifying `Block`
|
|
for `volumeMode` can support `ReadWriteOnce`, `ReadWriteMany`, and
|
|
`ReadOnlyMany` `accessMode` claims.
|
|
|
|
For example, to create a block-based `PersistentVolumeClaim` that utilizes
|
|
the `ceph-csi`-based `StorageClass` created above, the following YAML can be
|
|
used to request raw block storage from the `csi-rbd-sc` `StorageClass`::
|
|
|
|
$ cat <<EOF > raw-block-pvc.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: raw-block-pvc
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
volumeMode: Block
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
storageClassName: csi-rbd-sc
|
|
EOF
|
|
$ kubectl apply -f raw-block-pvc.yaml
|
|
|
|
The following demonstrates and example of binding the above
|
|
`PersistentVolumeClaim` to a `Pod` resource as a raw block device::
|
|
|
|
$ cat <<EOF > raw-block-pod.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: pod-with-raw-block-volume
|
|
spec:
|
|
containers:
|
|
- name: fc-container
|
|
image: fedora:26
|
|
command: ["/bin/sh", "-c"]
|
|
args: ["tail -f /dev/null"]
|
|
volumeDevices:
|
|
- name: data
|
|
devicePath: /dev/xvda
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: raw-block-pvc
|
|
EOF
|
|
$ kubectl apply -f raw-block-pod.yaml
|
|
|
|
To create a file-system-based `PersistentVolumeClaim` that utilizes the
|
|
`ceph-csi`-based `StorageClass` created above, the following YAML can be used to
|
|
request a mounted file system (backed by an RBD image) from the `csi-rbd-sc`
|
|
`StorageClass`::
|
|
|
|
$ cat <<EOF > pvc.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: rbd-pvc
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
volumeMode: Filesystem
|
|
resources:
|
|
requests:
|
|
storage: 1Gi
|
|
storageClassName: csi-rbd-sc
|
|
EOF
|
|
$ kubectl apply -f pvc.yaml
|
|
|
|
The following demonstrates and example of binding the above
|
|
`PersistentVolumeClaim` to a `Pod` resource as a mounted file system::
|
|
|
|
$ cat <<EOF > pod.yaml
|
|
---
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: csi-rbd-demo-pod
|
|
spec:
|
|
containers:
|
|
- name: web-server
|
|
image: nginx
|
|
volumeMounts:
|
|
- name: mypvc
|
|
mountPath: /var/lib/www/html
|
|
volumes:
|
|
- name: mypvc
|
|
persistentVolumeClaim:
|
|
claimName: rbd-pvc
|
|
readOnly: false
|
|
EOF
|
|
$ kubectl apply -f pod.yaml
|
|
|
|
.. _ceph-csi: https://github.com/ceph/ceph-csi/
|
|
.. _volumes: https://kubernetes.io/docs/concepts/storage/volumes/
|
|
.. _pods: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/
|
|
.. _Create a Pool: ../../rados/operations/pools#createpool
|
|
.. _Placement Groups: ../../rados/operations/placement-groups
|
|
.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
|
|
.. _RBD image features: ../rbd-config-ref/#image-features
|
|
.. _legacy V1 protocol: ../../rados/configuration/msgr2/#address-formats
|