mirror of
https://github.com/ceph/ceph
synced 2024-12-27 22:13:28 +00:00
479 lines
15 KiB
ReStructuredText
479 lines
15 KiB
ReStructuredText
=========================
|
|
Block Devices and Nomad
|
|
=========================
|
|
|
|
Like Kubernetes, Nomad can use Ceph Block Device. This is made possible by
|
|
`ceph-csi`_, which allows you to dynamically provision RBD images or import
|
|
existing RBD images.
|
|
|
|
Every version of Nomad is compatible with `ceph-csi`_, but the reference
|
|
version of Nomad that was used to generate the procedures and guidance in this
|
|
document is Nomad v1.1.2, the latest version available at the time of the
|
|
writing of the document.
|
|
|
|
To use Ceph Block Devices with Nomad, you must install
|
|
and configure ``ceph-csi`` within your Nomad environment. The following
|
|
diagram shows the Nomad/Ceph technology stack.
|
|
|
|
.. ditaa::
|
|
+-------------------------+-------------------------+
|
|
| Container | ceph--csi |
|
|
| | node |
|
|
| ^ | ^ |
|
|
| | | | |
|
|
+----------+--------------+-------------------------+
|
|
| | | |
|
|
| v | |
|
|
| Nomad | |
|
|
| | |
|
|
+---------------------------------------------------+
|
|
| ceph--csi |
|
|
| controller |
|
|
+--------+------------------------------------------+
|
|
| |
|
|
| configures maps |
|
|
+---------------+ +----------------+
|
|
| |
|
|
v v
|
|
+------------------------+ +------------------------+
|
|
| | | rbd--nbd |
|
|
| Kernel Modules | +------------------------+
|
|
| | | librbd |
|
|
+------------------------+-+------------------------+
|
|
| RADOS Protocol |
|
|
+------------------------+-+------------------------+
|
|
| OSDs | | Monitors |
|
|
+------------------------+ +------------------------+
|
|
|
|
.. note::
|
|
Nomad has many possible task drivers, but this example uses only a Docker container.
|
|
|
|
.. important::
|
|
``ceph-csi`` uses the RBD kernel modules by default, which may not support
|
|
all Ceph `CRUSH tunables`_ or `RBD image features`_.
|
|
|
|
Create a Pool
|
|
=============
|
|
|
|
By default, Ceph block devices use the ``rbd`` pool. Ensure that your Ceph
|
|
cluster is running, then create a pool for Nomad persistent storage:
|
|
|
|
.. prompt:: bash $
|
|
|
|
ceph osd pool create nomad
|
|
|
|
See `Create a Pool`_ for details on specifying the number of placement groups
|
|
for your pools. See `Placement Groups`_ for details on the number of placement
|
|
groups you should set for your pools.
|
|
|
|
A newly created pool must be initialized prior to use. Use the ``rbd`` tool
|
|
to initialize the pool:
|
|
|
|
.. prompt:: bash $
|
|
|
|
rbd pool init nomad
|
|
|
|
Configure ceph-csi
|
|
==================
|
|
|
|
Ceph Client Authentication Setup
|
|
--------------------------------
|
|
|
|
Create a new user for Nomad and `ceph-csi`. Execute the following command and
|
|
record the generated key:
|
|
|
|
.. code-block:: console
|
|
|
|
$ ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'
|
|
[client.nomad]
|
|
key = AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ==
|
|
|
|
|
|
Configure Nomad
|
|
---------------
|
|
|
|
Configuring Nomad to Allow Containers to Use Privileged Mode
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
By default, Nomad doesn't allow containers to use privileged mode. We must
|
|
configure Nomad so that it allows containers to use privileged mode. Edit the
|
|
Nomad configuration file by adding the following configuration block to
|
|
`/etc/nomad.d/nomad.hcl`::
|
|
|
|
plugin "docker" {
|
|
config {
|
|
allow_privileged = true
|
|
}
|
|
}
|
|
|
|
Loading the rbd module
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Nomad must have the `rbd` module loaded. Run the following command to confirm that the `rbd` module is loaded:
|
|
|
|
.. code-block:: console
|
|
|
|
$ lsmod | grep rbd
|
|
rbd 94208 2
|
|
libceph 364544 1 rbd
|
|
|
|
If the `rbd` module is not loaded, load it:
|
|
|
|
.. prompt:: bash $
|
|
|
|
sudo modprobe rbd
|
|
|
|
Restarting Nomad
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
Restart Nomad:
|
|
|
|
.. prompt:: bash $
|
|
|
|
sudo systemctl restart nomad
|
|
|
|
|
|
Create ceph-csi controller and plugin nodes
|
|
===========================================
|
|
|
|
The `ceph-csi`_ plugin requires two components:
|
|
|
|
- **Controller plugin**: communicates with the provider's API.
|
|
- **Node plugin**: executes tasks on the client.
|
|
|
|
.. note::
|
|
We'll set the ceph-csi's version in those files. See `ceph-csi release`_
|
|
for information about ceph-csi's compatibility with other versions.
|
|
|
|
Configure controller plugin
|
|
---------------------------
|
|
|
|
The controller plugin requires the Ceph monitor addresses of the Ceph
|
|
cluster. Collect both (1) the Ceph cluster unique `fsid` and (2) the monitor
|
|
addresses:
|
|
|
|
.. code-block:: console
|
|
|
|
$ ceph mon dump
|
|
<...>
|
|
fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
|
|
<...>
|
|
0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
|
|
1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
|
|
2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
|
|
|
|
Generate a ``ceph-csi-plugin-controller.nomad`` file similar to the example
|
|
below. Substitute the `fsid` for "clusterID", and the monitor addresses for
|
|
"monitors"::
|
|
|
|
|
|
job "ceph-csi-plugin-controller" {
|
|
datacenters = ["dc1"]
|
|
group "controller" {
|
|
network {
|
|
port "metrics" {}
|
|
}
|
|
task "ceph-controller" {
|
|
template {
|
|
data = <<EOF
|
|
[{
|
|
"clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
|
|
"monitors": [
|
|
"192.168.1.1",
|
|
"192.168.1.2",
|
|
"192.168.1.3"
|
|
]
|
|
}]
|
|
EOF
|
|
destination = "local/config.json"
|
|
change_mode = "restart"
|
|
}
|
|
driver = "docker"
|
|
config {
|
|
image = "quay.io/cephcsi/cephcsi:v3.3.1"
|
|
volumes = [
|
|
"./local/config.json:/etc/ceph-csi-config/config.json"
|
|
]
|
|
mounts = [
|
|
{
|
|
type = "tmpfs"
|
|
target = "/tmp/csi/keys"
|
|
readonly = false
|
|
tmpfs_options = {
|
|
size = 1000000 # size in bytes
|
|
}
|
|
}
|
|
]
|
|
args = [
|
|
"--type=rbd",
|
|
"--controllerserver=true",
|
|
"--drivername=rbd.csi.ceph.com",
|
|
"--endpoint=unix://csi/csi.sock",
|
|
"--nodeid=${node.unique.name}",
|
|
"--instanceid=${node.unique.name}-controller",
|
|
"--pidlimit=-1",
|
|
"--logtostderr=true",
|
|
"--v=5",
|
|
"--metricsport=$${NOMAD_PORT_metrics}"
|
|
]
|
|
}
|
|
resources {
|
|
cpu = 500
|
|
memory = 256
|
|
}
|
|
service {
|
|
name = "ceph-csi-controller"
|
|
port = "metrics"
|
|
tags = [ "prometheus" ]
|
|
}
|
|
csi_plugin {
|
|
id = "ceph-csi"
|
|
type = "controller"
|
|
mount_dir = "/csi"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
Configure plugin node
|
|
---------------------
|
|
|
|
Generate a ``ceph-csi-plugin-nodes.nomad`` file similar to the example below.
|
|
Substitute the `fsid` for "clusterID" and the monitor addresses for
|
|
"monitors"::
|
|
|
|
|
|
job "ceph-csi-plugin-nodes" {
|
|
datacenters = ["dc1"]
|
|
type = "system"
|
|
group "nodes" {
|
|
network {
|
|
port "metrics" {}
|
|
}
|
|
|
|
task "ceph-node" {
|
|
driver = "docker"
|
|
template {
|
|
data = <<EOF
|
|
[{
|
|
"clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
|
|
"monitors": [
|
|
"192.168.1.1",
|
|
"192.168.1.2",
|
|
"192.168.1.3"
|
|
]
|
|
}]
|
|
EOF
|
|
destination = "local/config.json"
|
|
change_mode = "restart"
|
|
}
|
|
config {
|
|
image = "quay.io/cephcsi/cephcsi:v3.3.1"
|
|
volumes = [
|
|
"./local/config.json:/etc/ceph-csi-config/config.json"
|
|
]
|
|
mounts = [
|
|
{
|
|
type = "tmpfs"
|
|
target = "/tmp/csi/keys"
|
|
readonly = false
|
|
tmpfs_options = {
|
|
size = 1000000 # size in bytes
|
|
}
|
|
}
|
|
]
|
|
args = [
|
|
"--type=rbd",
|
|
"--drivername=rbd.csi.ceph.com",
|
|
"--nodeserver=true",
|
|
"--endpoint=unix://csi/csi.sock",
|
|
"--nodeid=${node.unique.name}",
|
|
"--instanceid=${node.unique.name}-nodes",
|
|
"--pidlimit=-1",
|
|
"--logtostderr=true",
|
|
"--v=5",
|
|
"--metricsport=$${NOMAD_PORT_metrics}"
|
|
]
|
|
privileged = true
|
|
}
|
|
resources {
|
|
cpu = 500
|
|
memory = 256
|
|
}
|
|
service {
|
|
name = "ceph-csi-nodes"
|
|
port = "metrics"
|
|
tags = [ "prometheus" ]
|
|
}
|
|
csi_plugin {
|
|
id = "ceph-csi"
|
|
type = "node"
|
|
mount_dir = "/csi"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
Start plugin controller and node
|
|
--------------------------------
|
|
|
|
To start the plugin controller and the Nomad node, run the following commands:
|
|
|
|
.. prompt:: bash $
|
|
|
|
nomad job run ceph-csi-plugin-controller.nomad
|
|
nomad job run ceph-csi-plugin-nodes.nomad
|
|
|
|
The `ceph-csi`_ image will be downloaded.
|
|
|
|
Check the plugin status after a few minutes:
|
|
|
|
.. code-block:: console
|
|
|
|
$ nomad plugin status ceph-csi
|
|
ID = ceph-csi
|
|
Provider = rbd.csi.ceph.com
|
|
Version = 3.3.1
|
|
Controllers Healthy = 1
|
|
Controllers Expected = 1
|
|
Nodes Healthy = 1
|
|
Nodes Expected = 1
|
|
|
|
Allocations
|
|
ID Node ID Task Group Version Desired Status Created Modified
|
|
23b4db0c a61ef171 nodes 4 run running 3h26m ago 3h25m ago
|
|
fee74115 a61ef171 controller 6 run running 3h26m ago 3h25m ago
|
|
|
|
Using Ceph Block Devices
|
|
========================
|
|
|
|
Create rbd image
|
|
----------------
|
|
|
|
``ceph-csi`` requires the cephx credentials for communicating with the Ceph
|
|
cluster. Generate a ``ceph-volume.hcl`` file similar to the example below,
|
|
using the newly created nomad user id and cephx key::
|
|
|
|
id = "ceph-mysql"
|
|
name = "ceph-mysql"
|
|
type = "csi"
|
|
plugin_id = "ceph-csi"
|
|
capacity_max = "200G"
|
|
capacity_min = "100G"
|
|
|
|
capability {
|
|
access_mode = "single-node-writer"
|
|
attachment_mode = "file-system"
|
|
}
|
|
|
|
secrets {
|
|
userID = "admin"
|
|
userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
|
|
}
|
|
|
|
parameters {
|
|
clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
|
|
pool = "nomad"
|
|
imageFeatures = "layering"
|
|
}
|
|
|
|
After the ``ceph-volume.hcl`` file has been generated, create the volume:
|
|
|
|
.. prompt:: bash $
|
|
|
|
nomad volume create ceph-volume.hcl
|
|
|
|
Use rbd image with a container
|
|
------------------------------
|
|
|
|
As an exercise in using an rbd image with a container, modify the Hashicorp
|
|
`nomad stateful`_ example.
|
|
|
|
Generate a ``mysql.nomad`` file similar to the example below::
|
|
|
|
job "mysql-server" {
|
|
datacenters = ["dc1"]
|
|
type = "service"
|
|
group "mysql-server" {
|
|
count = 1
|
|
volume "ceph-mysql" {
|
|
type = "csi"
|
|
attachment_mode = "file-system"
|
|
access_mode = "single-node-writer"
|
|
read_only = false
|
|
source = "ceph-mysql"
|
|
}
|
|
network {
|
|
port "db" {
|
|
static = 3306
|
|
}
|
|
}
|
|
restart {
|
|
attempts = 10
|
|
interval = "5m"
|
|
delay = "25s"
|
|
mode = "delay"
|
|
}
|
|
task "mysql-server" {
|
|
driver = "docker"
|
|
volume_mount {
|
|
volume = "ceph-mysql"
|
|
destination = "/srv"
|
|
read_only = false
|
|
}
|
|
env {
|
|
MYSQL_ROOT_PASSWORD = "password"
|
|
}
|
|
config {
|
|
image = "hashicorp/mysql-portworx-demo:latest"
|
|
args = ["--datadir", "/srv/mysql"]
|
|
ports = ["db"]
|
|
}
|
|
resources {
|
|
cpu = 500
|
|
memory = 1024
|
|
}
|
|
service {
|
|
name = "mysql-server"
|
|
port = "db"
|
|
check {
|
|
type = "tcp"
|
|
interval = "10s"
|
|
timeout = "2s"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
Start the job:
|
|
|
|
.. prompt:: bash $
|
|
|
|
nomad job run mysql.nomad
|
|
|
|
Check the status of the job:
|
|
|
|
.. code-block:: console
|
|
|
|
$ nomad job status mysql-server
|
|
...
|
|
Status = running
|
|
...
|
|
Allocations
|
|
ID Node ID Task Group Version Desired Status Created Modified
|
|
38070da7 9ad01c63 mysql-server 0 run running 6s ago 3s ago
|
|
|
|
To check that data are persistent, modify the database, purge the job, then
|
|
create it using the same file. The same RBD image will be used (re-used,
|
|
really).
|
|
|
|
.. _ceph-csi: https://github.com/ceph/ceph-csi/
|
|
.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
|
|
.. _Create a Pool: ../../rados/operations/pools#createpool
|
|
.. _Placement Groups: ../../rados/operations/placement-groups
|
|
.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
|
|
.. _RBD image features: ../rbd-config-ref/#image-features
|
|
.. _nomad stateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
|
|
.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility
|