ceph/doc/start/kube-helm.rst

================================
Installation (Kubernetes + Helm)
================================

The ceph-helm_ project enables you to deploy Ceph in a Kubernetes environement.
This documentation assumes a Kubernetes environement is available.

Current limitations
===================

 - The public and cluster networks must be the same
 - If the storage class user id is not admin, you will have to manually create the user
   in your Ceph cluster and create its secret in Kubernetes
 - ceph-mgr can only run with 1 replica

Install and start helm
======================

Helm can be installed by following these instructions_.

Once installed, run::

    $ helm init
    $ helm serve &
    $ helm repo add local http://localhost:8879/charts

Add ceph-helm to Helm local repos
==================================
::

    $ git clone https://github.com/ceph/ceph-helm
    $ cd ceph-helm/ceph
    $ make

Configure your Ceph cluster
===========================

Create a ``~/ceph-overrides.yaml`` that will contain your Ceph configuration::

    $ cat ~/ceph-overrides.yaml
    network:
      public:   172.21.0.0/20
      cluster:   172.21.0.0/20

    osd_devices:
      - name: dev-sdd
        device: /dev/sdd
        zap: "1"
      - name: dev-sde
        device: /dev/sde
        zap: "1"

    storageclass:
      name: ceph-rbd
      pool: rbd
      user_id: k8s

.. note:: If journal is not set it will be colocated with device

.. note:: The ``ceph-helm/ceph/ceph/values.yaml`` file contains the full
          list of option that can be set

Configure RBAC permissions
==========================

Kubernetes >=v1.6 makes RBAC the default admission controller. ceph-helm provides RBAC roles and permissions for each component::

    $ kubectl create -f ~/ceph-helm/ceph/rbac.yaml

The ``rbac.yaml`` file assumes that the Ceph cluster will be deployed in the ``ceph`` namespace.

Label kubelets
==============

The following labels need to be set to deploy a Ceph cluster:
 - ceph-mon=enabled
 - ceph-mgr=enabled
 - ceph-osd=enabled
 - ceph-osd-device-<name>=enabled

The ``ceph-osd-device-<name>`` label is created based on the osd_devices name value defined in our ``ceph-overrides.yaml``.
From our example above we will have the two following label: ``ceph-osd-device-dev-sdb`` and ``ceph-osd-device-dev-sdc``.

For each Ceph Monitor::

    $ kubectl label node <nodename> ceph-mon=enabled ceph-mgr=enabled

For each OSD node::

    $ kubectl label node <nodename> ceph-osd=enabled ceph-osd-device-dev-sdb=enabled ceph-osd-device-dev-sdc=enabled

Ceph Deployment
===============

Run the helm install command to deploy Ceph::

    $ helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml
    NAME:   ceph
    LAST DEPLOYED: Wed Oct 18 22:25:06 2017
    NAMESPACE: ceph
    STATUS: DEPLOYED

    RESOURCES:
    ==> v1/Secret
    NAME                    TYPE    DATA  AGE
    ceph-keystone-user-rgw  Opaque  7     1s

    ==> v1/ConfigMap
    NAME              DATA  AGE
    ceph-bin-clients  2     1s
    ceph-bin          24    1s
    ceph-etc          1     1s
    ceph-templates    5     1s

    ==> v1/Service
    NAME      CLUSTER-IP      EXTERNAL-IP  PORT(S)   AGE
    ceph-mon  None            <none>       6789/TCP  1s
    ceph-rgw  10.101.219.239  <none>       8088/TCP  1s

    ==> v1beta1/DaemonSet
    NAME              DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE-SELECTOR                                     AGE
    ceph-mon          3        3        0      3           0          ceph-mon=enabled                                  1s
    ceph-osd-dev-sde  3        3        0      3           0          ceph-osd-device-dev-sde=enabled,ceph-osd=enabled  1s
    ceph-osd-dev-sdd  3        3        0      3           0          ceph-osd-device-dev-sdd=enabled,ceph-osd=enabled  1s

    ==> v1beta1/Deployment
    NAME                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
    ceph-mds              1        1        1           0          1s
    ceph-mgr              1        1        1           0          1s
    ceph-mon-check        1        1        1           0          1s
    ceph-rbd-provisioner  2        2        2           0          1s
    ceph-rgw              1        1        1           0          1s

    ==> v1/Job
    NAME                                 DESIRED  SUCCESSFUL  AGE
    ceph-mgr-keyring-generator           1        0           1s
    ceph-mds-keyring-generator           1        0           1s
    ceph-osd-keyring-generator           1        0           1s
    ceph-rgw-keyring-generator           1        0           1s
    ceph-mon-keyring-generator           1        0           1s
    ceph-namespace-client-key-generator  1        0           1s
    ceph-storage-keys-generator          1        0           1s

    ==> v1/StorageClass
    NAME     TYPE
    ceph-rbd  ceph.com/rbd

The output from helm install shows us the different types of ressources that will be deployed.

A StorageClass named ``ceph-rbd`` of type ``ceph.com/rbd`` will be created with ``ceph-rbd-provisioner`` Pods. These
will allow a RBD to be automatically provisioned upon creation of a PVC. RBDs will also be formatted when mapped for the first
time. All RBDs will use the ext4 filesystem. ``ceph.com/rbd`` does not support the ``fsType`` option.
By default, RBDs will use image format 2 and layering. You can overwrite the following storageclass' defaults in your values file::

  storageclass:
    name: ceph-rbd
    pool: rbd
    user_id: k8s
    user_secret_name: pvc-ceph-client-key
    image_format: "2"
    image_features: layering

Check that all Pods are running with the command below. This might take a few minutes::

    $ kubectl -n ceph get pods
    NAME                                    READY     STATUS    RESTARTS   AGE
    ceph-mds-3804776627-976z9               0/1       Pending   0          1m
    ceph-mgr-3367933990-b368c               1/1       Running   0          1m
    ceph-mon-check-1818208419-0vkb7         1/1       Running   0          1m
    ceph-mon-cppdk                          3/3       Running   0          1m
    ceph-mon-t4stn                          3/3       Running   0          1m
    ceph-mon-vqzl0                          3/3       Running   0          1m
    ceph-osd-dev-sdd-6dphp                  1/1       Running   0          1m
    ceph-osd-dev-sdd-6w7ng                  1/1       Running   0          1m
    ceph-osd-dev-sdd-l80vv                  1/1       Running   0          1m
    ceph-osd-dev-sde-6dq6w                  1/1       Running   0          1m
    ceph-osd-dev-sde-kqt0r                  1/1       Running   0          1m
    ceph-osd-dev-sde-lp2pf                  1/1       Running   0          1m
    ceph-rbd-provisioner-2099367036-4prvt   1/1       Running   0          1m
    ceph-rbd-provisioner-2099367036-h9kw7   1/1       Running   0          1m
    ceph-rgw-3375847861-4wr74               0/1       Pending   0          1m

.. note:: The MDS and RGW Pods are pending since we did not label any nodes with
          ``ceph-rgw=enabled`` or ``ceph-mds=enabled``

Once all Pods are running, check the status of the Ceph cluster from one Mon::

    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph -s
    cluster:
      id:     e8f9da03-c2d2-4ad3-b807-2a13d0775504
      health: HEALTH_OK

    services:
      mon: 3 daemons, quorum mira115,mira110,mira109
      mgr: mira109(active)
      osd: 6 osds: 6 up, 6 in

    data:
      pools:   0 pools, 0 pgs
      objects: 0 objects, 0 bytes
      usage:   644 MB used, 5555 GB / 5556 GB avail
      pgs:

Configure a Pod to use a PersistentVolume from Ceph
===================================================

Create a keyring for the k8s user defined in the ``~/ceph-overwrite.yaml`` and convert
it to base64::

    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- bash
    # ceph auth get-or-create-key client.k8s mon 'allow r' osd 'allow rwx pool=rbd'  | base64
    QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo=
    # exit

Edit the user secret present in the ``ceph`` namespace::

    $ kubectl -n ceph edit secrets/pvc-ceph-client-key

Add the base64 value to the key value with your own and save::

    apiVersion: v1
    data:
      key: QVFCLzdPaFoxeUxCRVJBQUVEVGdHcE9YU3BYMVBSdURHUEU0T0E9PQo=
    kind: Secret
    metadata:
      creationTimestamp: 2017-10-19T17:34:04Z
      name: pvc-ceph-client-key
      namespace: ceph
      resourceVersion: "8665522"
      selfLink: /api/v1/namespaces/ceph/secrets/pvc-ceph-client-key
      uid: b4085944-b4f3-11e7-add7-002590347682
    type: kubernetes.io/rbd

We are going to create a Pod that consumes a RBD in the default namespace.
Copy the user secret from the ``ceph`` namespace to ``default``::

    $ kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "default"' | kubectl create -f -
    secret "pvc-ceph-client-key" created
    $ kubectl get secrets
    NAME                  TYPE                                  DATA      AGE
    default-token-r43wl   kubernetes.io/service-account-token   3         61d
    pvc-ceph-client-key   kubernetes.io/rbd                     1         20s

Create and initialize the RBD pool::

    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- ceph osd pool create rbd 256
    pool 'rbd' created
    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd pool init rbd

.. important:: Kubernetes uses the RBD kernel module to map RBDs to hosts. Luminous requires
               CRUSH_TUNABLES 5 (Jewel). The minimal kernel version for these tunables is 4.5.
               If your kernel does not support these tunables, run ``ceph osd crush tunables hammer``


.. important:: Since RBDs are mapped on the host system. Hosts need to be able to resolve
                the ceph-mon.ceph.svc.cluster.local name managed by the kube-dns service. To get the
                IP address of the kube-dns service, run ``kubectl -n kube-system get svc/kube-dns``

Create a PVC::

    $ cat pvc-rbd.yaml
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: ceph-pvc
    spec:
      accessModes:
       - ReadWriteOnce
      resources:
        requests:
           storage: 20Gi
      storageClassName: ceph-rbd

    $ kubectl create -f pvc-rbd.yaml
    persistentvolumeclaim "ceph-pvc" created
    $ kubectl get pvc
    NAME       STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
    ceph-pvc   Bound     pvc-1c2ada50-b456-11e7-add7-002590347682   20Gi       RWO           ceph-rbd        3s

You can check that the RBD has been created on your cluster::

    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd ls
    kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915
    $ kubectl -n ceph exec -ti ceph-mon-cppdk -c ceph-mon -- rbd info kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915
    rbd image 'kubernetes-dynamic-pvc-1c2e9442-b456-11e7-9bd2-2a4159ce3915':
        size 20480 MB in 5120 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10762ae8944a
        format: 2
        features: layering
        flags:
        create_timestamp: Wed Oct 18 22:45:59 2017

Create a Pod that will use the PVC::

    $ cat pod-with-rbd.yaml
    kind: Pod
    apiVersion: v1
    metadata:
      name: mypod
    spec:
      containers:
        - name: busybox
          image: busybox
          command:
            - sleep
            - "3600"
          volumeMounts:
          - mountPath: "/mnt/rbd"
            name: vol1
      volumes:
        - name: vol1
          persistentVolumeClaim:
            claimName: ceph-pvc

    $ kubectl create -f pod-with-rbd.yaml
    pod "mypod" created

Check the Pod::

    $ kubectl get pods
    NAME      READY     STATUS    RESTARTS   AGE
    mypod     1/1       Running   0          17s
    $ kubectl exec mypod -- mount | grep rbd
    /dev/rbd0 on /mnt/rbd type ext4 (rw,relatime,stripe=1024,data=ordered)

Logging
=======

OSDs and Monitor logs can be accessed via the ``kubectl logs [-f]`` command. Monitors have multiple stream of logging,
each stream is accessible from a container running in the ceph-mon Pod.

There are 3 containers running in the ceph-mon Pod:
 - ceph-mon, equivalent of ceph-mon.hostname.log on baremetal
 - cluster-audit-log-tailer, equivalent of ceph.audit.log on baremetal
 - cluster-log-tailer, equivalent of ceph.log on baremetal or ``ceph -w``

Each container is accessible via the ``--container`` or ``-c`` option.
For instance, to access the cluster-tail-log, one can run::

    $ kubectl -n ceph logs ceph-mon-cppdk -c cluster-log-tailer

.. _ceph-helm: https://github.com/ceph/ceph-helm/
.. _instructions: https://github.com/kubernetes/helm/blob/master/docs/install.md