mirror of
https://github.com/ceph/ceph
synced 2025-02-17 16:07:37 +00:00
90b55cfdc2
This removes the constraint on the transport being TCP, allowing to use a Unix domain socket or other options. It also allows specifying export names which a) are needed in case of serving different content on different exports and b) some servers may require regardless. Additionally, NBD URIs are future proof as all that NBDStream needs to do is forward the string to libnbd. Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
401 lines
14 KiB
ReStructuredText
401 lines
14 KiB
ReStructuredText
======================
|
|
Image Live-Migration
|
|
======================
|
|
|
|
.. index:: Ceph Block Device; live-migration
|
|
|
|
RBD images can be live-migrated between different pools, image formats and/or
|
|
layouts within the same Ceph cluster; from an image in another Ceph cluster; or
|
|
from external data sources. When started, the source will be deep-copied to
|
|
the destination image, pulling all snapshot history while preserving the sparse
|
|
allocation of data where possible.
|
|
|
|
By default, when live-migrating RBD images within the same Ceph cluster, the
|
|
source image will be marked read-only and all clients will instead redirect
|
|
IOs to the new target image. In addition, this mode can optionally preserve the
|
|
link to the source image's parent to preserve sparseness, or it can flatten the
|
|
image during the migration to remove the dependency on the source image's
|
|
parent.
|
|
|
|
The live-migration process can also be used in an import-only mode where the
|
|
source image remains unmodified and the target image can be linked to an image
|
|
in another Ceph cluster or to an external data source such as a backing file,
|
|
HTTP(s) file, S3 object, or NBD export.
|
|
|
|
The live-migration copy process can safely run in the background while the new
|
|
target image is in use. There is currently a requirement to temporarily stop
|
|
using the source image before preparing a migration when not using the
|
|
import-only mode of operation. This helps to ensure that the client using the
|
|
image is updated to point to the new target image.
|
|
|
|
.. note::
|
|
Image live-migration requires the Ceph Nautilus release or later. Support for
|
|
external data sources requires the Ceph Pacific release of later. The
|
|
``krbd`` kernel module does not support live-migration at this time.
|
|
|
|
|
|
.. ditaa::
|
|
|
|
+-------------+ +-------------+
|
|
| {s} c999 | | {s} |
|
|
| Live | Target refers | Live |
|
|
| migration |<-------------*| migration |
|
|
| source | to Source | target |
|
|
| | | |
|
|
| (read only) | | (writable) |
|
|
+-------------+ +-------------+
|
|
|
|
Source Target
|
|
|
|
The live-migration process is comprised of three steps:
|
|
|
|
#. **Prepare Migration:** The initial step creates the new target image and
|
|
links the target image to the source. When not configured in the import-only
|
|
mode, the source image will also be linked to the target image and marked
|
|
read-only.
|
|
|
|
Similar to `layered images`_, attempts to read uninitialized data extents
|
|
within the target image will internally redirect the read to the source
|
|
image, and writes to uninitialized extents within the target will internally
|
|
deep-copy the overlapping source image block to the target image.
|
|
|
|
|
|
#. **Execute Migration:** This is a background operation that deep-copies all
|
|
initialized blocks from the source image to the target. This step can be
|
|
run while clients are actively using the new target image.
|
|
|
|
|
|
#. **Finish Migration:** Once the background migration process has completed,
|
|
the migration can be committed or aborted. Committing the migration will
|
|
remove the cross-links between the source and target images, and will
|
|
remove the source image if not configured in the import-only mode. Aborting
|
|
the migration will remove the cross-links, and will remove the target image.
|
|
|
|
Prepare Migration
|
|
=================
|
|
|
|
The default live-migration process for images within the same Ceph cluster is
|
|
initiated by running the `rbd migration prepare` command, providing the source
|
|
and target images::
|
|
|
|
$ rbd migration prepare migration_source [migration_target]
|
|
|
|
The `rbd migration prepare` command accepts all the same layout optionals as the
|
|
`rbd create` command, which allows changes to the immutable image on-disk
|
|
layout. The `migration_target` can be skipped if the goal is only to change the
|
|
on-disk layout, keeping the original image name.
|
|
|
|
All clients using the source image must be stopped prior to preparing a
|
|
live-migration. The prepare step will fail if it finds any running clients with
|
|
the image open in read/write mode. Once the prepare step is complete, the
|
|
clients can be restarted using the new target image name. Attempting to restart
|
|
the clients using the source image name will result in failure.
|
|
|
|
The `rbd status` command will show the current state of the live-migration::
|
|
|
|
$ rbd status migration_target
|
|
Watchers: none
|
|
Migration:
|
|
source: rbd/migration_source (5e2cba2f62e)
|
|
destination: rbd/migration_target (5e2ed95ed806)
|
|
state: prepared
|
|
|
|
Note that the source image will be moved to the RBD trash to avoid mistaken
|
|
usage during the migration process::
|
|
|
|
$ rbd info migration_source
|
|
rbd: error opening image migration_source: (2) No such file or directory
|
|
$ rbd trash ls --all
|
|
5e2cba2f62e migration_source
|
|
|
|
|
|
Prepare Import-Only Migration
|
|
=============================
|
|
|
|
The import-only live-migration process is initiated by running the same
|
|
`rbd migration prepare` command, but adding the `--import-only` optional
|
|
and providing a JSON-encoded ``source-spec`` to describe how to access
|
|
the source image data. This ``source-spec`` can either be passed
|
|
directly via the `--source-spec` optional, or via a file or STDIN via the
|
|
`--source-spec-path` optional::
|
|
|
|
$ rbd migration prepare --import-only --source-spec "<JSON>" migration_target
|
|
|
|
The `rbd migration prepare` command accepts all the same layout optionals as the
|
|
`rbd create` command.
|
|
|
|
The `rbd status` command will show the current state of the live-migration::
|
|
|
|
$ rbd status migration_target
|
|
Watchers: none
|
|
Migration:
|
|
source: {"stream":{"file_path":"/mnt/image.raw","type":"file"},"type":"raw"}
|
|
destination: rbd/migration_target (ac69113dc1d7)
|
|
state: prepared
|
|
|
|
The general format for the ``source-spec`` JSON is as follows::
|
|
|
|
{
|
|
"type": "<format-type>",
|
|
<format unique parameters>
|
|
"stream": {
|
|
"type": "<stream-type>",
|
|
<stream unique parameters>
|
|
}
|
|
}
|
|
|
|
The following formats are currently supported: ``native``, ``qcow``, and
|
|
``raw``. The following streams are currently supported: ``file``, ``http``,
|
|
``s3``, and ``nbd``.
|
|
|
|
Formats
|
|
~~~~~~~
|
|
|
|
The ``native`` format can be used to describe a native RBD image within a
|
|
Ceph cluster as the source image. Its ``source-spec`` JSON is encoded
|
|
as follows::
|
|
|
|
{
|
|
"type": "native",
|
|
["cluster_name": "<cluster-name>",] (specify if image in another cluster,
|
|
requires ``<cluster-name>.conf`` file)
|
|
["client_name": "<client-name>",] (for connecting to another cluster,
|
|
default is ``client.admin``)
|
|
"pool_name": "<pool-name>",
|
|
["pool_id": <pool-id>,] (optional alternative to "pool_name")
|
|
["pool_namespace": "<pool-namespace",] (optional)
|
|
"image_name": "<image-name>",
|
|
["image_id": "<image-id>",] (specify if image in trash)
|
|
"snap_name": "<snap-name>",
|
|
["snap_id": "<snap-id>",] (optional alternative to "snap_name")
|
|
}
|
|
|
|
Note that the ``native`` format does not include the ``stream`` object since
|
|
it utilizes native Ceph operations. For example, to import from the image
|
|
``rbd/ns1/image1@snap1``, the ``source-spec`` could be encoded as::
|
|
|
|
{
|
|
"type": "native",
|
|
"pool_name": "rbd",
|
|
"pool_namespace": "ns1",
|
|
"image_name": "image1",
|
|
"snap_name": "snap1"
|
|
}
|
|
|
|
The ``qcow`` format can be used to describe a QCOW (QEMU copy-on-write) block
|
|
device. Both the QCOW (v1) and QCOW2 formats are currently supported with the
|
|
exception of advanced features such as compression, encryption, backing
|
|
files, and external data files. Support for these missing features may be added
|
|
in a future release. The ``qcow`` format data can be linked to any supported
|
|
stream source described below. For example, its base ``source-spec`` JSON is
|
|
encoded as follows::
|
|
|
|
{
|
|
"type": "qcow",
|
|
"stream": {
|
|
<stream unique parameters>
|
|
}
|
|
}
|
|
|
|
The ``raw`` format can be used to describe a thick-provisioned, raw block device
|
|
export (i.e. `rbd export --export-format 1 <snap-spec>`). The ``raw`` format
|
|
data can be linked to any supported stream source described below. For example,
|
|
its base ``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
"type": "raw",
|
|
"stream": {
|
|
<stream unique parameters for HEAD, non-snapshot revision>
|
|
},
|
|
"snapshots": [
|
|
{
|
|
"type": "raw",
|
|
"name": "<snapshot-name>",
|
|
"stream": {
|
|
<stream unique parameters for snapshot>
|
|
}
|
|
},
|
|
] (optional oldest to newest ordering of snapshots)
|
|
}
|
|
|
|
The inclusion of the ``snapshots`` array is optional and currently only supports
|
|
thick-provisioned ``raw`` snapshot exports.
|
|
|
|
Additional formats such as RBD export-format v2 and RBD export-diff
|
|
snapshots will be added in a future release.
|
|
|
|
Streams
|
|
~~~~~~~
|
|
|
|
The ``file`` stream can be used to import from a locally accessible POSIX file
|
|
source. Its ``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
<format unique parameters>
|
|
"stream": {
|
|
"type": "file",
|
|
"file_path": "<file-path>"
|
|
}
|
|
}
|
|
|
|
For example, to import a raw-format image from a file located at
|
|
"/mnt/image.raw", its ``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
"type": "raw",
|
|
"stream": {
|
|
"type": "file",
|
|
"file_path": "/mnt/image.raw"
|
|
}
|
|
}
|
|
|
|
The ``http`` stream can be used to import from a remote HTTP or HTTPS web
|
|
server. Its ``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
<format unique parameters>
|
|
"stream": {
|
|
"type": "http",
|
|
"url": "<url-path>"
|
|
}
|
|
}
|
|
|
|
For example, to import a raw-format image from a file located at
|
|
``http://download.ceph.com/image.raw``, its ``source-spec`` JSON is encoded
|
|
as follows::
|
|
|
|
{
|
|
"type": "raw",
|
|
"stream": {
|
|
"type": "http",
|
|
"url": "http://download.ceph.com/image.raw"
|
|
}
|
|
}
|
|
|
|
The ``s3`` stream can be used to import from a remote S3 bucket. Its
|
|
``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
<format unique parameters>
|
|
"stream": {
|
|
"type": "s3",
|
|
"url": "<url-path>",
|
|
"access_key": "<access-key>",
|
|
"secret_key": "<secret-key>"
|
|
}
|
|
}
|
|
|
|
For example, to import a raw-format image from a file located at
|
|
`http://s3.ceph.com/bucket/image.raw`, its ``source-spec`` JSON is encoded
|
|
as follows::
|
|
|
|
{
|
|
"type": "raw",
|
|
"stream": {
|
|
"type": "s3",
|
|
"url": "http://s3.ceph.com/bucket/image.raw",
|
|
"access_key": "NX5QOQKC6BH2IDN8HC7A",
|
|
"secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn"
|
|
}
|
|
}
|
|
|
|
.. note::
|
|
The ``access_key`` and ``secret_key`` parameters support storing the keys in
|
|
the MON config-key store by prefixing the key values with ``config://``
|
|
followed by the path in the MON config-key store to the value. Values can be
|
|
stored in the config-key store via ``ceph config-key set <key-path> <value>``
|
|
(e.g. ``ceph config-key set rbd/s3/access_key NX5QOQKC6BH2IDN8HC7A``).
|
|
|
|
The ``nbd`` stream can be used to import from a remote NBD export. Its
|
|
``source-spec`` JSON is encoded as follows::
|
|
|
|
{
|
|
<format unique parameters>
|
|
"stream": {
|
|
"type": "nbd",
|
|
"uri": "<nbd-uri>",
|
|
}
|
|
}
|
|
|
|
For example, to import a raw-format image from an NBD export located at
|
|
``nbd://nbd.ceph.com`` with export name ``image.raw``, its ``source-spec``
|
|
JSON is encoded as follows::
|
|
|
|
{
|
|
"type": "raw",
|
|
"stream": {
|
|
"type": "nbd",
|
|
"uri": "nbd://nbd.ceph.com/image.raw",
|
|
}
|
|
}
|
|
|
|
``nbd-uri`` parameter should follow the `NBD URI specification`_. The
|
|
default NBD port is ``10809``.
|
|
|
|
|
|
Execute Migration
|
|
=================
|
|
|
|
After preparing the live-migration, the image blocks from the source image
|
|
must be copied to the target image. This is accomplished by running the
|
|
`rbd migration execute` command::
|
|
|
|
$ rbd migration execute migration_target
|
|
Image migration: 100% complete...done.
|
|
|
|
The `rbd status` command will also provide feedback on the progress of the
|
|
migration block deep-copy process::
|
|
|
|
$ rbd status migration_target
|
|
Watchers:
|
|
watcher=1.2.3.4:0/3695551461 client.123 cookie=123
|
|
Migration:
|
|
source: rbd/migration_source (5e2cba2f62e)
|
|
destination: rbd/migration_target (5e2ed95ed806)
|
|
state: executing (32% complete)
|
|
|
|
|
|
Commit Migration
|
|
================
|
|
|
|
Once the live-migration has completed deep-copying all data blocks from the
|
|
source image to the target, the migration can be committed::
|
|
|
|
$ rbd status migration_target
|
|
Watchers: none
|
|
Migration:
|
|
source: rbd/migration_source (5e2cba2f62e)
|
|
destination: rbd/migration_target (5e2ed95ed806)
|
|
state: executed
|
|
$ rbd migration commit migration_target
|
|
Commit image migration: 100% complete...done.
|
|
|
|
If the `migration_source` image is a parent of one or more clones, the `--force`
|
|
option will need to be specified after ensuring all descendent clone images are
|
|
not in use.
|
|
|
|
Committing the live-migration will remove the cross-links between the source
|
|
and target images, and will remove the source image::
|
|
|
|
$ rbd trash list --all
|
|
|
|
|
|
Abort Migration
|
|
===============
|
|
|
|
If you wish to revert the prepare or execute step, run the `rbd migration abort`
|
|
command to revert the migration process::
|
|
|
|
$ rbd migration abort migration_target
|
|
Abort image migration: 100% complete...done.
|
|
|
|
Aborting the migration will result in the target image being deleted and access
|
|
to the original source image being restored::
|
|
|
|
$ rbd ls
|
|
migration_source
|
|
|
|
|
|
.. _layered images: ../rbd-snapshot/#layering
|
|
.. _NBD URI specification: https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
|