mirror of
https://github.com/ceph/ceph
synced 2025-02-23 19:17:37 +00:00
Merge pull request #38694 from dillaman/wip-migration-import
librbd/migration: tweaks and initial set of documentation Reviewed-by: Mykola Golub <mgolub@suse.com>
This commit is contained in:
Mykola Golub
GitHub
commit
5fbae26e9e
@ -4,20 +4,33 @@
|
||||
|
||||
.. index:: Ceph Block Device; live-migration
|
||||
|
||||
RBD images can be live-migrated between different pools within the same cluster
|
||||
or between different image formats and layouts. When started, the source image
|
||||
will be deep-copied to the destination image, pulling all snapshot history and
|
||||
optionally preserving any link to the source image's parent to preserve
|
||||
sparseness.
|
||||
RBD images can be live-migrated between different pools within the same cluster;
|
||||
between different image formats and layouts; or from external data sources.
|
||||
When started, the source will be deep-copied to the destination image, pulling
|
||||
all snapshot history while preserving the sparse allocation of data where
|
||||
possible.
|
||||
|
||||
This copy process can safely run in the background while the new target image is
|
||||
in use. There is currently a requirement to temporarily stop using the source
|
||||
image before preparing a migration. This helps to ensure that the client using
|
||||
the image is updated to point to the new target image.
|
||||
By default, when live-migrating RBD images within the same Ceph cluster, the
|
||||
source image will be marked read-only and all clients will instead redirect
|
||||
IOs to the new target image. In addition, this mode can optionally preserve the
|
||||
link to the source image's parent to preserve sparseness, or it can flatten the
|
||||
image during the migration to remove the dependency on the source image's
|
||||
parent.
|
||||
|
||||
The live-migration process can also be used in an import-only mode where the
|
||||
source image remains unmodified and the target image can be linked to an
|
||||
external data source such as a backing file, HTTP(s) file, or S3 object.
|
||||
|
||||
The live-migration copy process can safely run in the background while the new
|
||||
target image is in use. There is currently a requirement to temporarily stop
|
||||
using the source image before preparing a migration when not using the
|
||||
import-only mode of operation. This helps to ensure that the client using the
|
||||
image is updated to point to the new target image.
|
||||
|
||||
.. note::
|
||||
Image live-migration requires the Ceph Nautilus release or later. The ``krbd``
|
||||
kernel module does not support live-migration at this time.
|
||||
Image live-migration requires the Ceph Nautilus release or later. Support for
|
||||
external data sources requires the Ceph Pacific release of later. The
|
||||
``krbd`` kernel module does not support live-migration at this time.
|
||||
|
||||
|
||||
.. ditaa::
|
||||
@ -36,11 +49,14 @@ the image is updated to point to the new target image.
|
||||
The live-migration process is comprised of three steps:
|
||||
|
||||
#. **Prepare Migration:** The initial step creates the new target image and
|
||||
cross-links the source and target images. Similar to `layered images`_,
|
||||
attempts to read uninitialized extents within the target image will
|
||||
internally redirect the read to the source image, and writes to
|
||||
uninitialized extents within the target will internally deep-copy the
|
||||
overlapping source image block to the target image.
|
||||
links the target image to the source. When not configured in the import-only
|
||||
mode, the source image will also be linked to the target image and marked
|
||||
read-only.
|
||||
|
||||
Similar to `layered images`_, attempts to read uninitialized data extents
|
||||
within the target image will internally redirect the read to the source
|
||||
image, and writes to uninitialized extents within the target will internally
|
||||
deep-copy the overlapping source image block to the target image.
|
||||
|
||||
|
||||
#. **Execute Migration:** This is a background operation that deep-copies all
|
||||
@ -51,14 +67,15 @@ The live-migration process is comprised of three steps:
|
||||
#. **Finish Migration:** Once the background migration process has completed,
|
||||
the migration can be committed or aborted. Committing the migration will
|
||||
remove the cross-links between the source and target images, and will
|
||||
remove the source image. Aborting the migration will remove the cross-links,
|
||||
and will remove the target image.
|
||||
remove the source image if not configured in the import-only mode. Aborting
|
||||
the migration will remove the cross-links, and will remove the target image.
|
||||
|
||||
Prepare Migration
|
||||
=================
|
||||
|
||||
The live-migration process is initiated by running the `rbd migration prepare`
|
||||
command, providing the source and target images::
|
||||
The default live-migration process for images within the same Ceph cluster is
|
||||
initiated by running the `rbd migration prepare` command, providing the source
|
||||
and target images::
|
||||
|
||||
$ rbd migration prepare migration_source [migration_target]
|
||||
|
||||
@ -91,6 +108,183 @@ usage during the migration process::
|
||||
5e2cba2f62e migration_source
|
||||
|
||||
|
||||
Prepare Import-Only Migration
|
||||
=============================
|
||||
|
||||
The import-only live-migration process is initiated by running the same
|
||||
`rbd migration prepare` command, but adding the `--import-only` optional
|
||||
and providing a JSON-encoded ``source-spec`` to describe how to access
|
||||
the source image data. This ``source-spec`` can either be passed
|
||||
directly via the `--source-spec` optional, or via a file or STDIN via the
|
||||
`--source-spec-file` optional::
|
||||
|
||||
$ rbd migration prepare --import-only --source-spec "<JSON>" migration_target
|
||||
|
||||
The `rbd migration prepare` command accepts all the same layout optionals as the
|
||||
`rbd create` command.
|
||||
|
||||
The `rbd status` command will show the current state of the live-migration::
|
||||
|
||||
$ rbd status migration_target
|
||||
Watchers: none
|
||||
Migration:
|
||||
source: {"stream":{"file_path":"/mnt/image.raw","type":"file"},"type":"raw"}
|
||||
destination: rbd/migration_target (ac69113dc1d7)
|
||||
state: prepared
|
||||
|
||||
The general format for the ``source-spec`` JSON is as follows::
|
||||
|
||||
{
|
||||
"type": "<format-type>",
|
||||
<format unique parameters>
|
||||
"stream": {
|
||||
"type": "<stream-type>",
|
||||
<stream unique parameters>
|
||||
}
|
||||
}
|
||||
|
||||
The following formats are currently supported: ``native`` and ``raw``. The
|
||||
following streams are currently supported: ``file``, ``http``, and ``s3``.
|
||||
|
||||
Formats
|
||||
~~~~~~~
|
||||
|
||||
The ``native`` format can be used to describe a native RBD image within a
|
||||
Ceph cluster as the source image. Its ``source-spec`` JSON is encoded
|
||||
as follows::
|
||||
|
||||
{
|
||||
"type": "native",
|
||||
"pool_name": "<pool-name>",
|
||||
["pool_id": <pool-id>,] (optional alternative to "pool_name")
|
||||
["pool_namespace": "<pool-namespace",] (optional)
|
||||
"image_name": "<image-name>",
|
||||
["image_id": "<image-id>",] (optional if image in trash)
|
||||
"snap_name": "<snap-name>",
|
||||
["snap_id": "<snap-id>",] (optional alternative to "snap_name")
|
||||
}
|
||||
|
||||
Note that the ``native`` format does not include the ``stream`` object since
|
||||
it utilizes native Ceph operations. For example, to import from the image
|
||||
``rbd/ns1/image1@snap1``, the ``source-spec`` could be encoded as::
|
||||
|
||||
{
|
||||
"type": "native",
|
||||
"pool_name": "rbd",
|
||||
"pool_namespace": "ns1",
|
||||
"image_name": "image1",
|
||||
"snap_name": "snap1"
|
||||
}
|
||||
|
||||
The ``raw`` format can be used to describe a thick-provisioned, raw block device
|
||||
export (i.e. `rbd export --export-format 1 <snap-spec>`). The ``raw`` format
|
||||
data can be linked to any supported stream source described below. For example,
|
||||
its base ``source-spec`` JSON is encoded as follows::
|
||||
|
||||
{
|
||||
"type": "raw",
|
||||
"stream": {
|
||||
<stream unique parameters for HEAD, non-snapshot revision>
|
||||
},
|
||||
"snapshots": [
|
||||
{
|
||||
"type": "raw",
|
||||
"name": "<snapshot-name>",
|
||||
"stream": {
|
||||
<stream unique parameters for snapshot>
|
||||
}
|
||||
},
|
||||
] (optional oldest to newest ordering of snapshots)
|
||||
}
|
||||
|
||||
The inclusion of the ``snapshots`` array is optional and currently only supports
|
||||
thick-provisioned ``raw`` snapshot exports.
|
||||
|
||||
Additional formats such as QCOW2, RBD export-format v2, and RBD export-diff
|
||||
snapshots will be added in a future release.
|
||||
|
||||
Streams
|
||||
~~~~~~~
|
||||
|
||||
The ``file`` stream can be used to import from a locally accessible POSIX file
|
||||
source. Its ``source-spec`` JSON is encoded as follows::
|
||||
|
||||
{
|
||||
<format unique parameters>
|
||||
"stream": {
|
||||
"type": "file",
|
||||
"file_path": "<file-path>"
|
||||
}
|
||||
}
|
||||
|
||||
For example, to import a raw-format image from a file located at
|
||||
"/mnt/image.raw", its ``source-spec`` JSON is encoded as follows::
|
||||
|
||||
{
|
||||
"type": "raw",
|
||||
"stream": {
|
||||
"type": "file",
|
||||
"file_path": "/mnt/image.raw"
|
||||
}
|
||||
}
|
||||
|
||||
The ``http`` stream can be used to import from a remote HTTP or HTTPS web
|
||||
server. Its ``source-spec`` JSON is encoded as follows::
|
||||
|
||||
{
|
||||
<format unique parameters>
|
||||
"stream": {
|
||||
"type": "http",
|
||||
"url": "<url-path>"
|
||||
}
|
||||
}
|
||||
|
||||
For example, to import a raw-format image from a file located at
|
||||
``http://download.ceph.com/image.raw``, its ``source-spec`` JSON is encoded
|
||||
as follows::
|
||||
|
||||
{
|
||||
"type": "raw",
|
||||
"stream": {
|
||||
"type": "http",
|
||||
"url": "http://download.ceph.com/image.raw"
|
||||
}
|
||||
}
|
||||
|
||||
The ``s3`` stream can be used to import from a remote S3 bucket. Its
|
||||
``source-spec`` JSON is encoded as follows::
|
||||
|
||||
{
|
||||
<format unique parameters>
|
||||
"stream": {
|
||||
"type": "s3",
|
||||
"url": "<url-path>",
|
||||
"access_key": "<access-key>",
|
||||
"secret_key": "<secret-key>"
|
||||
}
|
||||
}
|
||||
|
||||
For example, to import a raw-format image from a file located at
|
||||
`http://s3.ceph.com/bucket/image.raw`, its ``source-spec`` JSON is encoded
|
||||
as follows::
|
||||
|
||||
{
|
||||
"type": "raw",
|
||||
"stream": {
|
||||
"type": "s3",
|
||||
"url": "http://s3.ceph.com/bucket/image.raw",
|
||||
"access_key": "NX5QOQKC6BH2IDN8HC7A",
|
||||
"secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn"
|
||||
}
|
||||
}
|
||||
|
||||
.. note::
|
||||
The ``access_key`` and ``secret_key`` parameters support storing the keys in
|
||||
the MON config-key store by prefixing the key values with ``config://``
|
||||
followed by the path in the MON config-key store to the value. Values can be
|
||||
stored in the config-key store via ``ceph config-key set <key-path> <value>``
|
||||
(e.g. ``ceph config-key set rbd/s3/access_key NX5QOQKC6BH2IDN8HC7A``).
|
||||
|
||||
Execute Migration
|
||||
=================
|
||||
|
||||
|
||||
@ -75,13 +75,17 @@ test_import_native_format() {
|
||||
local base_image=$1
|
||||
local dest_image=$2
|
||||
|
||||
rbd migration prepare --import-only "rbd/${base_image}@2" ${dest_image}
|
||||
rbd migration abort ${dest_image}
|
||||
|
||||
local pool_id=$(ceph osd pool ls detail --format xml | xmlstarlet sel -t -v "//pools/pool[pool_name='rbd']/pool_id")
|
||||
cat > ${TEMPDIR}/spec.json <<EOF
|
||||
{
|
||||
"type": "native",
|
||||
"pool_id": ${pool_id},
|
||||
"pool_namespace": "",
|
||||
"image_name": "${base_image}"
|
||||
"image_name": "${base_image}",
|
||||
"snap_name": "2"
|
||||
}
|
||||
EOF
|
||||
cat ${TEMPDIR}/spec.json
|
||||
@ -91,11 +95,6 @@ EOF
|
||||
|
||||
compare_images "${base_image}@1" "${dest_image}@1"
|
||||
compare_images "${base_image}@2" "${dest_image}@2"
|
||||
compare_images "${base_image}" "${dest_image}"
|
||||
|
||||
rbd snap create ${dest_image}@head
|
||||
rbd bench --io-type write --io-pattern rand --io-size=32K --io-total=32M ${dest_image}
|
||||
compare_images "${base_image}" "${dest_image}@head"
|
||||
|
||||
rbd migration abort ${dest_image}
|
||||
|
||||
@ -105,24 +104,22 @@ EOF
|
||||
|
||||
compare_images "${base_image}@1" "${dest_image}@1"
|
||||
compare_images "${base_image}@2" "${dest_image}@2"
|
||||
compare_images "${base_image}" "${dest_image}"
|
||||
|
||||
rbd migration abort ${dest_image}
|
||||
|
||||
rbd migration prepare --import-only \
|
||||
--source-spec "{\"type\": \"native\", \"pool_id\": "${pool_id}", \"image_name\": \"${base_image}\"}" \
|
||||
--source-spec "{\"type\": \"native\", \"pool_id\": "${pool_id}", \"image_name\": \"${base_image}\", \"snap_name\": \"2\"}" \
|
||||
${dest_image}
|
||||
rbd migration abort ${dest_image}
|
||||
|
||||
rbd migration prepare --import-only \
|
||||
--source-spec "{\"type\": \"native\", \"pool_name\": \"rbd\", \"image_name\": \"${base_image}\"}" \
|
||||
--source-spec "{\"type\": \"native\", \"pool_name\": \"rbd\", \"image_name\": \"${base_image}\", \"snap_name\": \"2\"}" \
|
||||
${dest_image}
|
||||
rbd migration execute ${dest_image}
|
||||
rbd migration commit ${dest_image}
|
||||
|
||||
compare_images "${base_image}@1" "${dest_image}@1"
|
||||
compare_images "${base_image}@2" "${dest_image}@2"
|
||||
compare_images "${base_image}" "${dest_image}"
|
||||
|
||||
remove_image "${dest_image}"
|
||||
}
|
||||
|
||||
@ -11,9 +11,11 @@
|
||||
#include "include/neorados/RADOS.hpp"
|
||||
#include "include/rbd/features.h"
|
||||
#include "common/dout.h"
|
||||
#include "common/errno.h"
|
||||
#include "librbd/ImageCtx.h"
|
||||
#include "librbd/Features.h"
|
||||
|
||||
#include <boost/algorithm/string/predicate.hpp>
|
||||
#include <bitset>
|
||||
#include <random>
|
||||
|
||||
@ -23,6 +25,11 @@
|
||||
|
||||
namespace librbd {
|
||||
namespace util {
|
||||
namespace {
|
||||
|
||||
const std::string CONFIG_KEY_URI_PREFIX{"config://"};
|
||||
|
||||
} // anonymous namespace
|
||||
|
||||
const std::string group_header_name(const std::string &group_id)
|
||||
{
|
||||
@ -200,5 +207,37 @@ uint64_t reserve_async_request_id() {
|
||||
return ++async_request_seq;
|
||||
}
|
||||
|
||||
bool is_config_key_uri(const std::string& uri) {
|
||||
return boost::starts_with(uri, CONFIG_KEY_URI_PREFIX);
|
||||
}
|
||||
|
||||
int get_config_key(librados::Rados& rados, const std::string& uri,
|
||||
std::string* value) {
|
||||
auto cct = reinterpret_cast<CephContext*>(rados.cct());
|
||||
|
||||
if (!is_config_key_uri(uri)) {
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
std::string key = uri.substr(CONFIG_KEY_URI_PREFIX.size());
|
||||
std::string cmd =
|
||||
"{"
|
||||
"\"prefix\": \"config-key get\", "
|
||||
"\"key\": \"" + key + "\""
|
||||
"}";
|
||||
|
||||
bufferlist in_bl;
|
||||
bufferlist out_bl;
|
||||
int r = rados.mon_command(cmd, in_bl, &out_bl, nullptr);
|
||||
if (r < 0) {
|
||||
lderr(cct) << "failed to retrieve MON config key " << key << ": "
|
||||
<< cpp_strerror(r) << dendl;
|
||||
return r;
|
||||
}
|
||||
|
||||
*value = std::string(out_bl.c_str(), out_bl.length());
|
||||
return 0;
|
||||
}
|
||||
|
||||
} // namespace util
|
||||
} // namespace librbd
|
||||
|
||||
@ -276,6 +276,10 @@ SnapContext get_snap_context(
|
||||
|
||||
uint64_t reserve_async_request_id();
|
||||
|
||||
bool is_config_key_uri(const std::string& uri);
|
||||
int get_config_key(librados::Rados& rados, const std::string& uri,
|
||||
std::string* value);
|
||||
|
||||
} // namespace util
|
||||
} // namespace librbd
|
||||
|
||||
|
||||
@ -56,6 +56,7 @@ void HttpStream<I>::close(Context* on_finish) {
|
||||
|
||||
if (!m_http_client) {
|
||||
on_finish->complete(0);
|
||||
return;
|
||||
}
|
||||
|
||||
m_http_client->close(on_finish);
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
#include "librbd/migration/NativeFormat.h"
|
||||
#include "include/neorados/RADOS.hpp"
|
||||
#include "common/dout.h"
|
||||
#include "common/errno.h"
|
||||
#include "librbd/ImageCtx.h"
|
||||
#include "librbd/ImageState.h"
|
||||
#include "librbd/Utils.h"
|
||||
@ -29,6 +30,8 @@ const std::string POOL_NAME_KEY{"pool_name"};
|
||||
const std::string POOL_NAMESPACE_KEY{"pool_namespace"};
|
||||
const std::string IMAGE_NAME_KEY{"image_name"};
|
||||
const std::string IMAGE_ID_KEY{"image_id"};
|
||||
const std::string SNAP_NAME_KEY{"snap_name"};
|
||||
const std::string SNAP_ID_KEY{"snap_id"};
|
||||
|
||||
} // anonymous namespace
|
||||
|
||||
@ -123,6 +126,43 @@ void NativeFormat<I>::open(Context* on_finish) {
|
||||
return;
|
||||
}
|
||||
|
||||
auto& snap_name_val = m_json_object[SNAP_NAME_KEY];
|
||||
if (snap_name_val.type() == json_spirit::str_type) {
|
||||
m_snap_name = snap_name_val.get_str();
|
||||
} else if (snap_name_val.type() != json_spirit::null_type) {
|
||||
lderr(cct) << "invalid snap name" << dendl;
|
||||
on_finish->complete(-EINVAL);
|
||||
return;
|
||||
}
|
||||
|
||||
auto& snap_id_val = m_json_object[SNAP_ID_KEY];
|
||||
if (!m_snap_name.empty() && snap_id_val.type() != json_spirit::null_type) {
|
||||
lderr(cct) << "cannot specify both snap name and snap id" << dendl;
|
||||
on_finish->complete(-EINVAL);
|
||||
return;
|
||||
} else if (snap_id_val.type() == json_spirit::str_type) {
|
||||
try {
|
||||
m_snap_id = boost::lexical_cast<uint64_t>(snap_id_val.get_str());
|
||||
} catch (boost::bad_lexical_cast &) {
|
||||
}
|
||||
} else if (snap_id_val.type() == json_spirit::int_type) {
|
||||
m_snap_id = snap_id_val.get_uint64();
|
||||
}
|
||||
|
||||
if (snap_id_val.type() != json_spirit::null_type &&
|
||||
m_snap_id == CEPH_NOSNAP) {
|
||||
lderr(cct) << "invalid snap id" << dendl;
|
||||
on_finish->complete(-EINVAL);
|
||||
return;
|
||||
}
|
||||
|
||||
// snapshot is required for import to keep source read-only
|
||||
if (m_import_only && m_snap_name.empty() && m_snap_id == CEPH_NOSNAP) {
|
||||
lderr(cct) << "snapshot required for import" << dendl;
|
||||
on_finish->complete(-EINVAL);
|
||||
return;
|
||||
}
|
||||
|
||||
// TODO add support for external clusters
|
||||
librados::IoCtx io_ctx;
|
||||
int r = util::create_ioctx(m_image_ctx->md_ctx, "source image",
|
||||
@ -153,9 +193,63 @@ void NativeFormat<I>::open(Context* on_finish) {
|
||||
}
|
||||
|
||||
// open the source RBD image
|
||||
on_finish = new LambdaContext([this, on_finish](int r) {
|
||||
handle_open(r, on_finish); });
|
||||
m_image_ctx->state->open(flags, on_finish);
|
||||
}
|
||||
|
||||
template <typename I>
|
||||
void NativeFormat<I>::handle_open(int r, Context* on_finish) {
|
||||
auto cct = m_image_ctx->cct;
|
||||
ldout(cct, 10) << "r=" << r << dendl;
|
||||
|
||||
if (r < 0) {
|
||||
lderr(cct) << "failed to open image: " << cpp_strerror(r) << dendl;
|
||||
on_finish->complete(r);
|
||||
return;
|
||||
}
|
||||
|
||||
if (m_snap_id == CEPH_NOSNAP && m_snap_name.empty()) {
|
||||
on_finish->complete(0);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!m_snap_name.empty()) {
|
||||
std::shared_lock image_locker{m_image_ctx->image_lock};
|
||||
m_snap_id = m_image_ctx->get_snap_id(cls::rbd::UserSnapshotNamespace{},
|
||||
m_snap_name);
|
||||
}
|
||||
|
||||
if (m_snap_id == CEPH_NOSNAP) {
|
||||
lderr(cct) << "failed to locate snapshot " << m_snap_name << dendl;
|
||||
on_finish = new LambdaContext([on_finish](int) {
|
||||
on_finish->complete(-ENOENT); });
|
||||
m_image_ctx->state->close(on_finish);
|
||||
return;
|
||||
}
|
||||
|
||||
on_finish = new LambdaContext([this, on_finish](int r) {
|
||||
handle_snap_set(r, on_finish); });
|
||||
m_image_ctx->state->snap_set(m_snap_id, on_finish);
|
||||
}
|
||||
|
||||
template <typename I>
|
||||
void NativeFormat<I>::handle_snap_set(int r, Context* on_finish) {
|
||||
auto cct = m_image_ctx->cct;
|
||||
ldout(cct, 10) << "r=" << r << dendl;
|
||||
|
||||
if (r < 0) {
|
||||
lderr(cct) << "failed to set snapshot " << m_snap_id << ": "
|
||||
<< cpp_strerror(r) << dendl;
|
||||
on_finish = new LambdaContext([r, on_finish](int) {
|
||||
on_finish->complete(r); });
|
||||
m_image_ctx->state->close(on_finish);
|
||||
return;
|
||||
}
|
||||
|
||||
on_finish->complete(0);
|
||||
}
|
||||
|
||||
template <typename I>
|
||||
void NativeFormat<I>::close(Context* on_finish) {
|
||||
auto cct = m_image_ctx->cct;
|
||||
|
||||
@ -66,6 +66,11 @@ private:
|
||||
std::string m_pool_namespace;
|
||||
std::string m_image_name;
|
||||
std::string m_image_id;
|
||||
std::string m_snap_name;
|
||||
uint64_t m_snap_id = CEPH_NOSNAP;
|
||||
|
||||
void handle_open(int r, Context* on_finish);
|
||||
void handle_snap_set(int r, Context* on_finish);
|
||||
|
||||
};
|
||||
|
||||
|
||||
@ -89,8 +89,31 @@ void S3Stream<I>::open(Context* on_finish) {
|
||||
}
|
||||
|
||||
m_url = url_value.get_str();
|
||||
|
||||
librados::Rados rados(m_image_ctx->md_ctx);
|
||||
int r = 0;
|
||||
m_access_key = access_key.get_str();
|
||||
if (util::is_config_key_uri(m_access_key)) {
|
||||
r = util::get_config_key(rados, m_access_key, &m_access_key);
|
||||
if (r < 0) {
|
||||
lderr(m_cct) << "failed to retrieve access key from config: "
|
||||
<< cpp_strerror(r) << dendl;
|
||||
on_finish->complete(r);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
m_secret_key = secret_key.get_str();
|
||||
if (util::is_config_key_uri(m_secret_key)) {
|
||||
r = util::get_config_key(rados, m_secret_key, &m_secret_key);
|
||||
if (r < 0) {
|
||||
lderr(m_cct) << "failed to retrieve secret key from config: "
|
||||
<< cpp_strerror(r) << dendl;
|
||||
on_finish->complete(r);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
ldout(m_cct, 10) << "url=" << m_url << ", "
|
||||
<< "access_key=" << m_access_key << dendl;
|
||||
|
||||
@ -105,6 +128,7 @@ void S3Stream<I>::close(Context* on_finish) {
|
||||
|
||||
if (!m_http_client) {
|
||||
on_finish->complete(0);
|
||||
return;
|
||||
}
|
||||
|
||||
m_http_client->close(on_finish);
|
||||
|
||||
@ -1491,7 +1491,7 @@
|
||||
[--source-spec-path <source-spec-path>]
|
||||
[--source-spec <source-spec>] [--pool <pool>]
|
||||
[--namespace <namespace>] [--image <image>]
|
||||
[--dest-pool <dest-pool>]
|
||||
[--snap <snap>] [--dest-pool <dest-pool>]
|
||||
[--dest-namespace <dest-namespace>]
|
||||
[--dest <dest>] [--image-format <image-format>]
|
||||
[--new-format] [--order <order>]
|
||||
@ -1504,40 +1504,44 @@
|
||||
[--journal-splay-width <journal-splay-width>]
|
||||
[--journal-object-size <journal-object-size>]
|
||||
[--journal-pool <journal-pool>] [--flatten]
|
||||
<source-image-spec> <dest-image-spec>
|
||||
<source-image-or-snap-spec> <dest-image-spec>
|
||||
|
||||
Prepare image migration.
|
||||
|
||||
Positional arguments
|
||||
<source-image-spec> source image specification
|
||||
(example: [<pool-name>/[<namespace>/]]<image-name>)
|
||||
<dest-image-spec> destination image specification
|
||||
(example: [<pool-name>/[<namespace>/]]<image-name>)
|
||||
<source-image-or-snap-spec> source image or snapshot specification
|
||||
(example:
|
||||
[<pool-name>/[<namespace>/]]<image-name>[@<snap-n
|
||||
ame>])
|
||||
<dest-image-spec> destination image specification
|
||||
(example:
|
||||
[<pool-name>/[<namespace>/]]<image-name>)
|
||||
|
||||
Optional arguments
|
||||
--import-only only import data from source
|
||||
--source-spec-path arg source-spec file (or '-' for stdin)
|
||||
--source-spec arg source-spec
|
||||
-p [ --pool ] arg source pool name
|
||||
--namespace arg source namespace name
|
||||
--image arg source image name
|
||||
--dest-pool arg destination pool name
|
||||
--dest-namespace arg destination namespace name
|
||||
--dest arg destination image name
|
||||
--image-format arg image format [default: 2]
|
||||
--object-size arg object size in B/K/M [4K <= object size <= 32M]
|
||||
--image-feature arg image features
|
||||
[layering(+), exclusive-lock(+*), object-map(+*),
|
||||
deep-flatten(+-), journaling(*)]
|
||||
--image-shared shared image
|
||||
--stripe-unit arg stripe unit in B/K/M
|
||||
--stripe-count arg stripe count
|
||||
--data-pool arg data pool
|
||||
--mirror-image-mode arg mirror image mode [journal or snapshot]
|
||||
--journal-splay-width arg number of active journal objects
|
||||
--journal-object-size arg size of journal objects [4K <= size <= 64M]
|
||||
--journal-pool arg pool for journal objects
|
||||
--flatten fill clone with parent data (make it independent)
|
||||
--import-only only import data from source
|
||||
--source-spec-path arg source-spec file (or '-' for stdin)
|
||||
--source-spec arg source-spec
|
||||
-p [ --pool ] arg source pool name
|
||||
--namespace arg source namespace name
|
||||
--image arg source image name
|
||||
--snap arg source snapshot name
|
||||
--dest-pool arg destination pool name
|
||||
--dest-namespace arg destination namespace name
|
||||
--dest arg destination image name
|
||||
--image-format arg image format [default: 2]
|
||||
--object-size arg object size in B/K/M [4K <= object size <= 32M]
|
||||
--image-feature arg image features
|
||||
[layering(+), exclusive-lock(+*),
|
||||
object-map(+*), deep-flatten(+-), journaling(*)]
|
||||
--image-shared shared image
|
||||
--stripe-unit arg stripe unit in B/K/M
|
||||
--stripe-count arg stripe count
|
||||
--data-pool arg data pool
|
||||
--mirror-image-mode arg mirror image mode [journal or snapshot]
|
||||
--journal-splay-width arg number of active journal objects
|
||||
--journal-object-size arg size of journal objects [4K <= size <= 64M]
|
||||
--journal-pool arg pool for journal objects
|
||||
--flatten fill clone with parent data (make it independent)
|
||||
|
||||
Image Features:
|
||||
(*) supports enabling/disabling on existing images
|
||||
|
||||
@ -2644,13 +2644,15 @@ class TestMigration(object):
|
||||
create_image()
|
||||
with Image(ioctx, image_name) as image:
|
||||
image_id = image.id()
|
||||
image.create_snap('snap')
|
||||
|
||||
source_spec = json.dumps(
|
||||
{'type': 'native',
|
||||
'pool_id': ioctx.get_pool_id(),
|
||||
'pool_namespace': '',
|
||||
'image_name': image_name,
|
||||
'image_id': image_id})
|
||||
'image_id': image_id,
|
||||
'snap_name': 'snap'})
|
||||
dst_image_name = get_temp_image_name()
|
||||
RBD().migration_prepare_import(source_spec, ioctx, dst_image_name,
|
||||
features=63, order=23, stripe_unit=1<<23,
|
||||
@ -2667,6 +2669,12 @@ class TestMigration(object):
|
||||
|
||||
RBD().migration_execute(ioctx, dst_image_name)
|
||||
RBD().migration_commit(ioctx, dst_image_name)
|
||||
|
||||
with Image(ioctx, image_name) as image:
|
||||
image.remove_snap('snap')
|
||||
with Image(ioctx, dst_image_name) as image:
|
||||
image.remove_snap('snap')
|
||||
|
||||
RBD().remove(ioctx, dst_image_name)
|
||||
RBD().remove(ioctx, image_name)
|
||||
|
||||
|
||||
@ -138,7 +138,8 @@ void get_prepare_arguments(po::options_description *positional,
|
||||
"source-spec file (or '-' for stdin)")
|
||||
("source-spec", po::value<std::string>(),
|
||||
"source-spec");
|
||||
at::add_image_spec_options(positional, options, at::ARGUMENT_MODIFIER_SOURCE);
|
||||
at::add_image_or_snap_spec_options(positional, options,
|
||||
at::ARGUMENT_MODIFIER_SOURCE);
|
||||
at::add_image_spec_options(positional, options, at::ARGUMENT_MODIFIER_DEST);
|
||||
at::add_create_image_options(options, true);
|
||||
at::add_flatten_option(options);
|
||||
@ -146,13 +147,18 @@ void get_prepare_arguments(po::options_description *positional,
|
||||
|
||||
int execute_prepare(const po::variables_map &vm,
|
||||
const std::vector<std::string> &ceph_global_init_args) {
|
||||
bool import_only = vm["import-only"].as<bool>();
|
||||
|
||||
size_t arg_index = 0;
|
||||
std::string pool_name;
|
||||
std::string namespace_name;
|
||||
std::string image_name;
|
||||
std::string snap_name;
|
||||
int r = utils::get_pool_image_snapshot_names(
|
||||
vm, at::ARGUMENT_MODIFIER_SOURCE, &arg_index, &pool_name, &namespace_name,
|
||||
&image_name, nullptr, true, utils::SNAPSHOT_PRESENCE_NONE,
|
||||
&image_name, import_only ? &snap_name : nullptr, true,
|
||||
import_only ? utils::SNAPSHOT_PRESENCE_PERMITTED :
|
||||
utils::SNAPSHOT_PRESENCE_NONE,
|
||||
utils::SPEC_VALIDATION_NONE);
|
||||
if (r < 0) {
|
||||
return r;
|
||||
@ -169,7 +175,6 @@ int execute_prepare(const po::variables_map &vm,
|
||||
return r;
|
||||
}
|
||||
|
||||
bool import_only = vm["import-only"].as<bool>();
|
||||
std::string source_spec;
|
||||
if (vm.count("source-spec") && vm.count("source-spec-path")) {
|
||||
std::cerr << "rbd: cannot specify both source-image-spec and "
|
||||
@ -223,12 +228,18 @@ int execute_prepare(const po::variables_map &vm,
|
||||
}
|
||||
|
||||
if (import_only && source_spec.empty()) {
|
||||
if (snap_name.empty()) {
|
||||
std::cerr << "rbd: snapshot name was not specified" << std::endl;
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
std::stringstream ss;
|
||||
ss << R"({)"
|
||||
<< R"("type":"native",)"
|
||||
<< R"("pool_id":)" << io_ctx.get_id() << R"(,)"
|
||||
<< R"("pool_namespace":")" << io_ctx.get_namespace() << R"(",)"
|
||||
<< R"("image_name":")" << image_name << R"(")"
|
||||
<< R"("image_name":")" << image_name << R"(",)"
|
||||
<< R"("snap_name":")" << snap_name << R"(")"
|
||||
<< R"(})";
|
||||
source_spec = ss.str();
|
||||
|
||||
@ -238,12 +249,19 @@ int execute_prepare(const po::variables_map &vm,
|
||||
}
|
||||
io_ctx = dst_io_ctx;
|
||||
image_name = dst_image_name;
|
||||
snap_name = "";
|
||||
} else if (!import_only && !source_spec.empty()) {
|
||||
std::cerr << "rbd: --import-only must be used in combination with "
|
||||
<< "source-spec/source-spec-path" << std::endl;
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
if (!snap_name.empty()) {
|
||||
std::cerr << "rbd: snapshot name specified for a command that doesn't "
|
||||
<< "use it" << std::endl;
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
librbd::ImageOptions opts;
|
||||
r = utils::get_image_options(vm, true, &opts);
|
||||
if (r < 0) {
|
||||
|
||||
Loading…
Reference in New Issue
Block a user