Commit Graph

116432 Commits

Author SHA1 Message Date
Jason Dillaman
81f9b91d62 qa/tasks: support explicit disk configuration for qemu task
The 'disks' key will now be treated as a dictionary where all previous
global settings can be individually applied. Additionally, a disk can be
pre-created and provided for use by qemu.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-02 19:58:55 -05:00
Jason Dillaman
4ab2a183ca rbd: add new 'migration prepare --import-only' CLI optional
The '--import-only' optional can be combined with either a
'--source-spec' or '--source-spec-path' optional to define
the source for the read-only import.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-02 19:57:58 -05:00
Jason Dillaman
87bc86e311 rbd: show migration source spec in image status
When using the non-legacy mode image migration, the 'rbd status'
command will now show the source-spec for the migration if the
source pool id is invalid.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:44:30 -05:00
Jason Dillaman
51478deab2 rbd: ensure the help printer doesn't print past the end of the line
When long command names and long optional names are combined,
it's possible for the help text to be printed beyond the 80
character limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:44:30 -05:00
Jason Dillaman
9edc30366b librbd: replace direct reads from ImageRequest with ImageDispatchSpec
Read requests will always need to go via the image dispatch layer to
ensure migrations are properly handled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:44:29 -05:00
Jason Dillaman
8a9f6dde28 librbd/migration: allow native v2 images to be opened by name
When in import-only mode, allow the source image to be opened by
name in case it's a V2 image format w/o its image id specified in
the JSON.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
9304f93bc6 librbd/migration: close image if failed to open source
To remain consistent between the native which will automatically close
the ImageCtx if it fails to open the image, the raw format should also
close the ImageCtx.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
ecac0a7563 librbd/migration: initialize default source image layout
For backing source formats that don't care about the low-level
RBD layout details, provide safe default values for the layout
to ensure that striping logic will still function.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
bb38c391b6 librbd: fix potential recursive lock of ImageCtx::image_lock
If the "list_snaps" IO call returns immediately, it can result
in an attempted recursive lock starting from CopyupRequest.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
0ef9cec0bd librbd: migration execute should allow "EXECUTING" state
If the "execute" action was interrupted, we need to be able to
restart it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
d63a564f22 librbd: read image size when opening migration raw format
The migration prepare operation will need to know the size of
the source image so that it can create the destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
9ad4d7936a librbd/migration: fix issue with prepare-import and native-format
When preparing an import, the native format can not expect to have
the child image opened since it hasn't been created yet.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
d6133f9bf2 librbd: added new 'migration_prepare_import' API methods
These related methods accept a JSON-encoded source-spec that
describes how to read from the source image. The migration is
a one-way operation, so children/groups/etc won't be updated
to point to the new destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
e5398d1429 librbd: store migration progress on both source+destination images
When migrating from an import-only source-spec like an external
file, it will be important to also store the progress on the
destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
5869d13fb5 librbd: switch to migration source-spec builder
Remove all the hard-coded references to the NativeFormat
handler and allow the RawFormat+FileStream handlers to be
instantiated.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
c7bbb676a8 librbd: switch migration native format to use source-spec JSON object
Remove the dependency on the MigrationInfo struct and allow the native
format to directly pull parameters from the JSON-encoded object
structure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
c87fe97761 librbd: tweak native-format migration source-spec
Renamed "format" property to "type" and always include the original
"image_id".

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
43749377cb librbd/api: ImageCtx must be closed instead of deleted
On error, the destination image context will need to be properly
closed instead of just deleted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 09:22:38 -05:00
Jason Dillaman
6fef5198b3
Merge pull request #36626 from lixiaoy1/rbd_image_discard
librbd/rwl: discard cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-11-01 08:42:01 -05:00
Kefu Chai
8a58d7fdf5
Merge pull request #37870 from xxhdx1985126/wip-crimson-recovery-bug-fixes
crimson: recovery-related bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-11-01 21:23:59 +08:00
Kefu Chai
00b78a6ad5
Merge pull request #37910 from tchaikov/wip-crimson-osd-send-pg-temp
crimson/osd: merge pg_temp_wanted before sending them

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 21:21:42 +08:00
Yuval Lifshitz
2996d138cb
Merge pull request #37656 from yuvalif/allow_lua_to_set_metadata
rgw/lua: allow setting metadata via lua
2020-11-01 12:03:26 +02:00
Kefu Chai
4d17fda7e3 crimson/osd: merge pg_temp_wanted before sending them
there is chance that new pg_temp_wanted is added when we are sending
them in ShardServices::send_pg_temp(), but the pg temp are already
collected into the messages before seastar::parallel_for_each(). so,
we cannot move all elements from pg_temp_wanted to pg_temp_pending after
sending the messages, that'd also move the unsent pg temps to
pg_temp_pending as well. so we have to do this before do the async call,

this issue was root caused by Xuehan Xu <xxhdx1985126@gmail.com>.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-01 11:55:42 +08:00
Kefu Chai
dbbd2d32f3 crimson/osd: drop ShardServices::_sent_pg_temp()
there is no need to define a wrapper for moving pg_temp_wanted to
pg_temp_pending. and it hurts the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-01 11:55:03 +08:00
Xuehan Xu
124aa74deb crimson/osd: report statfs data
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 11:37:21 +08:00
Xuehan Xu
471531e25d crimson/osd: rename heartbeat_timer to tick_timer
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 11:37:21 +08:00
Xuehan Xu
8b7d0ae035 crimson/common: make Gated handle gate_close_exception
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 11:37:21 +08:00
Kefu Chai
6656cae082 crimson/osd: do not check HAVE_STDLIB_MAP_SPLICING
seastar always uses GCC-9 and up, so there is no need to check for
the existence of map::merge() before using it anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-01 11:28:28 +08:00
Xuehan Xu
d205776796 crimson/osd: clean up pending_txn when actingset is changed
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 10:39:24 +08:00
Xuehan Xu
c7e6b6f021 crimson/osd: fix pg info statistics error during recovery
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 10:38:45 +08:00
Xuehan Xu
3a8f133923 crimson/osd: make I/O reqs wait for object recovery if it's degraded or backfilling
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-01 10:25:22 +08:00
Kefu Chai
038750c78a
Merge pull request #37327 from kamoltat/wip-mgr-progress-global-event
mgr/progress: Global Recovery Event in ceph -s

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-10-31 11:43:27 +08:00
Kefu Chai
a45fab9f81
Merge pull request #37598 from ronen-fr/scrubstore-fix-rf
osd: fix signatures of get_store_errors() and friends

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
2020-10-31 11:42:35 +08:00
Kefu Chai
e96c45d9d2
Merge pull request #37751 from changchengx/nasm_avx512
common: use nasm to build isa-l and isa-l_crypto to prepare for AVX512

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-10-31 11:41:19 +08:00
Josh Durgin
8a2f2d82f4
Merge pull request #37773 from sunnyku/wip-f33-pyfix
do_cmake.sh: use python-3.9 with fedora version 33

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-10-30 08:15:59 -07:00
Jason Dillaman
0cd14ccff2
Merge pull request #37789 from trociny/wip-47827
mgr/rbd_support: create mirror snapshots asynchronously

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-10-30 09:40:10 -04:00
Yuval Lifshitz
e96331fdab rgw/lua: use lua exceptions instead of ceph_asserts
also use lua assert() in uni tests instead of print()
fix bug when accessing user id in an ACL grant

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2020-10-30 14:50:51 +02:00
Lenz Grimmer
21b24e0247
Merge pull request #37482 from rhcs-dashboard/datatable_click_prevent
mgr/dashboard: Prevent table items from getting selected while expanding

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-10-30 09:55:47 +01:00
Lenz Grimmer
587a84049b
Merge pull request #35769 from Codom/master
mgr/dashboard: Added Versioning to the REST API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-10-30 09:50:05 +01:00
Yuval Lifshitz
eaf48d1a3a rgw/lua: allow setting metadata via lua
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2020-10-30 10:40:06 +02:00
Kefu Chai
d162c00c66
Merge pull request #37788 from tchaikov/wip-zbd
rpm,cmake: s/WITH_LIBZBD/WITH_ZBD/ and enable ZBD on demand

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
2020-10-30 11:17:56 +08:00
Kefu Chai
92300bb747
Merge pull request #37888 from tchaikov/wip-crimson-gtest
crimson/gtest_seastar: do not keep a copy of argv

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2020-10-30 10:25:49 +08:00
Kefu Chai
2cdbbd1212
Merge pull request #37891 from tchaikov/wip-cmake-alienstore
cmake: build bluestore-tp before crimson-alienstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2020-10-30 10:25:06 +08:00
Kefu Chai
75c7a792e6
Merge pull request #37893 from tchaikov/wip-wunused-variable
osd/osd_types: mark unused variable [[maybe_unused]]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-30 10:24:11 +08:00
Jason Dillaman
190534ae08
Merge pull request #37864 from trociny/wip-48020
mgr/rbd_support: store global schedule without localized prefix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-10-29 20:58:44 -04:00
Jason Dillaman
54329691c0
Merge pull request #37880 from trociny/wip-rbd-finisher-cancel
librbd: relax requirements on finisher canceled callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-10-29 20:58:13 -04:00
Mykola Golub
08151a191f mgr/rbd_support: create mirror snapshots asynchronously
To scale up with number of images.

Fixes: https://tracker.ceph.com/issues/47827
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-10-29 19:12:01 +00:00
Mykola Golub
13ce488d7a pybind/rbd: add async mirror image get mode and info methods
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-10-29 19:12:01 +00:00
Mykola Golub
fd5e642cd7 librbd: add missing declaration of rbd_aio_mirror_image_get_mode
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-10-29 19:12:01 +00:00
Mykola Golub
b27db87c03 pybind/rbd: add image async open and close methods
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-10-29 19:12:01 +00:00