Commit Graph

116697 Commits

Author SHA1 Message Date
Lucian Petrut
4af1ae2093 rbd: allow mounting images on Windows
This change will allow mapping rbd images on Windows, leveraging the
WNBD[1] Virtual Storport Miniport driver [2].

The behavior and CLI is similar to the Linux rbd-nbd, with a few
notable differences:

* device paths cannot be requested. The disk number and path will
  be picked by Windows. If a device path is provided by the user
  when mapping an image, it will be used as an identifier, which
  can also be used when unmapping the image.
* the "show" command was added, which describes a specific mapping.
  This can be used for retrieving the disk path.
* the "service" command was added, allowing rbd-wnbd to run as a
  Windows service. All mappings are currently perisistent, being
  recreated when the service stops, unless explicitly unmapped.
  The service disconnects the mappings when being stopped.
* the "list" command also includes a "status" column.

The purpose of the "service" mode is to ensure that mappings survive
reboots and that the Windows service start order can be adjusted so
that rbd images can be mapped before starting services that may depend
on it, such as VMMS.

The mapped images can either be consumed by the host directly or exposed
to Hyper-V VMs.

While at it, we'll skip building rbd-mirror as it's quite unlikely that
this daemon is going to be used on Windows for now.

[1] https://github.com/cloudbase/wnbd
[2] https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/overview-of-storage-virtual-miniport-drivers

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
2020-11-18 10:31:24 +00:00
Lucian Petrut
c17c53f460 rbd: add missing FileStream::get_size
At the moment, a few methods will be stubbed if
BOOST_ASIO_HAS_POSIX_STREAM_DESCRIPTOR is undefined.

The issue is that FileStream::get_size is undefined, so we're
getting link issues on Windows.

In the future, we might consider using asio::windows::stream_handle
on Windows.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-18 10:31:24 +00:00
Lucian Petrut
8d35b2ecea build: disable stack protection on Windows
Passing "-fstack-protector-strong" doesn't seem to work with Mingw,
complaining about undefied "__stack_chk_fail". For this reason,
we'll disable it for now.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-18 10:31:24 +00:00
Lucian Petrut
e3e00875bd rbd: Update Windows readme
This change updates the Windows readme, describing rbd-wnbd
configuration and usage.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-18 10:31:24 +00:00
Lucian Petrut
1dbe03a073 rbd: fix import image path parsing on Windows
When importing an image, the rbd command uses only the file name
and expects "/" to be used as a separator. On Windows, it will
use the entire path as image name since the path separator is not
the same.

This change updates it so that the "\\" path separator can be
properly handled as well.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-18 10:31:24 +00:00
Lucian Petrut
00a5cd1d43 common: move Windows files to a separate folder
We ended up with quite a few files having the "_win32" suffix.
It's probably better if we move them to a separate folder and drop
the suffix.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-18 10:31:17 +00:00
Lucian Petrut
d2b5d85624 common: Add Windows service helpers
We'll add a class that implements Windows service hooks. Subclasses
must overide the start, stop and shutdown hooks.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
2020-11-18 09:23:51 +00:00
Lucian Petrut
32b33f372b common: Add Windows registry key utils
We'll add some helpers facilitating Windows registry key operations.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
2020-11-18 09:23:43 +00:00
Lucian Petrut
f1a9a3739c common: Add win32_strerror and errno_to_ntstatus
dlfcn_win32.cc provides a function converting Windows error codes
to string error messages. We'll move it to the common errno modules
so that it can easily be reused.

Add the same time, we're adding a function that's converting
errno values to NTSTATUS codes.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-11-12 10:23:22 +00:00
Kefu Chai
d48c6c4ab9
Merge pull request #37943 from xxhdx1985126/wip-crimson-recovery-bug-fix
crimson: recovery bug fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-11-10 18:07:16 +08:00
Kefu Chai
33da4a5a05
Merge pull request #37997 from rhcs-dashboard/48155-fix-api-test-full-health
mgr/dashboard: fix API test: test_full_health

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-11-10 17:53:50 +08:00
Alfonso Martínez
6add853fe4 mgr/dashboard: fix API test: test_full_health
Fixes: https://tracker.ceph.com/issues/48155
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
2020-11-10 09:29:45 +01:00
Jan Fajerski
e9713a085a Merge PR #37942 into master
* refs/pull/37942/head:
	ceph-volume: fix lvm help test
	ceph-volume: add a unit tests to lvm batch
	ceph-volume: fix lvm batch auto with full SSDs

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-11-10 08:30:55 +01:00
Xuehan Xu
1510eb0fdd crimson/osd: make non-first push ops' version to be the version of the recovering object
An object may take more than one push op to recover, those later push ops also need to
set their version accordingly

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-10 15:20:48 +08:00
Xuehan Xu
cf07974c89 crimson/osd: make PglogBasedRecovery op take recovering objs triggered elsewhere into account
PGRecovery::start_recovery_ops() should wait for all inflight recovery ops, whether they are
started by BackgroundRecovery or not, otherwise there may be circumstances in which BackgroundRecovery
keep recursively invoking its do_recovery when start_recovery_ops returns recovery done while there are
still missing objects.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-11-10 15:18:01 +08:00
David Zafman
ada78607e6
Merge pull request #37941 from dzafman/wip-48077
Allowing scrub configs begin_day/end_day to include 7 and begin_hour/end_hour to include 24 is confusing

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-11-09 15:12:34 -08:00
Josh Durgin
e90f0fb70f
Merge pull request #37539 from zdover23/wip-doc-ceph-df-2020-oct
doc/rados: ceph df output update

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-11-09 15:09:45 -08:00
David Zafman
d7abac3ea9 test: unittest_osdscrub: Remove invalid hour test and add day of week testing
Signed-off-by: David Zafman <dzafman@redhat.com>
2020-11-09 22:47:00 +00:00
David Zafman
d3cc647583 osd: Eliminate day of weeek 7 and hour 24
Add test case for permitted hours to make sure scrub doesn't start
Remove permitted hours in extended sleep test

Fixes: https://tracker.ceph.com/issues/48077

Signed-off-by: David Zafman <dzafman@redhat.com>
2020-11-09 22:47:00 +00:00
Zac Dover
f17b48f72e doc/rados: ceph df output update
This commit updates the "ceph df" output
so that it is current as of October 2020.
-Add correctly formatted `ceph df` output.
-Add explanation of "DIRTY" column.
-(DATA) remains to be defined (1 instance)
-(OMAP) remains to be defined (1 instance)
-USED remains to be defined (1 instance)
-Update prompts in "Checking OSD Status"

Fixes: https://tracker.ceph.com/issues/47523
Signed-off-by: Zac Dover <zac.dover@gmail.com>
2020-11-10 08:01:02 +10:00
Patrick Donnelly
8aae1dfdf1
Merge PR #37932 into master
* refs/pull/37932/head:
	src/script/ceph-backport.sh: abort if jq not available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-11-09 08:17:18 -08:00
Dimitri Savineau
879ed30984 ceph-volume: fix lvm help test
ed5ceb0 changed the LVM help code but not the associated test.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-09 10:55:18 -05:00
Guillaume Abrioux
13514a24cf ceph-volume: add a unit tests to lvm batch
This commit adds unit tests in order to cover `_sort_rotational_disks()`
call when deploying with full hdd/ssd or mixed hdd/sdd scenarios.

Fixes: https://tracker.ceph.com/issues/48150

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Co-authored-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-09 10:54:50 -05:00
Dimitri Savineau
2a854ca373 ceph-volume: fix lvm batch auto with full SSDs
The ceph-volume lvm batch --auto introduced by [1] breaks the backward
compatibility when using non rotational devices only (SSD and/or NVMe).
Those devices are reaffected as bluestore db or filestore journal
devices while we want them as data devices.

Fixes: https://tracker.ceph.com/issues/48106

[1] https://github.com/ceph/ceph/pull/34740

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-09 10:54:50 -05:00
Kefu Chai
154830be84
Merge pull request #37928 from tchaikov/wip-qa-ceph-ctx-archive
qa/tasks/ceph: extract update_archive_setting()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-11-09 23:48:30 +08:00
Lenz Grimmer
97182ebf5d
Merge pull request #37453 from tspmelo/wip-rbd-change-detection
mgr/dashboard: Improve Change Detection on RBD Snapshot 

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
2020-11-09 15:10:24 +01:00
Lenz Grimmer
610a2426a9
Merge pull request #37641 from henyxia/dev/henyxia/add-mgr-dashboard-prom-ssl-verify
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-11-09 15:07:57 +01:00
Kefu Chai
5c5a038acb
Merge pull request #37983 from tchaikov/wip-crimson-os-co-locate-alien-allocator
crimson/os: do not configure seastar allocator for alien threads

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-11-09 21:34:20 +08:00
Jan Fajerski
d544dfb549 Merge PR #32777 into master
* refs/pull/32777/head:
	common,log,msg: Fix win32 compiler warnings
	win32,common: provide a default ceph.conf location
	common: add newline before parsing config on Windows
	common: verify unix sockets support on Windows
	common: fix md_config_t::get_cluster_name
	global, common: Print config parse errors
	common: accept config files containing whitespaces
	librbd: avoid broken mingw rng
	common: define DEV_NULL
	tools,common,msg: Use O_BINARY for win32 compatibility
	log: fix timestamps on Windows
	common,librbd: add portable ceph_memzero_s
	common,msg: Fix socket handling
	common: Extend win32 compatbility layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-11-09 14:10:36 +01:00
Kefu Chai
dd073a0061
Merge pull request #37984 from tchaikov/wip-fio
cmake: drop WITH_SYSTEM_FIO option

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2020-11-09 21:08:13 +08:00
Lenz Grimmer
75cc8e1213
Merge pull request #37965 from rhcs-dashboard/about-modal-cleanup
mgr/dashboard: Moving copyright variable into the app.constants

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2020-11-09 13:37:55 +01:00
Jason Dillaman
ede99b592c
Merge pull request #37958 from ronen-fr/wip-ronenf-librbd-move1
librbd: removing a "pessimizing move"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-11-09 07:14:43 -05:00
Kefu Chai
9d2f700731
Merge pull request #37990 from tchaikov/wip-liburing-fPIC
cmake: build liburing with -fPIC

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-11-09 20:10:05 +08:00
Gregory Farnum
703524119a
Merge pull request #37915 from gregsfortytwo/wip-stretch-fixes
Fix several stretch mode issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-11-09 03:52:07 -08:00
Jan Fajerski
47842715ba Merge PR #37741 into master
* refs/pull/37741/head:
	ceph-volume: remove mention of dmcache from docs and help text

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-11-09 11:33:18 +01:00
Jan Fajerski
b5294b7680 Merge PR #37896 into master
* refs/pull/37896/head:
	ceph-volume: implement the --log-level flag

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-11-09 11:30:35 +01:00
Lenz Grimmer
a839fab336
Merge pull request #37506 from rhcs-dashboard/fix-47478-master
mgr/dashboard: Improvements in cluster > OSDs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-11-09 10:45:35 +01:00
Lenz Grimmer
2da2c15304
Merge pull request #37649 from rhcs-dashboard/add-header-tooltip
mgr/dashboard: add description for CRUSH Ruleset

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2020-11-09 10:43:50 +01:00
Kefu Chai
3510433f9c cmake: build liburing with -fPIC
in liburing,
75cad68b95
partially reverts
4e360f7113,
which builds liburing.a with -fPIC.

so we need to pass -fPIC by ourselves. otherwise we'd have

/usr/bin/ld: ../../liburing/src/liburing.a(setup.ol): relocation R_X86_64_PC32 against symbol `io_uring_queue_mmap' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
src/test/fio/CMakeFiles/fio_ceph_objectstore.dir/build.make:154: recipe for target 'lib/libfio_ceph_objectstore.so' failed

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 17:10:59 +08:00
Kefu Chai
c976f142f0 cmake: expose FIO_INCLUDE_DIR via libfio
always use the fio built from source for better consistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 16:01:59 +08:00
Kefu Chai
d16f7fb524 cmake: drop WITH_SYSTEM_FIO option
there is no well-known distro packaging fio.h, so drop this option.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 16:01:59 +08:00
Kefu Chai
8e7b9f27c6
Merge pull request #37981 from tchaikov/wip-make-dist
make-dist: add liburing to dist tarball

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2020-11-09 15:58:16 +08:00
Kefu Chai
b6bae7ba13 cmake: use make explicitly to build fio
we cannot assume that user uses "make" as the generator of cmake, if,
for instance, ninja is used, `$(MAKE)` is not a valid variable in the
generated `build.ninja`. so we should use "make" explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 15:49:26 +08:00
Kefu Chai
cafc6f3575 crimson/os: do not configure seastar allocator for alien threads
4cd2b00d2a allows us to colocate seastar
allocator used by seastar reactors and libc allocator used by alien threads,
there is no need to configure seastar allocator for alien thread
anymore.

this change partially reverts

- f7715b86ef
- b55919071f

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 14:47:39 +08:00
Kefu Chai
4430cec592 crimson/os: do not #include unused headers
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 14:44:36 +08:00
Kefu Chai
68eb5bcdce cmake: use src/liburing for building liburing if it exists
so we can build with liburing enabled if the dist tarball contains
liburing.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 13:11:35 +08:00
Kefu Chai
0d21f98d71 make-dist: use a loop to collect the ingredients of the dist tarball
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 12:33:47 +08:00
Kefu Chai
040d44894d make-dist: add liburing to dist tarball
since liburing is enabled by default, let's included it in the dist
tarball.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 12:33:47 +08:00
Kefu Chai
b77aa9d685 make-dist: extract download_from() out
so we can reuse it

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-09 12:33:47 +08:00
Patrick Donnelly
4052b11f18
Merge PR #36998 into master
* refs/pull/36998/head:
	doc/mgr/volumes: Document authorize/deauthorize cli commands
	qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`
	mgr/volumes: Allow/deny auth IDs access to FS subvolumes
	mon/MonCap: allow 'profile mgr' to create/update/del auth IDs

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-11-06 20:51:54 -08:00