Commit Graph

105193 Commits

Author SHA1 Message Date
Roman Penyaev
2268be9ff9 os/bluestore: introduce new io_uring IO engine
This implements low-level IO engine, which utilizes brand-new
io_uring IO interface: https://lwn.net/Articles/776428/

By default libaio is used.  If bluestore_ioring=true is set but kernel
does not support io_uring or architecture is not x86-64, libaio will be
used instead.

In current patch liburing library is used in order not to open code
everything.

In order to compile with liburing WITH_LIBURING=ON should be specified.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
2019-12-10 17:22:15 +08:00
Roman Penyaev
c05c1bd7b9 os/bluestore: make abstract interface for any sorts of AIO queues
In the next patch new io_uring API will be used instead of libaio.
So this prepares the abstract interface.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
2019-12-10 17:21:38 +08:00
Kefu Chai
6c7f395233
Merge pull request #32007 from tchaikov/wip-osd-cleanup
osd: use unique_ptr for managing life cycles

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2019-12-10 17:18:03 +08:00
Kefu Chai
425368c888
Merge pull request #31931 from majianpeng/bluestore-bugfix
os/bluestore: fix wakeup bug

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2019-12-10 17:13:21 +08:00
Kefu Chai
a11ae900e9
Merge pull request #32052 from mgfritch/wip-cd-standalone-tempfiles
ceph-daemon: clean-up tempfiles on EXIT

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-12-10 16:57:48 +08:00
Yehuda Sadeh
b976dc5aa3
Merge pull request #32100 from yehudasa/wip-43184
qa: radosgw-admin: remove dependency on bunch package

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-12-10 07:51:27 +02:00
Jos Collin
8df6e03d80
Merge pull request #31961 from joscollin/wip-B43038-subvolume-group-create-fail
mgr/volumes: check for string values in uid/gid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-10 11:12:03 +05:30
Patrick Donnelly
4635f943cd
Merge PR #32124 into master
* refs/pull/32124/head:
	doc/cephfs/disaster-recovery-experts: Add link for scrub and note for scrub_path

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-09 19:52:41 -08:00
Patrick Donnelly
12e7401a43
Merge PR #27894 into master
* refs/pull/27894/head:
	cephfs-shell: Add default list to df
	cephfs-shell: Add tests for df command
	cephfs-shell: Fix 'df' command errors

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-09 17:05:52 -08:00
Patrick Donnelly
6cd51077e2
Merge PR #32131 into master
* refs/pull/32131/head:
	codeowners: Add ceph2.py to @ceph/orchestrators

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-09 16:39:36 -08:00
Patrick Donnelly
b8635afe10
Merge PR #32038 into master
* refs/pull/32038/head:
	doc: reorganize CephFS landing page and ToC

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2019-12-09 16:36:59 -08:00
Sage Weil
dcb5e9b6d8 Merge PR #32098 into master
* refs/pull/32098/head:
	ceph-daemon: py2: tolerate whitespace before config key name

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-12-09 18:29:41 -06:00
Nathan Cutler
d9a0e9b0d7
Merge pull request #32037 from smithfarm/wip-43152-workaround
build/ops: make-dist: default to no dashboard frontend build parallelism

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2019-12-10 00:42:16 +01:00
Ilya Dryomov
0bc226f6e1
Merge pull request #32088 from idryomov/wip-krbd-exclusive-lock-fixups
qa: krbd_exclusive_option.sh: update for recent kernel changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-12-09 22:59:13 +01:00
Sage Weil
bffe2dd9e9 Merge PR #32046 into master
* refs/pull/32046/head:
	mgr/DaemonServer: fix 'osd ok-to-stop' for EC pools

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-12-09 15:34:57 -06:00
Sage Weil
ae60fd9519 Merge PR #31489 into master
* refs/pull/31489/head:
	pybind/rados: add WriteOp::writesame() and test WriteOp::writesame()
	pybind/rados: add Ioctx::aio_writesame() and test Ioctx::aio_writesame()
	pybind/rados: add Ioctx::writesame() and test Ioctx::writesame()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2019-12-09 14:46:40 -06:00
Sage Weil
946a8f7cb5 Merge PR #29214 into master
* refs/pull/29214/head:
	mgr: templatize/generalize metrics collection interface

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-09 14:46:09 -06:00
Sage Weil
5993dd82b0 Merge PR #32102 into master
* refs/pull/32102/head:
	mgr/diskprediction_local: import scipy early to fix self-test deadlock

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-12-09 14:44:37 -06:00
Jason Dillaman
703a4ebde4
Merge pull request #31794 from dillaman/wip-42889
librbd: disable zero-copy writes by default

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
2019-12-09 12:47:47 -05:00
Jason Dillaman
bd76f0d10c
Merge pull request #32105 from junxfl/test1
librbd: fix rbd_open_by_id, rbd_open_by_id_read_only 

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-12-09 12:37:37 -05:00
Daniel Gryniewicz
105c6d78aa
Merge pull request #32109 from yehudasa/wip-42868
rgw: list buckets: don't return buckets if limit=0
2019-12-09 12:23:12 -05:00
Sebastian Wagner
57c3e344ab codeowners: Add ceph2.py to @ceph/orchestrators
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-12-09 17:46:57 +01:00
Lenz Grimmer
22d5bd95e2
Merge pull request #32053 from votdev/left_align_badges
mgr/dashboard: Left align badge datatable columns

Reviewed-by: Laura Paduano <lpaduano@suse.com>
2019-12-09 16:44:56 +00:00
Lenz Grimmer
d5c54d55eb
Merge pull request #31927 from bk201/wip-43072
mgr/dashboard: hide in-use devices when creating OSDs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-12-09 16:42:54 +00:00
Lenz Grimmer
9960f9365a
Merge pull request #32084 from votdev/auth_service_observable
mgr/dashboard: Use Observable in auth.service

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
2019-12-09 16:34:50 +00:00
Lenz Grimmer
a494757a6e
mgr/dashboard: Improve badge column template. (#32026)
mgr/dashboard: Improve badge column template.

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
2019-12-09 16:32:03 +00:00
Sage Weil
f886610e44 Merge PR #31907 into master
* refs/pull/31907/head:
	mgr/dashboard: properly handle a missing rbd-mirror service status
	mgr: cull service daemons when the last instance has been removed
	mgr: ensure new daemons are properly indexed by hostname

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2019-12-09 10:30:26 -06:00
Ilya Dryomov
3aae140353 qa: kernel.sh: unlock before rolling back
"rbd snap rollback" expects an unlocked image, but we may get there
locked if object map is enabled (or if lock_on_read is specified in
rbd_default_map_options).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-12-09 17:29:13 +01:00
Ilya Dryomov
50933b863a qa: krbd_exclusive_option.sh: update for recent kernel changes
Since 5.3:
- a plain "rbd map" acquires the lock, so it's not different from
  "rbd map -o exclusive" in this regard
- if the lock is held by the exclusive peer, I/O is failed right away
  instead of blocking
- lock_timeout option is respected only by "rbd map" and not by I/O

Since 5.5:
- if the mapping is read-only, the lock isn't acquired

Added blacklisting test case, dropped lock_timeout test case.

Fixes: https://tracker.ceph.com/issues/43127
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-12-09 17:29:13 +01:00
Sage Weil
64b69572b5 Merge PR #32009 into master
* refs/pull/32009/head:
	doc: removes kube-helm installation instructions

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-12-09 08:23:44 -06:00
yangjun
3457192c24 librbd: fix rbd_open_by_id, rbd_open_by_id_read_only
These methods incorrectly delete ImageCtx on error, resulting
in double-free heap corruption.

Fixes: https://tracker.ceph.com/issues/43178
Signed-off-by: yangjun <yangjun@cmss.chinamobile.com>
2019-12-09 09:08:22 -05:00
Jos Collin
4c1029bc08
mgr/volumes: check for string values in uid/gid
chown allows strings as per bbbfb44453,
which caused this error. Eventhough uid/gid are input as CephInt, the
qa tests can pass in only string values to _fs_cmd. So try converting
the incoming uid/gid to int in create_subvolume and create_group.
It might be a valid string.

Fixes: https://tracker.ceph.com/issues/43038
Signed-off-by: Jos Collin <jcollin@redhat.com>
2019-12-09 19:25:46 +05:30
Sage Weil
4e75ee0f3a Merge PR #31800 into master
* refs/pull/31800/head:
	doc: Update commands in bootstrap.rst
	Update doc/bootstrap.rst
	Update doc/bootstrap.rst
	Merge branch 'master' into wip-doc-prompts-update
	doc: Update commands in bootstrap.rst

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-12-09 07:17:57 -06:00
Sage Weil
3036d11c60 ceph-daemon: py2: tolerate whitespace before config key name
The py2 ConfigParser doesn't like whitespace before the config option
name.  (The py3 version doesn't care.)  Filter it out before parsing.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-09 07:15:13 -06:00
Varsha Rao
f6241d1c07 doc/cephfs/disaster-recovery-experts: Add link for scrub and note for scrub_path
Fixes: https://tracker.ceph.com/issues/22204
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-12-09 18:17:41 +05:30
Jan Fajerski
1c8a4b0326
Merge pull request #32106 from taodd/ceph-volume-fix-env
ceph-volume: fix the integer overflow
2019-12-09 11:46:18 +01:00
Lenz Grimmer
713db2994d
mgr/dashboard: add smartctl data as a details tab of a host (#30594)
mgr/dashboard: add smartctl data as a details tab of a host

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-12-09 09:42:23 +00:00
Yehuda Sadeh
1bf21326aa qa: radosgw-admin: remove dependency on bunch package
Fixes: https://tracker.ceph.com/issues/43184

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-12-08 19:41:17 -08:00
Sage Weil
105428e8f4 Merge PR #29687 into master
* refs/pull/29687/head:
	test/store_test: test coverage for anti-spillover framework.
	os/bluestore: introduce bluestore_volume_selection_reserved cfg option.
	os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output
	os/bluestore: introduce 'bluestore bluefs stats' admin socket command
	os/bluestore: implement more intelligent DB volume space managment for BlueFS.
	os/kv: add static method to parse RocksDB options
	os/bluestore: introduce volume selector abstraction to BlueFS

Reviewed-by: Sage Weil <sage@redhat.com>
2019-12-08 15:24:13 -06:00
Sage Weil
d5d4868b9f Merge PR #32108 into master
* refs/pull/32108/head:
	ceph-daemon: remove prepare-host

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-12-08 13:26:32 -06:00
Sage Weil
ade391513c ceph-daemon: remove prepare-host
I thought I took this out of the PR but somehow it got merged in... must
have repushed and old branch and not realized.  :/

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-08 11:26:14 -06:00
Sage Weil
e2f8187b4c Merge PR #31959 into master
* refs/pull/31959/head:
	os/bluestore/BlueStore.cc: set priorities for compression stats

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2019-12-08 10:15:48 -06:00
Sage Weil
1542776e44 Merge PR #31995 into master
* refs/pull/31995/head:
	msg/async: No need lock for func _filter_addrs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-08 10:15:17 -06:00
Sage Weil
9634b20908 Merge PR #32101 into master
* refs/pull/32101/head:
	mgr/orchestrator_cli: make rgw rm take realm+zone
	qa/tasks/mgr/test_orchestrator_cli: fix rgw add test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-12-08 10:14:50 -06:00
Yehuda Sadeh
857fdfb8e0 rgw: list buckets: don't return buckets if limit=0
Fixes: https://tracker.ceph.com/issues/42868

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-12-08 03:08:46 -08:00
Sage Weil
99eb52b68a mgr/orchestrator_cli: make rgw rm take realm+zone
And adjust test accordingly

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-07 23:41:36 -06:00
taodd
81ff4be4d1 ceph-volume: fix the type mismatch, covert the tries and interval to int from string.
Fixes: https://tracker.ceph.com/issues/43186

Signed-off-by: dongdong tao <dongdong.tao@canonical.com>
2019-12-08 10:26:42 +08:00
Sage Weil
b9dea87ff1 mgr/diskprediction_local: import scipy early to fix self-test deadlock
We are seeing a hang on centos7 (but not ubuntu 18.04) from

        from .predictor import get_diskfailurepredictor_path

in _predict_life_expentancy.

Doing that same predictor import at the top of the file doesn't help--it
hangs right when the diskpredictor_local module is loaded.  Commenting
out the import scipy in predictor.py avoids the hang.

I'm not sure why, but doing the full scipy import here appears to work
around the problem.

Fixes: https://tracker.ceph.com/issues/42764
Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-07 15:34:59 -06:00
Sage Weil
28e9e25e93 Merge PR #32043 into master
* refs/pull/32043/head:
	os/bluestore: default bluestore_block_size 1T -> 100G

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2019-12-07 14:43:07 -06:00
Sage Weil
d73fe9cbd6 Merge PR #32044 into master
* refs/pull/32044/head:
	mon/OSDMonitor: make memory autotune disable itself if no rocksdb

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-12-07 14:42:49 -06:00