Commit Graph

107753 Commits

Author SHA1 Message Date
Nathan Cutler
fef7efeb8b
Merge pull request #33105 from smithfarm/wip-44014
backport-resolve-issue: populate tracker_description method

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-02-13 14:00:00 +01:00
nagpavan-chilakam
3ddac762f8 doc/architecture.rst: fix a typo in EC section
In erasure coding section under Architecture, there is a mention of k = 2 + M =1 for
number of data copies and redundancy copies respectively, which is a bit ambiguous.
The proposal is to change to k = 2, M = 1 as the + sign is not needed here.

Signed-off-by: Nag Pavan Chilakam <nagpavan.chilakam@gmail.com>
2020-02-13 20:01:37 +08:00
Kefu Chai
12e738f5a1
Merge pull request #32693 from CongMinYin/update-pmdk-version
cmake: Update pmdk version to 1.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-02-13 19:24:05 +08:00
Yin Congmin
6a102677e7 cmake: Update pmdk version to 1.7
nvml is renamed to pmdk and upgrading to 1.7 to fix gcc9 build issue

Signed-off-by: Yin, Congmin <congmin.yin@intel.com>
2020-02-14 01:50:07 +08:00
Sebastian Wagner
846761ef7a mgr/rook: Make use of rook-client-python when talking to Rook
Fixes:

* `CephFilesystem.spec.onlyManageDaemons` does not exist
* `CephObjectStroe.spec.gateway.allNodes` does not exist
* Added directory-osds to existsing nodes was broken

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:34:50 +01:00
Sebastian Wagner
615381659c cmake: Integrate Rook client generation
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
02359b3b51 mgr/rook: Automatically generate Rook client interface
using the official Rook client Python library.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
26424d3f0c Add submodule to rook-client-python.git
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
74125c74dd
Merge pull request #33236 from sebastian-philipp/python-common-py.typed
python-common: add py.typed (PEP 561)

Reviewed-by: Sage Weil <sage@redhat.com>
2020-02-13 11:32:46 +01:00
Sebastian Wagner
bd7f025ec2
Merge pull request #33212 from liewegas/wip-orch-newcli
mgr/orch: first phase of new cli

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:31:37 +01:00
Kefu Chai
8bb9c63861
Merge pull request #33250 from tchaikov/wip-seastar
crimson: pickup change to fix --cpuset support and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-02-13 18:08:52 +08:00
Kefu Chai
12da789dc9
Merge pull request #32647 from optimistyzy/115
NVMEDevice: Split the read I/O if the io size is large.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-02-13 15:35:19 +08:00
Jan Fajerski
fec03c5639
Merge pull request #33246 from jan--f/c-v-batch-fix-has_common_vg
ceph-volume: use get_device_vgs in has_common_vg
2020-02-13 08:00:47 +01:00
Jan Fajerski
5b5002d8c4
Merge pull request #33223 from dsavineau/issue_44069
ceph-volume: fix is_ceph_device for lvm batch
2020-02-13 08:00:30 +01:00
Kefu Chai
36963dd7cc cmake: disable -Wnon-virtual-dtor when compiling seastar
quite a few base classes with virtual functions mark their destructor
non-virtual and `protected` for better performance, as seastar destruct
them via the concrete type of the instance.

so let's disable this warning. but, please note, this newly added
CXX_FLAG in `Seastar_CXX_FLAGS` won't be populated to crimson, as it is
only added to the CXX_FLAGS used for compiling seastar itself. so we still
have `-Wnon-virtual-dtor` warnings when compiling crimson as long as seastar
headers are included.

so to silence these warnings, we need to add it also to `crimson::cflags`,
probably it's worth trading the noise caused by seastar's optimizations
with the potentially useful warning messages caused by our oversights.
in my case, there are over 300 lines of warnings split by GCC-10, so i
still think it'd be better to add it also to crimson to increase the
signal-to-noise ratio. we can aways remove it every once in a while to
check if we forget to mark the destructor of a base class `virtual`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
Kefu Chai
5f05a50bae crimson/thread: s/Condition/seastar::readable_eventfd/
in the latest version of seastar, we are not able to construct a
`seastar::pollable_fd_state` directly, as its constructor is now
`protected`, and only the reactor is able to create an instance of
`seastar::pollable_fd_state` now.

and `seastar::readable_eventfd` offers all we need to get notified
by reactor in an alien world. so let's used it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
Kefu Chai
71c0c491da seastar: pickup change to fix --cpuset support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
J. Eric Ivancich
4470ca8a3f rgw: clean up address 0-length listing results...
Some minor clean-ups to the previous commit, including adjust logging
messages, rename variable, convert a #define to a constexpr (and
adjust its scope).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2020-02-12 22:34:59 -05:00
J. Eric Ivancich
28bd8bada4 rgw: address 0-length listing results when non-vis entries dominate
A change to advance the marker in RGWRados::cls_bucket_list_ordered to
the last entry visited rather than the final entry in list to push
progress as far as possible.

Since non-vis entries tend to cluster on the same shard, such as
during incomplete multipart uploads, this can severely limit the
number of entries returned by a call to
RGWRados::cls_bucket_list_ordered since once that shard has provided
all its members, we must stop. This interacts with a recent
optimization to reduce the number of entries requested from each
shard. To address this the number of attempts is sent as a parameter,
so the number of entries requested from each shard can grow with each
attempt. Currently the growth is linear but perhaps exponential growth
(capped at number of entries requested) should be considered.

Previously RGWRados::Bucket::List::list_objects_ordered was capped at
2 attempts, but now we keep attempting to insure we make forward
progress and return entries when some exist. If we fail to make
forward progress, we log the error condition and stop looping.

Additional logging, mostly at level 20, is added to the two key
functions involved in ordered bucket listing to make it easier to
follow the logic and address potential future issues that might arise.

Additionally modify attempt number based on how many results were
received.

Change the per-shard request number, so it grows exponentially rather
than linearly as the attempts go up.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2020-02-12 20:38:44 -05:00
Jason Dillaman
a3a11a21cc rbd-mirror: prevent asok commands from dereferencing uninitialized members
If rbd-mirror fails to connect to the remote cluster, there is a window of time
where the asok commands might attempt to dereference the default namespace
replayer or access invalid librados IoCtxs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:56 -05:00
Jason Dillaman
1cb9e3b569 rbd: don't treat failure to gather mirror service daemons as a failure
The only negative result is that instance ids cannot be cross-referenced
to the matching rbd-mirror daemon service.

Fixes: https://tracker.ceph.com/issues/44066
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:56 -05:00
Jason Dillaman
41357d242b qa/workunits/rbd: added mgr 'profile rbd' caps to mirror user
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:30 -05:00
Jason Dillaman
14001eb17e qa/workunits/rbd: additional overrides for CEPH_ARGS
The 'ceph' CLI and 'rbd mirror pool/image status' commandsshould revert
to use the admin user so that it has proper credentials for the cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:21:48 -05:00
Sage Weil
81bad666e0 Merge PR #33216 into master
* refs/pull/33216/head:
	cephadm: avoid trigger old podman bug
	qa/workunits/cephadm/test_cephadm.sh: wait longer for grafana

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-02-12 19:09:05 -06:00
Sage Weil
92b49094e7 cephadm: avoid trigger old podman bug
This ticket seems to suggest that (1) the root cause is related to an
exec that is orphaned and screws up the container state (due to, e.g., ssh
dropping, or a timeout), (2) -f may be needed, sometimes, to recover, and
(3) newer versions fix it.

  https://github.com/containers/libpod/issues/3226

Way back in 26f9fe54cb we found that using
-f the first time around was a Bad Idea, so we'd rather avoid this.

Instead, just avoid triggering the bug.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 13:59:23 -06:00
Sage Weil
1f5e59d294 qa/workunits/cephadm/test_cephadm.sh: wait longer for grafana
Sometimes it is very slow to start (only on 18.04, strangely?).

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 13:59:23 -06:00
J. Eric Ivancich
210fb41806
Merge pull request #32562 from IlsooByun/fix_dm
rgw: do not remove delete marker when fixing versioned bucket
2020-02-12 14:47:44 -05:00
J. Eric Ivancich
59b3ccca7a
Merge pull request #30611 from ivancich/wip-make-reshard-status-enum-class
rgw: change cls rgw reshard status to enum class

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-12 14:46:23 -05:00
Michael Fritch
4d90d667b3
cephadm: fix deploy crash when no args.fsid
broken by 724199dd87 when the
crash agent was made to behave like all other services

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-02-12 11:51:34 -07:00
Sage Weil
b94673a03b mgr/orch: add 'orch service rm'
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
e5c264fc2d mgr/orch,cephadm: orch daemon rm ...
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
aacc9a650f mgr/orch: service ls -> ps, add DaemonDescription
- We keep ServiceDescription around unmodified (although it will need some
  cleanup later)
- We add DaemonDescription, and clean out the service-related ambiguities
- Add a new list_daemons() method for Orchestrator
- Add a new 'ceph orch ps' command
- In cephadm, drop get_services(), and implement list_daemons()
  - a million changes to make this work
- Adjust health alert and option names

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
4f4e043b3d vstart.sh: ceph orchestrator -> ceph orch
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
95ecaad0d5 mgr/orch: split out daemon_action from service_action
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
8817a232b7 Merge PR #33205 into master
* refs/pull/33205/head:
	mgr/cephadm: Bail if we cannot find a host for services
	mgr/cephadm: fix placement of new daemons (mds,rgw,rbd-m)
	mgr/orchestrator: minor change to improve type checking
	mgr/cephadm: test_cephadm: simplify matching strings

Reviewed-by: Sage Weil <sage@redhat.com>
2020-02-12 11:13:41 -06:00
zdover23
2b6819f0c9
Merge pull request #32587 from zdover23/wip-doc-nautilus-osd-fix-10Jan2020
doc: emphasize the importance of require-osd-release nautilus
2020-02-13 02:59:28 +10:00
Kefu Chai
a8dea3a8cf test: do not include unused headers
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 00:22:29 +08:00
Kefu Chai
103050c1b6 cmake: do not link against unused libs
if it's not used, there is no need to link against it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 00:22:28 +08:00
Seena Fallah
e34bc060d4 rgw: Add test for bucket policy subuser
Signed-off-by: Seena Fallah <seenafallah@gmail.com>
2020-02-12 19:34:18 +03:30
Jan Fajerski
60d80636e4 ceph-volume: add is_ceph_device unit tests
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2020-02-12 16:52:04 +01:00
Dimitri Savineau
a82582364c ceph-volume: fix is_ceph_device for lvm batch
This is a regression introduced by 634a709

The lvm batch command fails to prepare the OSDs on the created LV.
When using lvm batch, the LV/VG are created prior the OSD prepare.
During that creation, multiple tags are set with null value.

$ lvs -o lv_tags --noheadings
  ceph.cluster_fsid=null,ceph.osd_fsid=null,ceph.osd_id=null,ceph.type=null

Since we call is_ceph_device which returns True if the ceph.osd_id LVM
tag exists but doesn't test the value then we raise an execption.

When the tag value is set to 'null' then we can consider that the device
isn't part of the ceph cluster (because not yet prepared).

Closes: https://tracker.ceph.com/issues/44069

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-02-12 10:44:00 -05:00
Sebastian Wagner
19122bba3d
Merge pull request #32879 from shyukri/wip-42769-orch-log-handling-ench
mgr/orch: logging - handle lists output

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-12 16:32:34 +01:00
Jan Fajerski
2c5a8c3b40 ceph-volume: use get_device_vgs in has_common_vg
Fixes: https://tracker.ceph.com/issues/44099

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2020-02-12 16:03:58 +01:00
Sage Weil
070df40773 Merge PR #33229 into master
* refs/pull/33229/head:
	rpm: define weak_deps for el8

Reviewed-by: Sage Weil <sage@redhat.com>
2020-02-12 08:28:51 -06:00
Casey Bodley
1974b33285
Merge pull request #33230 from Rjerk/fix-multisite-doc-typo
doc/radosgw/multisite-sync-policy.rst: fix typo

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-02-12 08:48:31 -05:00
Nathan Cutler
0bd69b6500 backport-resolve-issue: address some linter complaints
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2020-02-12 14:01:59 +01:00
Nathan Cutler
9ea48e8f7e backport-resolve-issue: extract PR number more carefully
This addresses cases where the previous regex '\\d++' was promiscuously
matching strings of digits that were not GitHub PR IDs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2020-02-12 13:41:53 +01:00
Nathan Cutler
5959ac25c7 backport-resolve-issue: use Basic Authentication
Address "Deprecation notice for authentication via URL query parameters"
and other GitHub API behavior changes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2020-02-12 13:41:49 +01:00
Sebastian Wagner
8972080080 python-common: add py.typed (PEP 561)
Bugs found:

* Fixed documentation of how `mgr/Orchestrator.create_osds` is called
* mgr/Rook.create_osds: Added missing `.path` when querying paths.
* mgr/Rook.create_osds: Fixed progress message
* mgr/RookCluster.create_osds: Empty list instead of `None`
* python-common: use empty objects instead of `None`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-12 11:34:40 +01:00
Kefu Chai
b3783bf517
Merge pull request #33233 from tchaikov/wip-crimson-misc
crimson: fix build with GCC-10

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-02-12 18:02:31 +08:00