Commit Graph

107712 Commits

Author SHA1 Message Date
Sage Weil
6aedf7e8c1 Merge PR #33248 into master
* refs/pull/33248/head:
	cephadm: fix deploy crash when no `args.fsid`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2020-02-13 13:19:00 -06:00
J. Eric Ivancich
2a73229370
Merge pull request #31088 from joke-lee/rgw-storage-class-apend-obj
rgw: set correct storage class for append

Reviewed-By: Casey Bodley <cbodley@redhat.com>
2020-02-13 14:17:26 -05:00
J. Eric Ivancich
c490a1928b
Merge pull request #31661 from joke-lee/sts-args
rgw: sts: add all http args to req_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-13 14:16:21 -05:00
J. Eric Ivancich
43f1838c0a
Merge pull request #33053 from tpsilva/add_max_connections
rgw: make max_connections configurable in beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-02-13 14:13:53 -05:00
J. Eric Ivancich
63faa3e4fb
Merge pull request #33165 from clwluvw/policy-subuser
rgw: Add support bucket policy for subuser

Reviewed-By: Adam Emersen <aemerson@redhat.com>
2020-02-13 14:11:00 -05:00
J. Eric Ivancich
d5788d66a8
Merge pull request #32636 from ivancich/wip-avoid-0-length-ordered-list-result
rgw: address 0-length listing results when non-vis entries dominate

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2020-02-13 14:04:05 -05:00
Nathan Cutler
6dbe449efe
Merge pull request #32805 from smithfarm/wip-cmake-logging
rpm: add cmake_verbose_logging switch

Reviewed-by: David Disseldorp <ddiss@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-02-13 20:00:27 +01:00
Sage Weil
6846894670 Merge PR #29427 into master
* refs/pull/29427/head:
	mgr/rook: Make use of rook-client-python when talking to Rook
	cmake: Integrate Rook client generation
	mgr/rook: Automatically generate Rook client interface
	Add submodule to rook-client-python.git

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
2020-02-13 10:26:09 -06:00
Ali Maredia
1de3145af0
Merge pull request #32609 from joke-lee/rgw-website-43563
rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same …
2020-02-13 10:30:35 -05:00
Kefu Chai
009ed95195
Merge pull request #33247 from tchaikov/wip-cmake-cleanup
cmake: do not link against unused libs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-02-13 23:06:19 +08:00
Mykola Golub
78d664192e
Merge pull request #33219 from dillaman/wip-44066-2
rbd: fixed additional issues with CEPH_ARGS processing

Reviewed-by: Mykola Golub <mgolub@suse.com>
2020-02-13 16:20:52 +02:00
Nathan Cutler
5e98d22380 rpm: add cmake_verbose_logging switch
This switch sets CMAKE_VERBOSE_MAKEFILE and causes the CMakeOutput.log
and CMakeError.log files to be dumped after cmake execution finishes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2020-02-13 15:13:24 +01:00
Nathan Cutler
85e4913f48
Merge pull request #33182 from smithfarm/wip-cbs-basic-auth
ceph-backport.sh: use Basic Authentication instead of access_token

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-02-13 14:43:56 +01:00
Nathan Cutler
e36d47a8c3
Merge pull request #33173 from smithfarm/wip-bri-basic-auth
backport-resolve-issue: use Basic Authentication instead of "access_token"

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-02-13 14:00:58 +01:00
Nathan Cutler
fef7efeb8b
Merge pull request #33105 from smithfarm/wip-44014
backport-resolve-issue: populate tracker_description method

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-02-13 14:00:00 +01:00
Kefu Chai
12e738f5a1
Merge pull request #32693 from CongMinYin/update-pmdk-version
cmake: Update pmdk version to 1.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-02-13 19:24:05 +08:00
Yin Congmin
6a102677e7 cmake: Update pmdk version to 1.7
nvml is renamed to pmdk and upgrading to 1.7 to fix gcc9 build issue

Signed-off-by: Yin, Congmin <congmin.yin@intel.com>
2020-02-14 01:50:07 +08:00
Sebastian Wagner
846761ef7a mgr/rook: Make use of rook-client-python when talking to Rook
Fixes:

* `CephFilesystem.spec.onlyManageDaemons` does not exist
* `CephObjectStroe.spec.gateway.allNodes` does not exist
* Added directory-osds to existsing nodes was broken

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:34:50 +01:00
Sebastian Wagner
615381659c cmake: Integrate Rook client generation
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
02359b3b51 mgr/rook: Automatically generate Rook client interface
using the official Rook client Python library.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
26424d3f0c Add submodule to rook-client-python.git
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:33:55 +01:00
Sebastian Wagner
74125c74dd
Merge pull request #33236 from sebastian-philipp/python-common-py.typed
python-common: add py.typed (PEP 561)

Reviewed-by: Sage Weil <sage@redhat.com>
2020-02-13 11:32:46 +01:00
Sebastian Wagner
bd7f025ec2
Merge pull request #33212 from liewegas/wip-orch-newcli
mgr/orch: first phase of new cli

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-13 11:31:37 +01:00
Kefu Chai
8bb9c63861
Merge pull request #33250 from tchaikov/wip-seastar
crimson: pickup change to fix --cpuset support and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-02-13 18:08:52 +08:00
Kefu Chai
12da789dc9
Merge pull request #32647 from optimistyzy/115
NVMEDevice: Split the read I/O if the io size is large.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-02-13 15:35:19 +08:00
Jan Fajerski
fec03c5639
Merge pull request #33246 from jan--f/c-v-batch-fix-has_common_vg
ceph-volume: use get_device_vgs in has_common_vg
2020-02-13 08:00:47 +01:00
Jan Fajerski
5b5002d8c4
Merge pull request #33223 from dsavineau/issue_44069
ceph-volume: fix is_ceph_device for lvm batch
2020-02-13 08:00:30 +01:00
Kefu Chai
36963dd7cc cmake: disable -Wnon-virtual-dtor when compiling seastar
quite a few base classes with virtual functions mark their destructor
non-virtual and `protected` for better performance, as seastar destruct
them via the concrete type of the instance.

so let's disable this warning. but, please note, this newly added
CXX_FLAG in `Seastar_CXX_FLAGS` won't be populated to crimson, as it is
only added to the CXX_FLAGS used for compiling seastar itself. so we still
have `-Wnon-virtual-dtor` warnings when compiling crimson as long as seastar
headers are included.

so to silence these warnings, we need to add it also to `crimson::cflags`,
probably it's worth trading the noise caused by seastar's optimizations
with the potentially useful warning messages caused by our oversights.
in my case, there are over 300 lines of warnings split by GCC-10, so i
still think it'd be better to add it also to crimson to increase the
signal-to-noise ratio. we can aways remove it every once in a while to
check if we forget to mark the destructor of a base class `virtual`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
Kefu Chai
5f05a50bae crimson/thread: s/Condition/seastar::readable_eventfd/
in the latest version of seastar, we are not able to construct a
`seastar::pollable_fd_state` directly, as its constructor is now
`protected`, and only the reactor is able to create an instance of
`seastar::pollable_fd_state` now.

and `seastar::readable_eventfd` offers all we need to get notified
by reactor in an alien world. so let's used it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
Kefu Chai
71c0c491da seastar: pickup change to fix --cpuset support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 14:53:55 +08:00
J. Eric Ivancich
4470ca8a3f rgw: clean up address 0-length listing results...
Some minor clean-ups to the previous commit, including adjust logging
messages, rename variable, convert a #define to a constexpr (and
adjust its scope).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2020-02-12 22:34:59 -05:00
J. Eric Ivancich
28bd8bada4 rgw: address 0-length listing results when non-vis entries dominate
A change to advance the marker in RGWRados::cls_bucket_list_ordered to
the last entry visited rather than the final entry in list to push
progress as far as possible.

Since non-vis entries tend to cluster on the same shard, such as
during incomplete multipart uploads, this can severely limit the
number of entries returned by a call to
RGWRados::cls_bucket_list_ordered since once that shard has provided
all its members, we must stop. This interacts with a recent
optimization to reduce the number of entries requested from each
shard. To address this the number of attempts is sent as a parameter,
so the number of entries requested from each shard can grow with each
attempt. Currently the growth is linear but perhaps exponential growth
(capped at number of entries requested) should be considered.

Previously RGWRados::Bucket::List::list_objects_ordered was capped at
2 attempts, but now we keep attempting to insure we make forward
progress and return entries when some exist. If we fail to make
forward progress, we log the error condition and stop looping.

Additional logging, mostly at level 20, is added to the two key
functions involved in ordered bucket listing to make it easier to
follow the logic and address potential future issues that might arise.

Additionally modify attempt number based on how many results were
received.

Change the per-shard request number, so it grows exponentially rather
than linearly as the attempts go up.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2020-02-12 20:38:44 -05:00
Jason Dillaman
a3a11a21cc rbd-mirror: prevent asok commands from dereferencing uninitialized members
If rbd-mirror fails to connect to the remote cluster, there is a window of time
where the asok commands might attempt to dereference the default namespace
replayer or access invalid librados IoCtxs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:56 -05:00
Jason Dillaman
1cb9e3b569 rbd: don't treat failure to gather mirror service daemons as a failure
The only negative result is that instance ids cannot be cross-referenced
to the matching rbd-mirror daemon service.

Fixes: https://tracker.ceph.com/issues/44066
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:56 -05:00
Jason Dillaman
41357d242b qa/workunits/rbd: added mgr 'profile rbd' caps to mirror user
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:22:30 -05:00
Jason Dillaman
14001eb17e qa/workunits/rbd: additional overrides for CEPH_ARGS
The 'ceph' CLI and 'rbd mirror pool/image status' commandsshould revert
to use the admin user so that it has proper credentials for the cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-02-12 20:21:48 -05:00
Sage Weil
81bad666e0 Merge PR #33216 into master
* refs/pull/33216/head:
	cephadm: avoid trigger old podman bug
	qa/workunits/cephadm/test_cephadm.sh: wait longer for grafana

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-02-12 19:09:05 -06:00
Sage Weil
92b49094e7 cephadm: avoid trigger old podman bug
This ticket seems to suggest that (1) the root cause is related to an
exec that is orphaned and screws up the container state (due to, e.g., ssh
dropping, or a timeout), (2) -f may be needed, sometimes, to recover, and
(3) newer versions fix it.

  https://github.com/containers/libpod/issues/3226

Way back in 26f9fe54cb we found that using
-f the first time around was a Bad Idea, so we'd rather avoid this.

Instead, just avoid triggering the bug.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 13:59:23 -06:00
Sage Weil
1f5e59d294 qa/workunits/cephadm/test_cephadm.sh: wait longer for grafana
Sometimes it is very slow to start (only on 18.04, strangely?).

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 13:59:23 -06:00
J. Eric Ivancich
210fb41806
Merge pull request #32562 from IlsooByun/fix_dm
rgw: do not remove delete marker when fixing versioned bucket
2020-02-12 14:47:44 -05:00
J. Eric Ivancich
59b3ccca7a
Merge pull request #30611 from ivancich/wip-make-reshard-status-enum-class
rgw: change cls rgw reshard status to enum class

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-12 14:46:23 -05:00
Michael Fritch
4d90d667b3
cephadm: fix deploy crash when no args.fsid
broken by 724199dd87 when the
crash agent was made to behave like all other services

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-02-12 11:51:34 -07:00
Sage Weil
b94673a03b mgr/orch: add 'orch service rm'
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
e5c264fc2d mgr/orch,cephadm: orch daemon rm ...
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
aacc9a650f mgr/orch: service ls -> ps, add DaemonDescription
- We keep ServiceDescription around unmodified (although it will need some
  cleanup later)
- We add DaemonDescription, and clean out the service-related ambiguities
- Add a new list_daemons() method for Orchestrator
- Add a new 'ceph orch ps' command
- In cephadm, drop get_services(), and implement list_daemons()
  - a million changes to make this work
- Adjust health alert and option names

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
4f4e043b3d vstart.sh: ceph orchestrator -> ceph orch
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
95ecaad0d5 mgr/orch: split out daemon_action from service_action
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-12 11:14:17 -06:00
Sage Weil
8817a232b7 Merge PR #33205 into master
* refs/pull/33205/head:
	mgr/cephadm: Bail if we cannot find a host for services
	mgr/cephadm: fix placement of new daemons (mds,rgw,rbd-m)
	mgr/orchestrator: minor change to improve type checking
	mgr/cephadm: test_cephadm: simplify matching strings

Reviewed-by: Sage Weil <sage@redhat.com>
2020-02-12 11:13:41 -06:00
zdover23
2b6819f0c9
Merge pull request #32587 from zdover23/wip-doc-nautilus-osd-fix-10Jan2020
doc: emphasize the importance of require-osd-release nautilus
2020-02-13 02:59:28 +10:00
Kefu Chai
a8dea3a8cf test: do not include unused headers
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-13 00:22:29 +08:00