Commit Graph

104870 Commits

Author SHA1 Message Date
Casey Bodley
15aa3ad341 rgw: apply_olh_log filters out older instances of same epoch
when apply_olh_log() detects multiple instances with the same epoch, we
only promote the one that the bucket index lists as more current

Fixes: https://tracker.ceph.com/issues/39142

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-22 15:25:33 -05:00
Casey Bodley
ef039b27df rgw: factor out decode_olh_info()
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-22 15:25:32 -05:00
Sage Weil
a28e10a15a ceph-daemon: fix logrotate su line
The ceph group usually won't exist; rotate as root.root.

(If we leave this off, logrotate complains about directory ownership
and permissions and doesn't do anything.)

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-22 14:14:23 -06:00
Neha Ojha
941807d88e
Merge pull request #31657 from liewegas/fix-merge-vs-backoff
osd: release backoffs during merge

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2019-11-22 11:47:49 -08:00
Sage Weil
69e62a0e40 qa/tasks/ceph2: get ceph-daemon from same place as ceph
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-22 19:44:45 +00:00
Casey Bodley
6f52900c81 cls/rgw: only promote instance entries if they sort after current instance
to support bidirectional replication, where each zone could create its
own instances with the same olh epoch, the sort order of those instances
should be used to determine whether an entry should be promoted over the
current instance

Fixes: https://tracker.ceph.com/issues/39142

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-22 14:21:15 -05:00
Casey Bodley
a078120512 test/rgw: add test_concurrent_versioned_object_incremental_sync
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-22 14:21:15 -05:00
Ilya Dryomov
976d318b48 common/options: remove unused ms_msgr2_{sign,encrypt}_messages
These are unused since 1d29722f80 ("switch monc, daemons to use new
msgr2 auth frame exchange").  As they default to false, a confused user
might flip them to true and think that their client <-> OSD traffic is
encrypted.

The new set of options was added in c7ee66c3e5
("auth,msg/async/ProtocolV2: negotiate connection modes").

Fixes: https://tracker.ceph.com/issues/42976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-11-22 19:03:09 +01:00
Sage Weil
378fed3aa0 Merge PR #31698 into master
* refs/pull/31698/head:
	rpm,deb: add cephdaemon user and sudoers file
	ceph-daemon: use mgr/ssh command to create the ssh key
	mgr/ssh: add 'ssh generate-key' and 'ssh clear-key' commands
	mgr/ssh: add mode option
	mgr/ssh: add 'ssh get-pub-key' and 'ssh get-user' commands
	mgr/ssh: convert to command decorations
	mgr/orchestrator: move command annotation helpers to orchestrator.py

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-11-22 11:24:27 -06:00
Sage Weil
47514d5154 Merge PR #31790 into master
* refs/pull/31790/head:
	Revert "Merge pull request #16715 from adamemerson/wip-I-Object!"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-11-22 11:22:27 -06:00
Kefu Chai
93c2ccc474 test/librados: free AioCompletion using unique_ptr
always destroy AioCompletion pointers even if test fails

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-11-23 00:43:45 +08:00
Kefu Chai
91c7ecb490 librados: release pimpl pointer in destructor
before this change, the librados applications are responsible to call
`AioCompletion::release()` explicitly to release its internal pimpl
pointer. this is error prone and not intuitive.

after this change, the destructor of `AioCompletion` and
`PoolAsyncCompletion` will do this automatically. while
`AioCompletion::release()` and `PoolAsyncCompletion::release()` still
delete the instance as they did before. so this change is backward
compatible, as existing librados clients can still use `ptr->release()`
to free the completion instance, while new clients can just `delete
ptr`.

librados_test_stub is updated accordingly to match the new model

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-11-23 00:43:45 +08:00
Jan Fajerski
c1bd09f259 ceph-volume: import mock.mock instead of unittest.mock (py2)
Fixes: bb4de1a3fc
Fixes: https://tracker.ceph.com/issues/42970

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-11-22 17:27:08 +01:00
Jan Fajerski
5ae7bcab31
Merge pull request #31814 from jan--f/c-v-check-selinux-py2
ceph-volume: check if we run in an selinux environment, now also in py2
2019-11-22 17:26:45 +01:00
Sage Weil
5bb652cc2d Merge PR #31788 into master
* refs/pull/31788/head:
	ceph-daemon: append newline before public key string

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-11-22 09:33:36 -06:00
Sage Weil
54ddf2c897 Revert "Merge pull request #16715 from adamemerson/wip-I-Object!"
This reverts commit 669453138d, reversing
changes made to 36f5fcbb97.

Signed-off-by: Sage Weil <sage@redhat.com>

- conflicts due to code rearrangement in 14b0db908f
2019-11-22 09:24:25 -06:00
Jan Fajerski
12c5f89ddb ceph-volume: python2 raises OSError on Popen with missing binary.
For compatibility alias FileNotFoundError to OSError.

Fixes: 72b79f71fa
Fixes: https://tracker.ceph.com/issues/42967

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-11-22 15:51:00 +01:00
Sage Weil
2c7aa779be Merge PR #31798 into master
* refs/pull/31798/head:
	ceph-daemon: ceph-volume works without an fsid
	ceph-daemon: several commands that can infer fsids still require them
	ceph-daemon: fix fsid inference

Reviewed-by: Michael Fritch <mfritch@suse.com>
2019-11-22 08:50:52 -06:00
Jan Fajerski
114aa57df2 ceph-volume: py2 compatibility for selinux test
Fixes: af30c7f956

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-11-22 15:49:41 +01:00
Alfredo Deza
842b10180d
Merge pull request #31809 from jan--f/c-v-check-selinux
ceph-volume: check if we run in an selinux environment

Reviewed-by: Alfredo Deza <adeza@redhat.com>
2019-11-22 09:28:21 -05:00
Sage Weil
d3f51251e5 ceph-daemon: ceph-volume works without an fsid
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-22 07:44:56 -06:00
Sage Weil
80ab64b2ae ceph-daemon: several commands that can infer fsids still require them
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-22 07:44:56 -06:00
Sage Weil
82cbfbe919 ceph-daemon: fix fsid inference
All return paths should return func.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-22 07:44:56 -06:00
Willem Jan Withagen
67ffff38ed mgr: fix errors on using a reference in a Lambda function
This is the extra info on this matter:

https://bugs.llvm.org/show_bug.cgi?id=35984.

Fixes: https://tracker.ceph.com/issues/42905
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2019-11-22 14:07:31 +01:00
Tim Serong
0520ff571c mgr/PyModule: correctly remove config options
Previously, incorrect parameters were being passed to "config rm",
causing it to do nothing.  This commit also ensures the correct
error message is shown for both the set and remove failure cases.
I've also moved the update of the in-memory config map to *after*
the value is persisted, to ensure the config map actually reflects
what's stored.

Fixes: https://tracker.ceph.com/issues/42958
Signed-off-by: Tim Serong <tserong@suse.com>
2019-11-22 23:25:33 +11:00
Ilya Dryomov
6c7a23b343
Merge pull request #31771 from idryomov/wip-krbd-read-only-test
qa: update krbd_blkroset.t and add krbd_get_features.t

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
2019-11-22 12:43:13 +01:00
Xiubo Li
06dc16250e mds: remove the incorrect comments
The kclient will decode most of them since commit(e9e427f0a14f7).

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2019-11-22 05:48:17 -05:00
Jan Fajerski
72b79f71fa ceph-volume: don't assume SELinux
Expect selinux related binaries to not be present.

Fixes: 33c8a64a54
Fixes: https://tracker.ceph.com/issues/42957

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-11-22 11:37:36 +01:00
Kefu Chai
39b3743261
Merge pull request #30621 from wjwithagen/wjw-fix-freebsd-linking
cmake: use GNU linker on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-11-22 18:27:02 +08:00
Jan Fajerski
af30c7f956 ceph-volume: fix test test_selinuxenabled_doesnt_exist
Popen does not open a shell by default, so A FileNotFoundError is
raised instead of returning a 127 return code.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-11-22 11:11:47 +01:00
Sebastian Wagner
e6909311bd spec,debian: ceph-mgr-ssh depends on openssh{-client{s}}
Container images don't include ssh by default. Thus we
have to explictly depend on it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-11-22 10:32:14 +01:00
Nathan Cutler
77226cd201
Merge pull request #31777 from smithfarm/wip-mimic-v13.2.7-release-notes
doc/releases: add release notes for mimic v13.2.7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-11-22 09:13:20 +01:00
Kefu Chai
4c3127d282
Merge pull request #30387 from rzarzynski/wip-crimson-errorator
crimson: bring ceph::errorator with its first appliances

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-11-22 13:30:27 +08:00
Patrick Donnelly
2106883abb
Merge PR #31793 into master
* refs/pull/31793/head:
	mds: release free heap pages after trim

Reviewed-by: Sage Weil <sage@redhat.com>
2019-11-21 20:01:53 -08:00
Chunsong Feng
3222bbd81a msg/async/dpdk: fix icmp_hdr build error
‘struct icmp_hdr’ is defined in both IP.h and rte_icmp.h. DPDK.cc include
rte_ethdev.h that references rte_icmp.h, and include IP.h. It cause
redefinition of ‘struct icmp_hdr’ error. So move the definition of
‘struct icmp_hdr’ from IP.h to IP.cc.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
2019-11-22 11:27:12 +08:00
Yuri Weinstein
824873c4e8
Merge pull request #31799 from yuriw/wip-yuriw-crontab-master
qa/tests: trying to fix syntax error that prevented mimic-x to be add…
2019-11-21 17:44:28 -08:00
Yuri Weinstein
9d57de1e78 qa/tests: trying to fix syntax error that prevented mimic-x to be added to the schedule
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-11-21 16:43:54 -08:00
Sage Weil
a15af0a053 Merge PR #31465 into master
* refs/pull/31465/head:
	doc/install/containers: add summary of containers and branches

Reviewed-by: Dan Mick <dmick@redhat.com>
2019-11-21 16:26:04 -06:00
Sage Weil
2d4ac3859c common/blkdev: drop is_nvme() method
No more callers (and I don't think it works anyway).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 16:20:58 -06:00
Sage Weil
4515299568 os/bluestore/KernelDevice: get rid of 'nvme' type
We are either 'hdd' or 'ssd' based on the rotational flag.  Previously,
we would try to distinguish between an nvme vs SATA/SAS ssd and set the
class to 'nvme'.  This was misguided--the interface is not important and
has no direct bearing on the device performance.  Moreover, the HDD
manufacturers are planning to produce rotation HDDs that use the nvme
interface instead of SATA/SAS.

So, drop this.

This may be somewhat disruptive to clusters where we were detecting
nvme but now are not.  However, the good news is that this doesn't seem
to trigger for real deployments because LVM breaks the is_nvme()
method.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 16:19:17 -06:00
Sage Weil
7948f5c3ef Merge PR #31702 into master
* refs/pull/31702/head:
	ceph-daemon: infer fsid for shell, enter, ceph-volume, unit, logs

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-11-21 15:41:56 -06:00
Sage Weil
96e9b8bc29 qa/tasks/ceph2: use safe_while
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 15:35:36 -06:00
Sage Weil
660233935e rpm,deb: add cephdaemon user and sudoers file
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 15:34:07 -06:00
Nathan Cutler
96bea8a174 doc/releases: add release notes for mimic v13.2.7
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-11-21 22:07:02 +01:00
Casey Bodley
8aa638b0fa
Merge pull request #30960 from yuvalif/wip-yuval-add-kafka-notif-endpoint
rgw/pubsub: add kafka notification endpoint

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-11-21 14:42:23 -05:00
Sage Weil
4bfcb5db6b ceph-daemon: use mgr/ssh command to create the ssh key
The mgr/ssh module has a command now to do this; use that instead of
generating the key ourselves and manually injecting it.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 12:57:13 -06:00
Sage Weil
4c9c01098b mgr/ssh: add 'ssh generate-key' and 'ssh clear-key' commands
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 12:57:13 -06:00
Patrick Donnelly
af4cac5ec7
mds: release free heap pages after trim
MDS free heap space can grow to large for some workloads (like smallfile
and recursive deletes). This can cause the MDS mapped memory to grow
well beyond memory targets.

When we finally use the PriorityCache in the MDS, this will not be
necessary anymore as the PriorityCache already does this.

Fixes: https://tracker.ceph.com/issues/42938
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-11-21 10:54:34 -08:00
Sage Weil
822f8ed2f5 mgr/ssh: add mode option
Set the mgr/ssh/mode option to ceph-daemon-package to switch to a mode
where we assume ceph-daemon is installed on the remote host and we can
ssh as user cephdaemon.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 12:41:02 -06:00
Sage Weil
9ba5225f6f mgr/ssh: add 'ssh get-pub-key' and 'ssh get-user' commands
Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-21 12:40:53 -06:00