Commit Graph

110914 Commits

Author SHA1 Message Date
Xiubo Li
c541bae6b1 qa/vstart_runner: remove the debug info
Fixes: https://tracker.ceph.com/issues/45175
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-23 20:55:35 -04:00
Matthew Oliver
d3de8697e6 cephadm: Check for pool existance for iscsi And NFS
Currently both iscsi and NFS require pools to be specified when they are
deployed. However, we don't actaully check these pools exist. Leading to
broken containers.

This patch uses the rados client that is part of the mgrmodule to check
that the specified pool exists. As we need to check in 2 different
daemons, a helper method:

   def _check_pool_exists(self, pool, service_name):

was added to `cephadm/module.py`.

Fixes: https://tracker.ceph.com/issues/45161
Signed-off-by: Matthew Oliver <moliver@suse.com>
2020-04-24 00:44:54 +00:00
Michael Fritch
b454b995cb
cephadm: catch Exception during is_mgr_available
Traceback (most recent call last):
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 4578, in <module>
    r = args.func()
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 1121, in _default_image
    return func()
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 2489, in command_bootstrap
    is_available('mgr', is_mgr_available)
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 895, in is_available
    if func():
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 2486, in is_mgr_available
    out = cli(['status', '-f', 'json-pretty'], timeout=timeout)
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 2403, in cli
    return CephContainer(
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 2140, in run
    out, _, _ = call_throws(
  File "test_tmp/tmp.ykMmPiFqIT/cephadm", line 838, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-04-23 15:12:56 -06:00
Neha Ojha
7c8b627eaa qa/*/osd-scrub-repair.sh: don't fail if PG is in active+clean+wait
a0b453ad33 added the wait state, which can
make PGs stay in active+clean+wait for a while instead of going into
active+clean directly. As far as TEST_auto_repair_bluestore_failed is
concerned, we only care about the repair state being cleared.

Fixes: https://tracker.ceph.com/issues/45075
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-23 20:24:28 +00:00
Igor Fedotov
e792e19331 os/bluestore: invoke _prepare_ondisk_format_super as the last op
Fixes: https://tracker.ceph.com/issues/45133

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-23 22:37:54 +03:00
Jason Dillaman
fbede28d1a rbd-mirror: skip snapshot image sync if mirror snapshot is marked clean
This is currently only utilized for the case where a newly created image
has mirroring enabled at time of creation, but it could be expanded in the
future if we track writes.

Fixes: https://tracker.ceph.com/issues/44596
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
ea64ba5a6f librbd: EnableRequest now accepts a boolean to indicate a clean image
If the image is clean, it's treated as if it was newly created and
therefore clean since snapshot id 0. The CreateRequest and
CloneRequest state machines pass true for this bool if mirroring
is being enabled during creation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
959d5e60d8 librbd: pass clean_since_snap_id to CreatePrimaryRequest
This will be stored in the primary MirrorSnapshotNamespace for use by
the rbd-mirror snapshot replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
50702eece0 cls/rbd: added clean_since_snap_id to MirrorSnapshotNamespace
It is only valid for primary snapshots so we can re-use the space
previously used by the primary_snap_id without the need to bump the
version id.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
6da56d7982 rbd: add '--mirror-image-mode' optional to image creation actions
This allows the user to enable mirroring while creating, cloning,
copying, importing, or migrating an image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
25c034f0a0 librbd: save requested mirror image mode during migration prepare
Once the migration is committed, the requested mode will be applied.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
407f2e5c17 librbd: permit mirroring to be enabled upon image cloning
The librbd clone API's ImageOptions parameter now supports passing
the mirror image mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
a90042b254 librbd: add non-primary global image id to mirror enable
The constructor that took an ImageCtx did not support passing the
non-primary global image id and instead defaulted to an empty string.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
37f6fa8fe9 librbd: permit mirroring to be enabled upon image creation
The librbd create API's ImageOptions parameter now supports passing
the mirror image mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:49 -04:00
Jason Dillaman
931c9f0fd9 librbd: mirror enable state machine might need to open image
If attempting to create a snapshot-based mirroring primary snapshot,
the image needs to first be opened. If we weren't supplied an image,
open the image, create the snapshot, and close the image again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 13:19:48 -04:00
Casey Bodley
ce5f3e3047
Merge pull request #34570 from cbodley/wip-bucket-layout-json
rgw: bucket layout fixes

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2020-04-23 11:16:55 -04:00
Casey Bodley
f43e58071e
Merge pull request #34645 from cbodley/wip-rgw-reshard-target-layout
rgw: use target bucket layout for reshard

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2020-04-23 11:16:44 -04:00
Jason Dillaman
92b3122a06
Merge pull request #33937 from changchengx/unsigned_compare
librbd: fix unsigned value compared with zero

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-04-23 10:48:33 -04:00
Boris Ranto
05573aa57d
Merge pull request #34694 from mikechristie/ceph-iscsi-selinux-fixes
ceph-iscsi:  selinux fixes

Reviewed-by: Boris Ranto <branto@redhat.com>
2020-04-23 13:52:36 +02:00
Sebastian Wagner
9f13c126f6 doc/man: add cephadm
Fixes: https://tracker.ceph.com/issues/44971

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 13:01:02 +02:00
Sebastian Wagner
bf395f253e mgr/cephadm: unify apply tests
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 12:39:32 +02:00
Sebastian Wagner
8eaadfe612 mgr/cephadm: unified add_daemon tests
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 12:39:32 +02:00
Sebastian Wagner
61393da911 mgr/cephadm: add cleanup to test_cephadm
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 12:39:32 +02:00
Sebastian Wagner
33b6c501f1 mgr/cephadm: remove_service: Feedback, if spec not found
`ceph orch rm my-unknown-service` still returned
`Removed service my-unknown-service`.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 12:26:04 +02:00
Sebastian Wagner
9e91081f06
Merge pull request #34678 from sebastian-philipp/doc-cephadm-install-fix-placement
doc/cephadm: install.rst: fix placement spec.

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-04-23 12:20:30 +02:00
Kiefer Chang
00881adf55 mgr/dashboard: fixing RBD purge error in backend
Current UTC datetime should be used as `now`, otherwise the expiration
comparison might fail on machines with non-UTC localtime.

Fixes: https://tracker.ceph.com/issues/45149
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2020-04-23 18:18:27 +08:00
Sebastian Wagner
6a588788b7
Merge pull request #34674 from sebastian-philipp/cephadm-ceph-volume-lock-if-fsid
cephadm: Aquire lock, if fsid != None

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-04-23 12:09:22 +02:00
Sebastian Wagner
aab7045dec
Merge pull request #34620 from mgfritch/cephadm-timeout-test-fixup
qa/workunits/cephadm/test_cephadm.sh: extend timeout test
2020-04-23 12:03:20 +02:00
Sebastian Wagner
3784f63551
Merge pull request #34600 from dsavineau/issue_45120
cephadm: fix adopt prometheus command

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-04-23 12:02:31 +02:00
Sebastian Wagner
8dcb67de84
Merge pull request #34688 from sebastian-philipp/qa-test-cephadm-octopus-version
qa/workunits/cephadm/test_cephadm: Also test octopus

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-04-23 12:01:48 +02:00
Sebastian Wagner
bf2988ab10
Merge pull request #34691 from sebastian-philipp/cephadm-ssk-keyring-check_call
mgr/cephadm: use check_call  for `ssh-keygen`

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-04-23 12:01:20 +02:00
Sebastian Wagner
c681fbb41d mgr/cephadm: raise, if crash keyring fails
Fixes: https://tracker.ceph.com/issues/44826
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-23 11:13:21 +02:00
Igor Fedotov
70839d99dd
Merge pull request #33065 from ifed01/wip-ifed-spurious-readerr-alert
os/bluestore: introduce health alert on spurious read errors.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-23 11:32:25 +03:00
Laura Paduano
1a616c207c
Merge pull request #34666 from rhcs-dashboard/44405-test-totp-increase-time
mgr/dashboard: fix tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-04-23 09:47:58 +02:00
Kefu Chai
0c2f0bde8a
Merge pull request #34483 from dabukalam/master
doc/rados: Remove obsolete placement group note from EC pool creation

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-23 15:33:06 +08:00
Jasper Spaans
8d0da4ea9c
osd: Make the _day operator"" internals big-endian.
A rather trivial change inspired by PR #34702, where an instance of
7*60*60*24 (used to describe the number of seconds in a week) was fixed,
decided to have a look at the code and cleaned a similar case of weird-endian-ness.

Signed-off-by: Jasper Spaans <jasper@startmail.com>
2020-04-23 09:30:34 +02:00
Kefu Chai
5a7fb2c2e8
Merge pull request #33916 from aclamk/fix-bs-avl-memaccounting
os/bluestore: Adds shutdown in destructor of AvlAllocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-23 15:22:19 +08:00
Kefu Chai
e6fe7f44af
Merge pull request #34559 from tchaikov/wip-19076-refactor
mon/OSDMonitor: refactor loops for better readability

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2020-04-23 10:34:26 +08:00
Josh Durgin
d61f57d078
Merge pull request #34328 from liewegas/cpatch
script/cpatch: initial version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2020-04-22 18:34:36 -07:00
Jason Dillaman
e9130bcf0b
Merge pull request #34430 from yison/rwl-image-writeback-cache-seq4
rbd/cache: Replicated Write Log core codes part 4

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-04-22 19:28:17 -04:00
Michael Fritch
4e73255ffa
mgr/cephadm: allow config for an iscsi container_image
ceph config set client.iscsi container_image <container_image>

Fixes: https://tracker.ceph.com/issues/45162
Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-04-22 13:35:17 -06:00
Mike Christie
53be181653 selinux: Fix ceph-iscsi etc access
This fixes the selinux errors like this for /etc/target

-----------------------------------
Additional Information:
Source Context                system_u:system_r:ceph_t:s0
Target Context                system_u:object_r:targetd_etc_rw_t:s0
Target Objects                target [ dir ]
Source                        rbd-target-api
Source Path                   rbd-target-api
Port                          <Unknown>
Host                          ans8
Source RPM Packages
Target RPM Packages
Policy RPM                    selinux-policy-3.14.3-20.el8.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     ans8
Platform                      Linux ans8 4.18.0-147.el8.x86_64 #1 SMP
Thu Sep 26
                              15:52:44 UTC 2019 x86_64 x86_64
Alert Count                   1
First Seen                    2020-01-08 18:39:48 EST
Last Seen                     2020-01-08 18:39:48 EST
Local ID                      9a13ee18-eaf2-4f2a-872f-2809ee4928f6

Raw Audit Messages
type=AVC msg=audit(1578526788.148:69): avc:  denied  { search } for
pid=995 comm="rbd-target-api" name="target" dev="sda1" ino=52198
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:targetd_etc_rw_t:s0 tclass=dir permissive=1

Hash: rbd-target-api,ceph_t,targetd_etc_rw_t,dir,search

which are a result of the rtslib library the ceph-iscsi daemons use
accessing /etc/target to read/write a file which stores meta data the
target uses.

Signed-off-by: Mike Christie <mchristi@redhat.com>
2020-04-22 11:52:02 -05:00
Mike Christie
8187235c91 selinux: Fix ceph-iscsi configfs access
This fixes the the following selinux error when using ceph-iscsi's
rbd-target-api daemon (rbd-target-gw has the same issue). They are
a result of the a python library, rtslib, which the daemons use.

Additional Information:
Source Context                system_u:system_r:ceph_t:s0
Target Context                system_u:object_r:configfs_t:s0
Target Objects
/sys/kernel/config/target/iscsi/iqn.2003-01.com.re
                              dhat:ceph-iscsi/tpgt_1/attrib/authentication
[
                              file ]
Source                        rbd-target-api
Source Path                   /usr/libexec/platform-python3.6
Port                          <Unknown>
Host                          ans8
Source RPM Packages           platform-python-3.6.8-15.1.el8.x86_64
Target RPM Packages
Policy RPM                    selinux-policy-3.14.3-20.el8.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     ans8
Platform                      Linux ans8 4.18.0-147.el8.x86_64 #1 SMP
Thu Sep 26
                              15:52:44 UTC 2019 x86_64 x86_64
Alert Count                   1
First Seen                    2020-01-08 18:39:47 EST
Last Seen                     2020-01-08 18:39:47 EST
Local ID                      6f8c3415-7a50-4dc8-b3d2-2621e1d00ca3

Raw Audit Messages
type=AVC msg=audit(1578526787.577:68): avc:  denied  { ioctl } for
pid=995 comm="rbd-target-api"
path="/sys/kernel/config/target/iscsi/iqn.2003-01.com.redhat:ceph-iscsi/tpgt_1/attrib/authentication"
dev="configfs" ino=25703 ioctlcmd=0x5401
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:configfs_t:s0 tclass=file permissive=1

type=SYSCALL msg=audit(1578526787.577:68): arch=x86_64 syscall=ioctl
success=no exit=ENOTTY a0=34 a1=5401 a2=7ffd4f8f1f60 a3=3052cd2d95839b96
items=0 ppid=1 pid=995 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=rbd-target-api
exe=/usr/libexec/platform-python3.6 subj=system_u:system_r:ceph_t:s0
key=(null)

Hash: rbd-target-api,ceph_t,configfs_t,file,ioctl

Signed-off-by: Mike Christie <mchristi@redhat.com>
2020-04-22 11:52:00 -05:00
Sebastian Wagner
02c46b6d81 cephadm: improve warn message
make it more fancy!

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-authored-by:  Michael Fritch <mfritch@suse.com>
2020-04-22 18:26:27 +02:00
Yuval Lifshitz
f04f01ad42 rgw/notifications: fix zero size in notifications
delete notifications and multipart upload complete notifications
send the correct object size

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Fixes: https://tracker.ceph.com/issues/45150
2020-04-22 18:06:16 +03:00
Casey Bodley
9a3aecbaf3
Merge pull request #34669 from yuvalif/remove_queue_dependency
cls/2pc_queue: remove the dependency of cls_2pc_queue with cls_queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-04-22 10:35:48 -04:00
Casey Bodley
b80dee0123
Merge pull request #34664 from yuvalif/clarify_topic_creation_format
rgw/doc: clarify topic creation format

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-04-22 10:35:28 -04:00
Sebastian Wagner
370bddb301 mgr/cephadm: use check_call for ssh-keygen
Mainly to provide any reasonable error

Fixes: https://tracker.ceph.com/issues/44832
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-22 16:10:27 +02:00
Casey Bodley
3466890e50
Merge pull request #34613 from cbodley/wip-44661
rgw: use DEFER_DROP_PRIVILEGES flag unconditionally

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2020-04-22 10:06:06 -04:00
Sebastian Wagner
bf9cd06c6b qa/workunits/cephadm/test_cephadm: Also test octopus
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-22 15:31:19 +02:00