Commit Graph

110645 Commits

Author SHA1 Message Date
Xiubo Li
5c24d91327 qa/tasks/cephfs: add mount_wait() support to simplify the code
Mostly we should wait the mountpoint to get ready, especially for
the fuse mountpoint, sometimes it may take a few seconds to get
ready.

Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:04 -04:00
Xiubo Li
ed5b737d00 qa/tasks/cephfs/test_misc: switch to netns suspend/resume
https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:04 -04:00
Xiubo Li
eb2fe8440b qa/tasks/cephfs/test_client_recovery: switch to netns suspend/resume
Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:03 -04:00
Xiubo Li
8728da9c08 qa/cephfs/fuse-mount: do not use the 'stat' to check the mount state
If the network couldn't response due to some reasons, the 'stat' cmd
will stuck until the network recovery, the best case is it will stuck
forever.

Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:03 -04:00
Xiubo Li
f0c67256b9 qa/vstart_runner: add unsharing network namespace support
This will isolate the network namespace for each mount point with
a private ip address and iptables, etc.

For the kill() stuff it will just do DOWN the veth interface instead
of sending ipmi request for kernel mount and kill the fuse processes
for the fuse mount. This could avoid sending the socket FIN to the
ceph cluster.

Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:03 -04:00
Xiubo Li
e7f3cf3497 unshare-ns-mount: add isolating netns for mount helper support
This helper script will help to unshare the network namespace from
the os and with its own private veth interface, IP, iptable rules,etc.

It could help us to simulate some test case like client node crash.

Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-04-14 07:47:02 -04:00
Lenz Grimmer
cd271d9516
Merge pull request #34065 from tspmelo/wip-iscsi-advanced
mgr/dashboard: Prevent iSCSI target recreation when editing controls

Reviewed-by: Ricardo Marques <rimarques@suse.com>
2020-04-14 13:31:37 +02:00
Lucian Petrut
6bf7a3eb40 common: Add dlfcn_compat.h for win32
We'll emulate the dlfcn.h interface using Windows functions, which
will allow loading Windows DLLs.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
2020-04-14 11:11:32 +00:00
Lenz Grimmer
9b88f847be
Merge pull request #34009 from tspmelo/wip-iscsi-auth
mgr/dashboard: Fix iSCSI's username and password validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2020-04-14 12:53:14 +02:00
Kefu Chai
1b6fc24552
Merge pull request #33816 from rzarzynski/wip-msg-no-extra-alloc-in-secure
msg/async: fix unnecessary 4 kB allocation in secure mode.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-04-14 17:52:46 +08:00
Igor Fedotov
deb0af6347 doc/rados/operations/health-checks: document bluestore spurious read
errors alert.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-14 12:08:30 +03:00
Igor Fedotov
38103582d4 os/bluestore: raise health alerts when spurious read errors observed
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2020-04-14 12:08:27 +03:00
Varsha Rao
8234765b43 cephfs-shell: Remove undefined exception instance
This patch fixes the following flake8 error:
F821 undefined name 'e'

Fixes: https://tracker.ceph.com/issues/44645
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-04-14 14:22:11 +05:30
Varsha Rao
6f63432cdf cephfs-shell: Fix flake8 indentation issues
This patch fixes the following flake8 error:
E128 continuation line under-indented for visual indent
E122 continuation line missing indentation or outdented

Fixes: https://tracker.ceph.com/issues/44645
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-04-14 12:58:29 +05:30
Varsha Rao
b294be2927 cephfs-shell: Add whitespace around arithmetic operator
This patch fixes the following flake8 error:
E226 missing whitespace around arithmetic operator

Fixes: https://tracker.ceph.com/issues/44645
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-04-14 12:29:44 +05:30
Varsha Rao
7d19c6d616 cephfs-shell: Remove redundant backslash between brackets
This patch fixes the following flake8 error:
E502 the backslash is redundant between brackets

Fixes: https://tracker.ceph.com/issues/44645
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-04-14 12:21:58 +05:30
Varsha Rao
319d6656dd cephfs-shell: Add blank line between functions
This patch fixes the following flake8 error:
E302 expected 2 blank lines, found 1

Fixes: https://tracker.ceph.com/issues/44645
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-04-14 12:11:21 +05:30
Neha Ojha
321faa9c6b qa/standalone/mon/osd-pool-df.sh: fix test to check for the right values
Though the test passed, we weren't checking for the correct values:

.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  jq .stats.total_avail_bytes
../qa/standalone/mon/osd-pool-df.sh:62: TEST_ceph_df:  local global_avail=0
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  jq '.pools | map(select(.name == "$rep_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:63: TEST_ceph_df:  local rep_avail=null
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  ceph df -f json
.../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  jq '.pools | map(select(.name == "$ec_poolname"))[0].stats.max_avail'
../qa/standalone/mon/osd-pool-df.sh:64: TEST_ceph_df:  local ec_avail=null
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  echo '0 >= null*3'
../qa/standalone/mon/osd-pool-df.sh:66: TEST_ceph_df:  bc
1
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  echo '0 >= null*1.5'
../qa/standalone/mon/osd-pool-df.sh:67: TEST_ceph_df:  bc
1

Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-14 00:05:02 +00:00
Neha Ojha
480afa61b6 qa/standalone/mgr/balancer.sh: adapt test
Now that the balancer is on by default the test needs these changes.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-14 00:05:02 +00:00
Neha Ojha
d4fbaf7ea9 pybind/mgr/balancer/module.py: turn on balancer in upmap mode by default
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-14 00:05:02 +00:00
Neha Ojha
bca3b3f6e0 qa/workunits/cephtool/test.sh: don't set-require-min-compat-client luminous
This should not be required since mon_osd_initial_require_min_compat_client
is now luminous.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-04-14 00:05:02 +00:00
Sebastian Wagner
38ae96e1c9
Merge pull request #34344 from jmolmo/rook_inventory
mgr/rook: Fix wrong data output when listing devices

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-04-13 21:15:43 +02:00
Casey Bodley
482b44f921 rgw: dont use RGWOmapAppend for error_repo
the error_repo writes need to be synchronous

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:19 -04:00
Casey Bodley
52fa773270 rgw: remove per-bucket-shard sync leases
bucket sync now gets a const pointer to the DataSyncShard's lease to
check whether the lease has expired

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:19 -04:00
Casey Bodley
16a7df28e2 rgw: ContinuousLeaseCR::is_locked() is const
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:19 -04:00
Casey Bodley
2075addc65 rgw: ContinuousLeaseCR doesn't need locks/atomics
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
7fb98eaf0a rgw: use objv_tracker to read/write bucket sync status
use cls_version on bucket sync status to detect racing writes - whether
from other gateways, or from radosgw-admin commands like 'bucket sync'
or 'bucket sync init'

classes that require a non-null version tracker take it by reference

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
91d326c164 rgw: RadosRemoveCR takes objv_tracker
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
8d715d1814 rgw: SimpleRadosReadAttrsCR takes objv_tracker
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
4e12bde42b rgw: SysObj service applies objv on set_attrs
like write(), we need to apply the writev back to readv

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
8b4a7da7cb rgw: data sync no longer needs index_key_to_marker
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
9826a11514 rgw: DataSyncSingleEntry loops based on progress
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
b65b5b5a9f rgw: DataSyncSingleEntry does not duplicate bucket sync
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
04dc0d0d77 rgw: DataSyncSingleEntry takes cached state
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
a7d859ed2d test/rgw: add unittest_rgw_bucket_sync_cache
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
8fdde41118 rgw: add data sync cache for bucket shards
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Casey Bodley
dec531ad88 rgw: track last timestamp written for bucket sync
bucket sync remembers the latest timestamp that it successfully wrote to
the bucket sync status. data sync can use this to make future decisions
without having to reread its sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 14:08:18 -04:00
Mykola Golub
0bcaa26151 mgr/rbd_support: attempt to unregister watcher on mgr exit
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-04-13 18:47:35 +01:00
Mykola Golub
69259c8d37 mgr/rbd_support: make mirror_snapshot_schedule rescan only updated pools
Signed-off-by: Mykola Golub <mgolub@suse.com>
2020-04-13 18:45:41 +01:00
Kefu Chai
9eef41235a
Merge pull request #33989 from ifed01/wip-ifed-better-bluefs-expand
os/bluestore: open DB in read-only when expanding DB/WAL.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
2020-04-14 00:28:22 +08:00
Kefu Chai
9070f9c90b
Merge pull request #33782 from aclamk/wip-alloc-aging-logging-simpler
os/bluestore: Simplified logic of sliding window.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2020-04-14 00:25:43 +08:00
Kefu Chai
a79fc42188
Merge pull request #33434 from ifed01/wip-ifed-big-writes-deferred
os/bluestore: go deferred for 'big' writes.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2020-04-14 00:25:24 +08:00
Kefu Chai
153f70c09f
Merge pull request #33715 from trociny/wip-pybind-rados-watch
pybind/rados: add watch object bindings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-04-14 00:23:21 +08:00
Casey Bodley
0f16f13af8 rgw: store timestamp in InitBucketShardSyncStatus
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
217f57ebaa rgw: add struct data_sync_obligation
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
dc533b9c16 rgw: add sync_single_entry() helper function
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
1ebe74ee8e rgw: use optional for DataSyncShardMarkerTrack
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
5b97f04d37 rgw: parse bucket-shard before DataSyncSingleEntry
it's easier for DataSyncShard to handle parsing failures before calling
MarkerTrack::start() and DataSyncSingleEntry

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
d03fa0d310 rgw: data sync tracks timestamps for error repo
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00
Casey Bodley
59df17f1d9 rgw: add error repo primitives with timestamp comparisons
the sync error repo stores omap keys for each datalog entry that needs a
retry. this adds a new primitive for that, that also stores a timestamp
with each omap key, and will only allow overwrites/removals when a newer
timestamp is provided

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-04-13 11:06:46 -04:00