Commit Graph

834 Commits

Author SHA1 Message Date
Venky Shankar
cecc3a6982 test: add symbolic link synchronization test for cephfs-mirror
Fixes: http://tracker.ceph.com/issues/49711
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-03-12 03:53:42 -05:00
Venky Shankar
8ee88f5735 pybind/mirroring: add interface to list file system mirror peers
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-03-11 03:41:52 -05:00
Venky Shankar
3e13f48937 test: add tests for mirroring bootstrap interfaces
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-03-11 03:41:52 -05:00
Venky Shankar
d1585af77b test: add test for failed filesystem mirror instances
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-03-11 03:41:52 -05:00
Sebastian Wagner
1d2009c3e9
Merge pull request #39640 from sebastian-philipp/cephadm-cephfs-mirror
cephadm: cephfs-mirror

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2021-03-10 18:34:03 +01:00
Patrick Donnelly
4b506bbe4e
Merge PR #38913 into master
* refs/pull/38913/head:
	qa/tasks/cephfs/nfs: Add tests for updating fs exports
	mgr/volumes/nfs: Handle rook restart error
	doc/cephfs/nfs: Add about update export interface
	mgr/volumes/nfs: Add command to update cephfs exports
	pybind/volumes/nfs: set mds caps according to user specified access type
	mgr/volumes/module: Remove unused json module

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-10 07:01:27 -08:00
Sebastian Wagner
340281fe76 qa/tasks: some type annotations
Mostly for making my IDE aware of things

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-03-10 15:02:41 +01:00
Patrick Donnelly
1cbad56fd2
Merge PR #37711 into master
* refs/pull/37711/head:
	qa/cephfs: create class LocalCephFSMount
	vstart_runner: reuse FuseMount methods in LocalFuseMount
	vstart_runner: reuse KernelMount methods in LocalKernelMount
	qa/cephfs: move common and generic methods mount.py
	qa/cephfs: refactor FuseMount.mount method
	qa/cephfs: refactor KernelMount.mount method

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-09 13:41:53 -08:00
Patrick Donnelly
be2958f489
Merge PR #39408 into master
* refs/pull/39408/head:
	test: add test for validating cephfs-mirror daemon service status
	pybind/mirroring: interface to fetch mirror daemon status
	cephfs-mirror: register mirror daemon with service manager
	cephfs-mirror: remove unneeded json_spirit header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-09 13:40:09 -08:00
Patrick Donnelly
fe31c23964
Merge PR #39814 into master
* refs/pull/39814/head:
	qa/cephfs: move is_blocklisted() to filesystem.CephCluster

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-09 13:36:40 -08:00
Kefu Chai
434c1ce400
Merge pull request #39775 from rishabh-d-dave/fs-qa-accept-cmds-as-str
qa/ceph_manager: accepts commands as str too

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-07 23:52:54 +08:00
Rishabh Dave
4d0f56fcc5 qa/cephfs: move is_blocklisted() to filesystem.CephCluster
Using self.fs.mon_manager in mount.py can lead to a crash since self.fs
can be None. Move is_blocklisted() to tasks.filesystem.CephCluster where
it can get access to mon_manager without depending on objects
representing Ceph FSs.

Fixes: https://tracker.ceph.com/issues/49511
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-06 09:11:36 +05:30
Venky Shankar
c0632f6bd5 test: add test for validating cephfs-mirror daemon service status
Fixes: http://tracker.ceph.com/issues/48943
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-03-05 01:00:01 -05:00
Rishabh Dave
5e71e9b065 qa/cephfs: move common and generic methods mount.py
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-05 10:03:12 +05:30
Rishabh Dave
21cbda91f1 qa/cephfs: refactor FuseMount.mount method
Make FuseMount.mount() such that it can LocalFuseMount can reuse it
instead of duplicating it. For this, move subtasks required for mounting
Ceph FS (like creating mountpoint directory) to a different method, so
that these methods can be overriden in LocalFuseMount as per
requirement.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-05 09:55:10 +05:30
Rishabh Dave
86467b14d7 qa/cephfs: refactor KernelMount.mount method
Make KernelMount.mount() such that it can LocalKernelMount can reuse it
instead of duplicating it. For this, move subtasks required for mounting
Ceph FS (like creating mountpoint directory) to a different method, so
that these methods can be overriden in LocalKernelMount as per
requirement.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-05 08:53:26 +05:30
Patrick Donnelly
ec13ed7c50
Merge PR #39724 into master
* refs/pull/39724/head:
	qa: skip exit-on-first-failure option for valgrind on ubuntu
	mds,qa: exit instead of respawn under valgrind
	qa: skip chdir for fuse_mount
	qa: ignore all slow request warnings
	qa: add new mds beacon grace mon config
	qa: wait for MDS to join fsmap
	qa: move get_valgrind_args to qa

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-03-04 13:32:10 -08:00
Patrick Donnelly
13c50a9923
Merge PR #38684 into master
* refs/pull/38684/head:
	qa: add _check_scrub_status helper to simplify the code
	qa: add run_scrub helper in filesystem class
	qa: add get_scrub_status helper in filesystem class
	qa: wait the scrub task to complete
	qa: remove passed_validation check for test_damage
	qa: move wait_until_scrub_complete helper to filesystem class
	mds: simplify the C_MDS_EnqueueScrub finish code

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-04 13:27:45 -08:00
Varsha Rao
0928858db0 qa/tasks/cephfs/nfs: Add tests for updating fs exports
Signed-off-by: Varsha Rao <varao@redhat.com>
2021-03-04 15:18:07 +05:30
Rishabh Dave
a1dc6b6c19 qa/ceph_manager: accepts commands as str too
Modify CephManager.run_cluster_cmd() to accept command arguments as
string as well since typing commands as strings is much lesser effort
than typing as list. This brings the interface a step closer to
teuthology.orchestra.remote.run()'s interface since it too can accept
commands arguments as string.

The change in cephfs_test_case.py is just to allow testing this PR
locally and on teuthology.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-04 09:42:44 +05:30
Patrick Donnelly
3e5e03d4d2
qa: skip chdir for fuse_mount
The use of chdir will muck up the use of nsenter with valgrind:

    2021-03-03T02:13:49.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/client.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' --id 0 /home/ubuntu/cephtest/mnt.0
    2021-03-03T02:13:49.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo modprobe fuse
    2021-03-03T02:13:49.914 INFO:teuthology.orchestra.run:Running command with timeout 30
    2021-03-03T02:13:49.914 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections
    2021-03-03T02:13:49.919 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi144.stderr:nsenter: failed to execute cd: No such file or directory

It's not necessary to chdir at all to do the mount, so don't.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-03 09:30:21 -08:00
Patrick Donnelly
3681e3a1a8
qa: move get_valgrind_args to qa
This method is unused in the teuthology repo. The helper method better
belongs here where it is more easily modified.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-03 09:30:08 -08:00
Patrick Donnelly
c991e28a92
Merge PR #39680 into master
* refs/pull/39680/head:
	mds: allow `fs authorize` command to work

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-03-01 12:07:32 -08:00
Patrick Donnelly
4e6d9e6a04
qa: delete all fs during tearDown
During the ceph task Unwind, the MDS are stopped. If any file system
still exists, we will see failover messages in the cluster log.

Fixes: https://tracker.ceph.com/issues/49510
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-02-26 09:13:21 -08:00
Patrick Donnelly
8ebe66f362
Merge PR #39607 into master
* refs/pull/39607/head:
	qa/cephfs: check for Ceph FS before mounting
	qa/cephfs: remove parameter createfs from mount classes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-02-24 10:34:29 -08:00
Rishabh Dave
95eb0043bf qa/cephfs: check for Ceph FS before mounting
If the given Ceph FS, or the default Ceph FS when no Ceph FS is given,
is absent, abort the execution with AsssertionError and an error
message.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-02-24 19:10:52 +05:30
Ramana Raja
22ed4f43d9 mds: allow fs authorize command to work
... on filesystems with '-' in their names.

Fixes: https://tracker.ceph.com/issues/49301
Signed-off-by: Ramana Raja <rraja@redhat.com>
2021-02-23 22:38:50 -05:00
Ramana Raja
99e34b4ebf
Merge pull request #39572 from kotreshhr/version-auth-metadata-mgr
mgr/volumes: Bump up AuthMetadataManager's version

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-02-22 23:16:08 -05:00
Patrick Donnelly
a3591378a5
Merge PR #39138 into master
* refs/pull/39138/head:
	qa: valgrind test for cephfs-mirror daemon
	cephfs-mirror: use preforker for daemonizing
	test: adjust sleep time to account for valgrind runs
	cephfs-mirror: gracefully shutdown threads, timers, etc..
	cephfs-mirror: call ceph_release() to cleanup mount alloc
	cephfs-mirror: shutdown filesystem/cluster connections on shutdown
	cephfs-mirror: set init failed flag on FSMirror::init() failure

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-02-22 10:35:32 -08:00
Kotresh HR
818c7781ff mgr/volumes: Bump up AuthMetadataManager's version
With ceph_volume_client and mgr-volumes co-existing
for sometime, the version of both needs to be same.
The ceph_volume_client version <=5 can't decode
'subvolumes' key in auth-metadata file. Hence to
handle version in-compatibility, the version of
ceph_volume_client is bumped up to 6 and the same
needs to be done in mgr-volume's AuthMetadataManager

Fixes: https://tracker.ceph.com/issues/49374
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-02-21 12:59:40 +05:30
Rishabh Dave
db0b85b440 qa/cephfs: remove parameter createfs from mount classes
It's better to get rid of this paramter since it doesn't actually create
any new Ceph file system.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-02-20 15:32:52 +05:30
Ramana Raja
324c672a56
Merge pull request #39327 from kotreshhr/qa-fix-mgr-vol-test-cases
qa: Fix a few mgr/volume test cases

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-02-11 23:59:22 -05:00
Kotresh HR
4952d2132f qa: Fix a few mgr/volume test cases
Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.

Fixes: https://tracker.ceph.com/issues/49192
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-02-11 19:22:44 +05:30
Kefu Chai
5ca820fb9e Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-02-08 18:03:14 +08:00
Sage Weil
6d8c0e4722 Merge PR #39147 into master
* refs/pull/39147/head:
	qa/tasks/ceph_fuse: do not createfs
	qa/tasks/cephfs/fuse_mount: pass admin_socket path
	qa/suites/fs/cephadm/multivolume: add basic multivolume test
	mgr/mds_autoscaler: some fixes and cleanup
	mgr/volumes: deploy MDSs when creating fs

Reviewed-by: Milind Changire <mchangir@redhat.com>
2021-02-04 12:19:25 -05:00
fullerdj
d4088d8313
Merge pull request #37954 from joscollin/wip-F46226-log-dq-throttle-limit-new
msg,mon,common: log when DispatchQueue throttle limit is reached
2021-02-03 10:42:40 -05:00
Venky Shankar
ebf248b143 test: adjust sleep time to account for valgrind runs
Valgrind make everything run slow. So, increase wait timeouts
wherever necessary.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-02-02 09:25:20 -05:00
Sage Weil
c2d53135fd qa/tasks/cephfs/fuse_mount: pass admin_socket path
For ceph.py this comes from the ceph.conf.template, but it's not there
for cephadm.  Instead of inflicting this on the config inside the
container, just pass it to ceph-fuse incantation here.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-02-01 10:50:33 -06:00
Ramana Raja
23f0c5b2dd
Merge pull request #38786 from kotreshhr/evict-clients-of-deauthorized-authids
mgr/volumes: Evict clients based on auth-IDs and subvolume path

Reviewed-by: Victoria Martinez de la Cruz <victoria@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-01-27 19:44:17 +05:30
Kotresh HR
269adcc8b8 mgr/volumes: Evict clients based on auth-IDs and subvolume mounted
Add subvolume evict command which evicts the subvolume mounts
which are mounted using particular auth-ID.

Fixes: https://tracker.ceph.com/issues/44928
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-01-25 15:53:14 +05:30
Jos Collin
8367448765
qa: test DispatchQueue throttling
Fixes: https://tracker.ceph.com/issues/46226
Signed-off-by: Jos Collin <jcollin@redhat.com>
2021-01-25 11:18:04 +05:30
Venky Shankar
f81e8f1e88 test: add tests for mirroring module w/ daemon verification
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-01-19 01:08:10 -05:00
Venky Shankar
8334bea5a6 test: optionally create a backup filesystem on startup
Also filter out client-id's starting with "mirror" when
cleaning leftover auth-ids since teuthology would be
configured to create client.mirror and client.mirror_remote
clients before executing mirroring tests.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-01-19 01:08:10 -05:00
Patrick Donnelly
60510411be
Merge PR #38160 into master
* refs/pull/38160/head:
	doc/mgr/volumes: Document 'fs subvolume authorized_list' cli
	qa: Add tests for list auth-ids of a subvolume
	mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file
	mgr/volumes: Optionally authorize existing auth-ids
	mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id
	mgr/volumes: Disallow authorize existing auth_id
	mgr/volumes: Add subvolume authorized_list command
	mgr/volumes: Add tenant_id option to subvolume authorize
	mgr/volumes: Persist auth and subvolume metadata

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-15 11:53:06 -08:00
Kotresh HR
1b98e63e75 qa: Add tests for list auth-ids of a subvolume
Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-01-15 10:10:22 +05:30
Patrick Donnelly
78a7df1500
Merge PR #38846 into master
* refs/pull/38846/head:
	*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-01-14 08:10:19 -08:00
Patrick Donnelly
d73cfde0b8
Merge PR #38835 into master
* refs/pull/38835/head:
	qa: add test for fs rm idempotency
	mon/FSCommands: restore idempotent behavior of fs rm
	qa: add ceph cmd helper
	qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-01-13 08:24:12 -08:00
Kotresh HR
bd49b6409b mgr/volumes: Filter inherited snapshots while listing snapshots
Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-01-13 15:09:35 +05:30
Patrick Donnelly
1e9622cff9
Merge PR #38793 into master
* refs/pull/38793/head:
	qa: check for rank hole without racy joinable flag gymnastics
	qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-01-12 07:27:46 -08:00
Patrick Donnelly
780a1dd5ff
qa: add test for fs rm idempotency
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-12 07:24:33 -08:00