Commit Graph

1398 Commits

Author SHA1 Message Date
Rishabh Dave
954ed30f3c
Merge pull request #48027 from lxbsz/wip-57154
mds/client: check the cephx mds auth access in client side

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2023-09-22 20:08:10 +05:30
Rishabh Dave
c4f968b931
Merge pull request #52042 from rishabh-d-dave/better-errmsg-for-perm
MDSAuthCaps: print better error message for perm flag in MDS caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2023-09-22 20:02:40 +05:30
Rishabh Dave
3234c696de
Merge pull request #47533 from rishabh-d-dave/ceph-fuse-cmd
qa/cephfs: cleanup code for ceph-fuse command

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-22 20:01:13 +05:30
Milind Changire
29eb1ca580
Merge PR #52892 into main
* refs/pull/52892/head:
	qa: add test to validate periodic checks by async threads
	mgr/volumes: periodically check for async work

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
2023-09-19 12:53:29 +05:30
Milind Changire
a94089bde2
Merge PR #52692 into main
* refs/pull/52692/head:
	qa/tasks/cephfs: reset the client_inject_fixed_oldest_tid after test

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-19 12:51:52 +05:30
Patrick Donnelly
6c387554d8
Merge PR #52638 into main
* refs/pull/52638/head:
	mds: flush monc log before abort
	qa: check for expected cluster log message
	qa: ignore expected cluster warning from damage tests

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-15 14:38:02 -04:00
Venky Shankar
98758eee89 Merge PR #53070 into main
* refs/pull/53070/head:
	PendingReleaseNotes: add note about 'm' and 'M' periods
	doc/cephfs: add note about monthly scheduling and retention period
	qa: update minute period multiplier to m
	qa: change minute duration specifier from M to m
	mgr/snap_schedule: change minute duration specifier from M to m

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-14 15:39:07 +05:30
Venky Shankar
99adb599e7 Merge PR #51891 into main
* refs/pull/51891/head:
	qa: fix cephfs-top tests for the new fields
	cephfs-top: include the missing fields in --dump output

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
2023-09-14 09:29:07 +05:30
Venky Shankar
59d1f7655c Merge PR #53153 into main
* refs/pull/53153/head:
	Revert "mds: align quota.max_bytes to 4MB or 4KB"

Reviewed-by: Xiubo Li <xiubli@redhat.com>
2023-09-13 11:10:59 +05:30
Jos Collin
0241a82b4d
qa: fix cephfs-top tests for the new fields
Fixes: https://tracker.ceph.com/issues/61397
Signed-off-by: Jos Collin <jcollin@redhat.com>
2023-09-12 17:35:30 +05:30
Milind Changire
e7218e03ae
qa: change minute duration specifier from M to m
Fixes: https://tracker.ceph.com/issues/62494
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 13:58:26 +05:30
Rishabh Dave
7d1f6d70f1 qa/cephfs: set omit_sudo for cmd mounting FUSE conn dir
vstart_runner.py removes sudo from the command arguments when "omit_sudo"
is not explicitly set to "False". This leads to 2 error messages in output
of vstart_runner.py which are copied at the end of this commit message.
Setting "omit_sudo=False" for the execution of command should remove both
of these error messages.

This also fixes the potential crash of vstart_runner.py when the FUSE
connections directory is not mounted on the machine and it requires
superuser privileges to mount it.

The 2 error messages copied as it is from the log -

DEBUG:__main__:"sudo" was omitted from the following cmd args before execution and logging using function overriding; check vstart_runner.py for more details.
DEBUG:__main__:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections
mount: /sys/fs/fuse/connections: must be superuser to use mount.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-12 13:26:00 +05:30
Rishabh Dave
c4b2ccd9d6 qa/cephfs: check before mounting fuse connections directory
Check if "/sys/fs/fuse/connections" is already mounted before attempting
to mount it again. Doing so leads to unnecessary error messages in the
vstart_runner.py's output (and most likely in logs for teuthology test
runs too) which confuses the users. See end for the error message.

Also, remove "check_status=False" so that the when the commands crashes
the testing is halted immediately. Initially, this must've been added to
ignore errors generated from re-mounting FUSE connections directory
repeatedly.

The error message -

DEBUG:__main__:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections
mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-12 13:25:45 +05:30
Ramana Raja
a04044670c qa/tasks/cephfs/test_admin: run root_squash tests only for FUSE client
kclient doesn't have CEPHFS_FEATURE_MDS_AUTH_CAPS required to
enforce root_squash. Run root_squash tests only for FUSE client.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2023-09-11 09:29:46 +08:00
Ramana Raja
c8bae07f70 qa/tasks/cephfs: Add reproducer for https://tracker.ceph.com/issues/56067
A kernel CephFS client with MDS root_squash caps is able to write to a
file as non-root user. However, the data written is lost after clearing
the kernel client cache, or re-mounting the client. This issue is not
observed with a FUSE CephFS client.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2023-09-11 09:29:46 +08:00
Xiubo Li
28023f84d7 qa: add test for checking access in client side of root_squash
Test the 'chown' and 'truncate', which will call the setattr and
'cat' will open the files. Before each testing will open the file
by non-root user and keep it to make sure the Fxw caps are issued,
and then user the 'sudo' do to the tests, which will set the uid/gid
to 0/0.

Fixes: https://tracker.ceph.com/issues/57154
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2023-09-11 09:29:46 +08:00
Xiubo Li
4af6a3f744 qa: add sudo paramter for read_file()
Fixes: https://tracker.ceph.com/issues/57154
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2023-09-11 09:29:46 +08:00
Rishabh Dave
8e1e322054 AuthMonitor: no need to check permission in MDS caps
For "fs authorize" command, AuthMonitor.cc checks if permissions is "r"
or begins with "rw". This check is redundant now.
AuthMonitor::valid_caps() runs MDSAuthCaps.parse() which now runs same
check for the MDS caps, regardless of the command.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-08 20:17:37 +05:30
Rishabh Dave
f163dd3ef1 MDSAuthCaps: print a special error message for wrong permissions
Permissions mentioned in MDS caps flags can either begin with "r" or
"rw", or can be "*" and "all". But it can't start with or be just "w" or
something else. This is confusing for some CephFS users since MON caps
can be just "w".

Command "ceph fs authorize" complains about this to the user. But other
commands (specifically, "ceph auth add", "ceph auth caps",
"ceph auth get-or-create" and "ceph auth get-or-create-key") don't. Make
these commands too print a helpful message, the way "ceph fs authorize"
command does.

Fixes: https://tracker.ceph.com/issues/61666
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-08 20:17:37 +05:30
Rishabh Dave
849fbd1d5b
Merge pull request #41779 from rishabh-d-dave/fs-auth-subcmd
mon,osd,mds: allow fs authorize to update caps

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-07 18:39:58 +05:30
Rishabh Dave
c9058dc7a5 qa/cephfs: CephFSTestCase.create_client() must keyring
Replace call to run_ceph_cmd() by call to get_ceph_cmd_stdout() in
method qa.tasks.cephfs.cephfs_test_case.CephFSTestCase.create_client().
run_ceph_cmd() will not return keyring which is wrong.
get_ceph_cmd_stdout() will return the stdout of "ceph auth add"
command, which is the keyring that is expected to be returned by
CephFSTestCase.create_client().

Fixes: https://tracker.ceph.com/issues/62246
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-07 11:37:17 +05:30
Rishabh Dave
f9a01cf385
Merge pull request #51942 from trociny/wip-61599
mds: optionally forbid to use standby for another fs as last resort

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2023-09-06 20:26:33 +05:30
Venky Shankar
6d8cb29d1e Revert "mds: align quota.max_bytes to 4MB or 4KB"
This seems to be causing quota related test failures in the fs
suoite. @lxbsz will push a separate PR addressing the issue in
tracker #56397.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2023-09-06 18:26:39 +05:30
Rishabh Dave
00f23e9a03
Merge pull request #52924 from rishabh-d-dave/test-nfs-pr-52556
qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase

Reviewed-by: Adam King <adking@redhat.com>
2023-09-05 20:21:30 +05:30
Rishabh Dave
1414bb3e0e qa/cephfs: set omit_sudo=False when ceph-fuse is executed
Although setting "omit_sudo" to "False" will have no impact on ceph-fuse
mount command during testing with teuthology (because teuthology code
does nothing whether omit_sudo is set or unset), let's set it to False
so that there's no chance for any bugs in future.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-09-04 22:17:32 +05:30
Milind Changire
d7dfac8111
Merge PR #52686 into main
* refs/pull/52686/head:
	PendingReleaseNotes: note about mandatory fs argument
	doc/cephfs: add note about mandatory --fs argument to snap-schedule
	qa: add test for mandatory fs argument to snap-schedule commands
	mgr/snap-schedule: tweaks to keep mypy happy
	mgr/snap_schedule: validate fs before execution

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-08-29 20:13:50 +05:30
Venky Shankar
52a908a605 Merge PR #52940 into main
* refs/pull/52940/head:
	qa: Wait for purge to complete in test_volume_info_pending_subvol_deletions

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-08-28 11:51:11 +05:30
Venky Shankar
82fa0661b2 qa: add test to validate periodic checks by async threads
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2023-08-28 02:17:35 -04:00
Milind Changire
484336e1c3
qa: add test for mandatory fs argument to snap-schedule commands
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-08-25 21:34:40 +05:30
Venky Shankar
e9f8be4bac Merge PR #52944 into main
* refs/pull/52944/head:
	PendingReleaseNotes: add a note for `mds_session_metadata_threshold` mds config
	test: add test to verify that a buggy client is blocklisted
	mds: add perf counter to track number of sessions evicted due to metadata threshold being exceeded
	mds: blocklist clients with "bloated" session metadata

Reviewed-by: Robin H. Johnson <robbat2@orbis-terrarum.net>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-25 18:46:20 +05:30
Rishabh Dave
4b369cf18e qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase
MgrTestCase also needs RunCephCmd. If RunCephCmd is inherited by
CephTestCase, instead of CephFSTestCase, MgrTestCase will automatically
inherit RunCephCmd because it inhertis CephTestCase.

Fixes: https://tracker.ceph.com/issues/62084
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-08-24 23:45:27 +05:30
Venky Shankar
726e5d7dde Merge PR #52676 into main
* refs/pull/52676/head:
	mds/Server: mark a cap acquisition throttle event in the request

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2023-08-24 17:35:04 +05:30
Venky Shankar
f2e17e40e7 Merge PR #52741 into main
* refs/pull/52741/head:
	qa/cephfs: switch to python3 for centos stream 9

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-08-22 13:23:40 +05:30
Venky Shankar
84df4b3d0c test: add test to verify that a buggy client is blocklisted
... when its session metadata is bloated due to buildup of
`completed_requests`.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2023-08-22 00:47:27 -04:00
Leonid Usov
749c770676 mds/Server: mark a cap acquisition throttle event in the request
Fixes: https://tracker.ceph.com/issues/59067
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
2023-08-18 18:11:29 +03:00
Venky Shankar
9545e578a4 Merge PR #52547 into main
* refs/pull/52547/head:
	qa: add test cases for vanilla ops commands
	mds: dump locks when printing mutation ops
	common/TrackedOp: support overriding the _dump method
	mds: remove op field obsoleted by more usable "reqid"
	mds: dump metareq_t instead of full op
	mds: add lock type to formatter dump of SimpleLock
	mds: mark print methods const
	mds: drop MDRequestImpl::msg_lock
	mds: lock TrackedOp when dumping
	mds: avoid recursive locks dumping state
	common/TrackedOp: fix race updating description with proper lock
	common/Formatter: add support for dumping null
	common/Formatter: refactor generating xml name

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-08-14 17:52:47 +05:30
Kotresh HR
72125396d4 qa: Wait for purge to complete in test_volume_info_pending_subvol_deletions
Fixes: https://tracker.ceph.com/issues/62278
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2023-08-11 14:27:29 +05:30
Venky Shankar
53f89ea09b Merge PR #52765 into main
* refs/pull/52765/head:
	mgr/volumes: Fix pending_subvolume_deletions in volume info
	qa: Add testcase for pending_subvolume_deletions count

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
2023-08-11 11:39:52 +05:30
Patrick Donnelly
ca4d0dc42b
qa: add test cases for vanilla ops commands
To test they work, not that the output is useful.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-08 08:58:42 -04:00
Leonid S. Usov
8262586cd0
Merge pull request #52792 from leonid-s-usov/bulk-data-pool
mgr/volumes: create bulk data pool for new volumes
2023-08-08 11:24:59 +03:00
Kotresh HR
8b1303f4b1 qa: Add testcase for pending_subvolume_deletions count
Fixes: https://tracker.ceph.com/issues/62278
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2023-08-04 12:47:24 +05:30
Leonid Usov
9a8219cc2b mgr/volumes: set the 'bulk' flag for data pools created automatically for a new volume
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/61595
2023-08-03 19:41:12 +03:00
Venky Shankar
cd18c51548 Merge PR #48732 into main
* refs/pull/48732/head:
	doc: add MDS treatise on segments
	pybind/mgr/dashboard: bump teuthology version
	qa/tasks/vstart_runner: stop overriding _run_python
	qa: stop overriding ceph_w prefix in vstart_runner
	qa/tasks/vstart_runner: update teuthology helper tool paths
	qa/tasks/vstart_runner: allow writing to command's stdin
	vstart.sh: always add CEPH_CONF to vstart_environment.sh
	qa: use stdin-killer for python3 command
	qa: add killpoint testing for mds shutdown
	qa: fix background exit condition
	qa: add filesystem helper for setting transient config
	qa: add helper for waiting for a rank to fail
	mds: add incompat feature for minor log segments
	mds: introduce ELid event to create/close log
	mds: change EResetJournal to major segment boundary
	mds: add killpoints for MDS shutdown
	qa: add numerous subtree test
	mds: track larger log events in perf dump
	mds: add minor LogSegment boundaries
	mds: obviate MDLog::start_entry
	mds: retype to properly sized unsigned ints
	mds: use unsigned type for event count
	mds: use base Context class for generalization
	mds: optimize segment lookup
	mds: add stream dump for LogSegment
	mds: handle conf changes in mdlog
	mds: remove redundant comment
	mds: remove unused method
	mds: set a reasonable minimum number of segments
	mds: sort configs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-08-02 22:07:58 +05:30
Rishabh Dave
0584cd34b6
Merge pull request #52709 from rishabh-d-dave/cephfs-test_snapshots
qa/cephfs: fix test_disallow_monitor_managed_snaps_for_fs_pools

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-08-02 19:18:07 +05:30
Xiubo Li
0a296183e0 qa/cephfs: switch to python3 for centos stream 9
Fixes: https://tracker.ceph.com/issues/62277
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2023-08-02 18:45:56 +08:00
Casey Bodley
568a21d83c qa/cephfs: redefinition of unused 'random' from line 7
seeing this run-tox-qa failure about tasks/cephfs/test_client_recovery.py:

246/285 Test #264: run-tox-qa ................................***Failed   58.54 sec
Requirement already satisfied: tox in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (4.6.4)
Requirement already satisfied: cachetools>=5.3.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.3.1)
Requirement already satisfied: chardet>=5.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.1.0)
Requirement already satisfied: colorama>=0.4.6 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (0.4.6)
Requirement already satisfied: filelock>=3.12.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.12.2)
Requirement already satisfied: packaging>=23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (23.1)
Requirement already satisfied: platformdirs>=3.8 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.10.0)
Requirement already satisfied: pluggy>=1.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.2.0)
Requirement already satisfied: pyproject-api>=1.5.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.5.3)
Requirement already satisfied: tomli>=2.0.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (2.0.1)
Requirement already satisfied: virtualenv>=20.23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (20.24.2)
Requirement already satisfied: distlib<1,>=0.3.7 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from virtualenv>=20.23.1->tox) (0.3.7)
flake8: install_deps /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -I -m pip install flake8
flake8: freeze /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -m pip freeze --all
flake8: flake8==6.1.0,mccabe==0.7.0,pip==22.3.1,pycodestyle==2.11.0,pyflakes==3.1.0,setuptools==65.6.3,wheel==0.38.4
flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/test_client_recovery.py:12:1: F811 redefinition of unused 'random' from line 7
flake8: exit 1 (3.72 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox pid=706315
flake8: FAIL ✖ in 15.42 seconds

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-08-01 12:31:44 -04:00
Patrick Donnelly
20184c23d3
qa: use stdin-killer for python3 command
This relies on the new stdin-killer [1] teuthology helper that allows
interacting with the command's stdin.

[1] https://github.com/ceph/teuthology/pull/1846

Fixes: 8bb77ed9e1
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-01 11:16:26 -04:00
Patrick Donnelly
936da39a15
qa: add killpoint testing for mds shutdown
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-01 11:16:02 -04:00
Patrick Donnelly
1fa0039a98
qa: fix background exit condition
This change causes the program to exit gracefully when stdin is closed
rather than with a Python exception.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-01 11:16:02 -04:00
Patrick Donnelly
1962322f59
qa: add filesystem helper for setting transient config
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-08-01 11:16:02 -04:00