Commit Graph

169 Commits

Author SHA1 Message Date
Rishabh Dave
b2030edc15 qa/vstart_runner: dont log "not Ceph bin" msg too often
The message is logged everytime a binary not from Ceph repo's build
directory is executed, which it too often.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-03-02 16:41:09 +05:30
Rishabh Dave
db0b85b440 qa/cephfs: remove parameter createfs from mount classes
It's better to get rid of this paramter since it doesn't actually create
any new Ceph file system.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-02-20 15:32:52 +05:30
Patrick Donnelly
488f10c62f
qa: move cephfs_ec_profile under cephfs
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-07 12:55:25 -08:00
Patrick Donnelly
2bce009d99
qa: add debug vstart_runner option
The default was changed to INFO recently but there was no way to restore
visibility of DEBUG messages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-06 11:36:49 -08:00
Aashish Sharma
f5dd0195cd mgr/dashboard:minimize console log traces of Dashboard backend API tests
Changed log levels in some files to reduce console log traces

Fixes: https://tracker.ceph.com/issues/45822
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2020-12-23 11:32:43 +05:30
Rishabh Dave
b0cc7ac1a8 vstart_runner: print result line unless opt_rotate_log is set
Unless the option to rotate logs is set, don't delete the final result
line printed by unittest.

Fixes: https://tracker.ceph.com/issues/48447
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-12-03 16:20:30 +05:30
Ramana Raja
7016a2001d qa/tasks: allow per file system config setting
Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-11-20 13:23:21 +05:30
Xiubo Li
871ac6b376 qa/tasks/vstart_runner.py: disable the ns unsharing if using localhost
If the -l option is specified in :

$ MDS=3 MON=3 MGR=1 OSD=3 ../src/vstart.sh -n -l

The ceph cluster will use the localhost IP address as the mon host,
and then in the network namespace container the ceph-fuse daemon
couldn't connect to it.

Fixes: https://tracker.ceph.com/issues/45342
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-11-12 08:54:57 +08:00
Patrick Donnelly
c569036c5d
Merge PR #37629 into master
* refs/pull/37629/head:
	qa/cephfs: add session_timeout option support
	qa/cephfs: move the cephfs's opertions setting to create()
	qa/cephfs: add 'cephfs:' section support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-25 16:26:36 -07:00
Xiubo Li
0422673b61 qa/cephfs: add session_timeout option support
When the mds revoking the Fwbl caps, the clients need to flush
the dirty data back to the OSDs, but the flush may make the OSDs
to be overloaded and slow, which may take more than 60 seconds to
finish. Then the MDS daemons will report the WRN messages.

For the teuthology test cases, let's just increase the timeout
value to make it work.

Fixes: https://tracker.ceph.com/issues/47565
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-23 14:27:37 +08:00
Rishabh Dave
f9f4c78101 vstart_runner: omit result line only when testcases are run one by one
vstart_runner omits the result line ("Ran X tests in Y") generated by
unittest unconditionally. Don't do so when vstart_runner triggers entire
test module at once.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-10-13 09:12:45 +05:30
Rishabh Dave
80ecc017e8 vstart_runner: don't quit running testsuite on a test failure
Add an option that makes it possible to not to quit running testsuite on
a test failure. This way user can get a better idea on the current state
of the testsuite and test its own code patches more effectively and
easily.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-10-13 09:12:45 +05:30
Patrick Donnelly
5c9f77cb65
Merge PR #36457 into master
* refs/pull/36457/head:
	qa/tasks/cephfs: fix fs authorize cmd args
	mds: add root_squash mode in MDS auth caps
	vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-08 13:23:34 -07:00
Rishabh Dave
864a15e7ca vstart_runner: rotate logs between testcases optionally
Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-29 14:52:47 +05:30
Rishabh Dave
6673531804 vstart_runner: rearrange exec_test()
Rearrange the code that triggers the test runner so that in the later
commits the way tests are triggered can be changed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-29 14:49:51 +05:30
Ramana Raja
a177b470aa vstart_runner: add write_file and sudo_write_file
... methods in LocalRemote class. These methods are called in some of
the recently added cephfs tests. They were implemented in teuthology's
Remote class, but not in vstart_runner's LocalRemote class. Hence some
cephfs tests couldn't be run locally using vstart_runner without this
change.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-09-25 14:34:35 +05:30
Rishabh Dave
352a41d1b3 qa/cephfs: add tests for multi-FS auth tests
Add testsuite for testing authorization on Ceph cluster with multiple
file systems and enable it to be executable with Teuthology framework.

Also add helper methods required to setup the test environment for
multi-FS tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-11 18:02:48 +05:30
Rishabh Dave
04ed58fec2 qa/cephfs: modify delete_all_filesystems() in filesystem.py
Modify filesystem.Filesystem.delete_all_filesystems() method to make it
more succinct, move it to class MDSCluster instead and update every call
to it accordingly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 23:56:59 +05:30
Rishabh Dave
9a36a0abd0 qa/cephfs: allow not aborting execution when mount command fails
This commit adds a new argument check_status to mount methods of
KernelMount, FuseMount, LocalKernelMount and LocalFuseMount. When value
of this argument is False, these methods would catch the
CommandFailedError exception and would return a tuple consisting of the
exception itself, and stdout and stderr of the mount command. This
allows reusing these mount methods while running negative tests for
commands.

The name "check_status" is selected so since teuthology's run() and
vstart_runner's run() use a variable with same name for the very same
purpose.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
07e493ffb5 qa/cephfs: allow reusing mount objects and add remount method
This commit introduces following two set of changes -

First, make client keyring path, mountpoint on host FS and CephFS and
CephFS's name attributes of the object representing the mount
and update all the mount object creation calls accordingly. Also,
rewrite all the mount object creation to use keyword arguments instead
of positional arguments to avoid mistakes, especially since a new
argument was added in this commit.

Second, add remount method to mount.py so that it's possible to unmount
safely, modify the attributes of the object representing the mount and
mount again based on new state of the object *in a single call*. The
method is placed in mount.py to avoid duplication.

This change has two leads to two more changes: upgrading interface of
mount() and mount_wait() and upgrading testsuites to adapt to these
change.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
464ec3dbdc vstart_runner: log commands in a more usable form
Currently commands are printed as ['arg1', 'arg2', 'arg3']. Instead, log
them as '> arg1 arg2 arg3' so that it's simpler to copy and run them
manually. The reason behind prepending '> ' to these logs entries is
just to follow the practice followed by teuthology logs.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-02 22:52:27 +05:30
Rishabh Dave
530982129e qa: add method run ceph cluster command with better interface
This new method should allow better control on the process launched by
the passed command. This is achieved by allowing arguments provided by
teuthology.orchestra.run.run().

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-21 22:16:21 +05:30
Ernesto Puerta
64ec1112bb
Revert "vstart_runner: rotate logs between testcases"
Fixes: https://tracker.ceph.com/issues/47032
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2020-08-19 19:25:22 +02:00
Rishabh Dave
bb325e332c vstart_runner.py: rotate logs between testcases
Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-18 14:47:21 +05:30
Rishabh Dave
90ff87b1e8 vstart_runner: run one test method at a time...
instead of running entire testsuite at once; this would allow us more
control on the test runner.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-18 14:46:33 +05:30
Ernesto Puerta
88865cbfb1
mgr/dashboard/api: reduce amount of daemon logs
Fixes: https://tracker.ceph.com/issues/46920
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2020-08-13 13:45:17 +02:00
Xiubo Li
47600028ae qa: defer cleaning the mountpoint's netnses and the bridge
The netnses maybe created/deleted many times in the whole test cases,
we can defer cleaning them untile the last mountpoint is unmounted
or when the test is exiting.

Fixes: https://tracker.ceph.com/issues/46282
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 20:58:36 -04:00
Xiubo Li
c0c5550ed7 qa/tasks/cephfs/mount.py: remove the stale netnses and bridge
If the previous test cases failed, the netnses and bridge will be
left. Here will remove them when new test cases begin.

Fixes: https://tracker.ceph.com/issues/45806
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 08:41:18 +08:00
Kefu Chai
91b76afa20 qa/tasks/vstart_runner: drop py2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-05 10:58:28 +08:00
Jos Collin
63a9ddc57f
qa/tasks: make sh() in vstart_runner.py identical with teuthology.orchestra.remote.sh
Fixes: https://tracker.ceph.com/issues/46069
Signed-off-by: Jos Collin <jcollin@redhat.com>
2020-07-01 17:49:35 +05:30
Patrick Donnelly
4636a1bda8
Merge PR #35664 into master
* refs/pull/35664/head:
	qa: add omit_sudo=False for commands ran with sudo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2020-06-25 14:31:33 -07:00
Patrick Donnelly
b107163146
qa: fix flake8 error
./tasks/vstart_runner.py:33:1: F401 'io.BytesIO' imported but unused

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-24 12:20:33 -07:00
Patrick Donnelly
9436a18686
Merge PR #35665 into master
* refs/pull/35665/head:
	vstart_runner.py: improve a conditional in _do_run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-24 11:42:29 -07:00
Patrick Donnelly
af4d4ee6f1
Merge PR #35522 into master
* refs/pull/35522/head:
	vstart_runner: set default values of stdout and stderr to None

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-24 11:34:04 -07:00
Kefu Chai
663d64bd3b qa/tasks/vstart_runner: flake8 fixes
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-23 23:01:27 +08:00
Rishabh Dave
90727ee474 qa: add omit_sudo=False for commands ran with sudo
Commands that cleanup FUSE and kernel mount and that setup and
teardown/cleanup network namespaces are that ones that use sudo. Set
omit_sudo to False while running these commands.

Fixes: https://tracker.ceph.com/issues/46101
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-19 14:21:04 +05:30
Rishabh Dave
1ff990c2a1 vstart_runner.py: improve a conditional in _do_run
Before checking for "/" in args[0], check if it's an instance of Raw.
Since Raw instances aren't iterable, the execution would crash here.

Fixes: https://tracker.ceph.com/issues/46100
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-19 12:13:57 +05:30
Patrick Donnelly
af98e6afbe
Merge PR #35539 into master
* refs/pull/35539/head:
	vstart_runner: set omit_sudo to True by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-17 09:38:36 -07:00
Rishabh Dave
cc8f15818a vstart_runner: set default values of stdout and stderr to None
Not doing so leads to tests run successfully with vstart_runner.py but
crash when triggered with teuthology since the default values of these
variables there is None.

Fixes: https://tracker.ceph.com/issues/45815
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-17 14:42:53 +05:30
Patrick Donnelly
7952dc58b2
Merge PR #35261 into master
* refs/pull/35261/head:
	qa/vstart_runner.py: ignores when source and destination are same

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2020-06-12 14:08:13 -07:00
Rishabh Dave
e68736961f vstart_runner: set omit_sudo to True by default
Let's set omit_sudo to True by default since we don't to run
commands with sudo on developer's machine as far as possible.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-11 17:05:40 +05:30
Kefu Chai
954e9a6fa6 qa/tasks/vstart_runner: do not teardown test_path if "create-cluster-only"
otherwise we could be removing a "None" directory when tearing down the cluster,
and have following failure:

Exception ignored in: <bound method LocalContext.__del__ of <__main__.LocalContext object at 0x7f99fd4a6cc0>>
Traceback (most recent call last):
  File "../qa/tasks/vstart_runner.py", line 1189, in __del__
    shutil.rmtree(self.teuthology_config['test_path'])
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 477, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/tmp/tmp.mmM2ugspuR/venv/lib/python3.6/shutil.py", line 475, in rmtree
    orig_st = os.lstat(path)
TypeError: lstat: path should be string, bytes or os.PathLike, not NoneType

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-03 15:24:16 +08:00
Patrick Donnelly
182985455a
Merge PR #34838 into master
* refs/pull/34838/head:
	vstart_runner: don't use namespaces by default
	qa/cephfs: run nsenter commands with superuser privileges
	qa/cephfs: look for mountpoint in cmdline file

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-02 12:01:52 -07:00
Patrick Donnelly
963bb62ec9
Merge PR #34782 into master
* refs/pull/34782/head:
	qa/tasks/cephfs/mount.py: remove netns name parsing in mountpoint setter
	qa/tasks/vstart_runner.py: add kwargs parameter to ignore the ones it does not understand

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-02 07:40:39 -07:00
Rishabh Dave
49e9faee67 qa/vstart_runner.py: ignores when source and destination are same
put_file() in vstart_runner.py should ignore the error when source and
destination paths supplied are the same. Source and destination paths
for put_file() method represents different locations for teuthology
since source and destination are on the different machines.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-02 17:29:19 +05:30
Rishabh Dave
a4040bf0d2 vstart_runner: don't use namespaces by default
And add option --usens to vstart_runner.py to use namespaces.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-02 17:10:56 +05:30
Rishabh Dave
04d501d6ff qa/cephfs: run nsenter commands with superuser privileges
And add a method that sets self.fuse_daemon.subproc.pid to the PID of
the process that doesn't have sudo in its arguments. For example, when
"sudo ceph-fuse /mnt/cephfs" is run on the shell, it launches process
with arguments "ceph-fuse /mnt/cephfs". The added method gets PID of
latter/child process and sets that as the fuse daemon's PID. Not doing
so kills the former/parent process but the not the child process.

Also, since we are around cleanup this method a bit.

Fixes: https://tracker.ceph.com/issues/45339
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-06-02 17:07:49 +05:30
Xiubo Li
9f757c18a8 qa/tasks/vstart_runner.py: add kwargs parameter to ignore the ones it does not understand
INFO:__main__:test_replay_create (tasks.cephfs.test_cap_flush.TestCapFlush) ... ERROR
INFO:__main__:Stopped test: test_replay_create (tasks.cephfs.test_cap_flush.TestCapFlush) in 22.251901s
INFO:__main__:
INFO:__main__:======================================================================
INFO:__main__:ERROR: test_replay_create (tasks.cephfs.test_cap_flush.TestCapFlush)
INFO:__main__:----------------------------------------------------------------------
INFO:__main__:Traceback (most recent call last):
INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/cephfs_test_case.py", line 145, in setUp
INFO:__main__:    self.mounts[i].mount_wait()
INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/mount.py", line 420, in mount_wait
INFO:__main__:    mount_options=mount_options)
INFO:__main__:TypeError: mount() got an unexpected keyword argument 'mountpoint'
INFO:__main__:
INFO:__main__:----------------------------------------------------------------------
INFO:__main__:Ran 1 test in 22.253s
INFO:__main__:
INFO:__main__:FAILED (errors=1)
INFO:__main__:

Fixes: https://tracker.ceph.com/issues/45300
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-05-27 21:18:53 -04:00
Rishabh Dave
418dea32c2 vstart_runner: set mounted to True at the end of mount()
Not doing so prevents cleanup methods from running.

Fixes: https://tracker.ceph.com/issues/45723
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-05-27 14:45:59 +05:30
Michael Fritch
44bcee8d1a
qa/tasks: logger.warn() -> logger.warning()
logger.warn() is the same as logger.warning(), but was removed in py3.3

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-05-13 17:18:13 -06:00