When the mds revoking the Fwbl caps, the clients need to flush
the dirty data back to the OSDs, but the flush may make the OSDs
to be overloaded and slow, which may take more than 60 seconds to
finish. Then the MDS daemons will report the WRN messages.
For the teuthology test cases, let's just increase the timeout
value to make it work.
Fixes: https://tracker.ceph.com/issues/47565
Signed-off-by: Xiubo Li <xiubli@redhat.com>
In filesystem.py, don't set value of reset_obj_attrs to False.
Fixes: https://tracker.ceph.com/issues/47526
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Add testsuite for testing authorization on Ceph cluster with multiple
file systems and enable it to be executable with Teuthology framework.
Also add helper methods required to setup the test environment for
multi-FS tests.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Modify filesystem.Filesystem.delete_all_filesystems() method to make it
more succinct, move it to class MDSCluster instead and update every call
to it accordingly.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Modify cephfs.filesystem.Filesystem.recreate() method to delete only the
FS represented by the object instead of deleting the every FS on the
Ceph cluster.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
And reset_obj_attrs parameter to it so that the caller of the method can
choose to destroy the Ceph FS represented by the object without
disturbing the object attributes.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
This commit introduces following two set of changes -
First, make client keyring path, mountpoint on host FS and CephFS and
CephFS's name attributes of the object representing the mount
and update all the mount object creation calls accordingly. Also,
rewrite all the mount object creation to use keyword arguments instead
of positional arguments to avoid mistakes, especially since a new
argument was added in this commit.
Second, add remount method to mount.py so that it's possible to unmount
safely, modify the attributes of the object representing the mount and
mount again based on new state of the object *in a single call*. The
method is placed in mount.py to avoid duplication.
This change has two leads to two more changes: upgrading interface of
mount() and mount_wait() and upgrading testsuites to adapt to these
change.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
Before export (and ephemeral) pinned subtrees are stuck in cache
forever.
Add qa test for checking export pinned directories can be trimmed.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Pretty print output once. Use --format=json so the stdout on teuthology
is not pretty printed, taking hundreds of lines.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
this change partially reverts e46eb8348e.
xattrs could contain non-utf8 encoded data, and should be captured using
BytesIO. moreover, it will be fed to `ceph-dencoder`, which expects
binary when performing "import".
Signed-off-by: Kefu Chai <kchai@redhat.com>
collect the keys instead of filtering a dict,
to address following failure:
```
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 29, in test_files_throttle
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: self._test_throttling(self.FILES_THROTTLE)
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 96, in _test_throttling
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: return self._do_test_throttling(throttle_type)
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 176, in _do_test_throttling
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: mds_max_purge_ops = int(self.fs.get_config("mds_max_purge_ops", 'mds'))
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/filesystem.py", line 285, in get_config
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: service_id = random.sample(filter(lambda i: self.mds_daemons[i].running(), self.mds_daemons), 1)[0]
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/virtualenv/lib/python3.5/random.py", line 311, in sample
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: raise TypeError("Population must be a sequence or set. For dicts, use list(d).")
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:TypeError: Population must be a sequence or set. For dicts, use list(d).
```
Signed-off-by: Kefu Chai <kchai@redhat.com>
in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer index.
there are three use cases when we access these views in python3:
1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
the view
4. iterate thru the view
in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.
in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.
in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.
in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This provides a generic framework for modifying Ceph configuration
changes in tests through the monitors rather than the asok interface or
local ceph.conf changes. Any changes are reverted during test teardown.
A future patch will convert existing tests manipulating the local
ceph.conf or admin socket.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
* refs/pull/31494/head:
qa: force creation of fs with EC default data pool
qa: add tests for adding EC data pools
mon/MDSMonitor: warn when creating fs with default EC data pool
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
In the future, we should add the EC data pool as a supplementary data
pool but that requires a mount to setup which is awkward in the code
here. When cephfs-shell is more widely available, this will be easier.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
To be able to catch problems with python2 *and* python3, run flake8
with both versions. From the flake8 homepage:
It is very important to install Flake8 on the correct version of
Python for your needs. If you want Flake8 to properly parse new
language features in Python 3.5 (for example), you need it to be
installed on 3.5 for Flake8 to understand those features. In many
ways, Flake8 is tied to the version of Python on which it runs.
Also fix the problems with python3 on the way.
Note: This requires now the six module for teuthology. But this is
already an install_require in teuthology itself.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
In the test
test_cephfs_shell.TestMisc.test_issue_cephfs_shell_cmd_at_invocation,
don't set the shell option with in the command to be tested since
CephFS shell can now read it's value from ceph.conf.
Also, add methods to set config options for vstart as well teuthology
testing.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
* refs/pull/30816/head:
qa: use small default pg count for CephFS pools
*: remove config of "mon pg warn min per osd" in testing
Reviewed-by: Laura Paduano <lpaduano@suse.com>
The pg count needs to be a power-of-two since
dff5697464.
Also, mon_pg_warn_min_per_osd is disabled by default now (or set to a
low value in vstart/testing) so there's no need to base the pg count on
this value.
Ideally someday we can remove this so that the default cluster value is
used but we need to keep this for deployments of older versions of Ceph.
Fixes: https://tracker.ceph.com/issues/42228
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
* refs/pull/26012/head:
qa: add test that down fs does not ERR
mon/MDSMonitor: skip offline ERR for down fs
Reviewed-by: Douglas Fuller <dfuller@redhat.com>
