Commit Graph

2736 Commits

Author SHA1 Message Date
Kiefer Chang
a9accaeccf mgr/dashboard: support Orchestrator and user-defined Ganesha clusters
This change make the Dashboard support two types of Ganesha clusters:

- Orchestrator clusters (Since Octopus)
  - Deployed by the Orchestrator.
  - The Dashboard gets the pool/namespace that stores Ganesha
    configuration objects from the Orchestrator.
  - The Dashboard gets the daemons in a cluster from the Orchestrator.

- User-defined clusters (Since Nautilus)
  - Clusters defined by using `ceph dashboard
    set-ganesha-clusters-rados-pool-namespace` command is treated as
    user-defined clusters.
  - Each daemon has its own RADOS configuration objects. The
    Dashboard uses these objects to deduce daemons.

Fixes: https://tracker.ceph.com/issues/46492
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2020-10-19 19:11:14 +08:00
Kefu Chai
3ec03efc70
Merge pull request #37175 from changchengx/log_start
common: remove log_early configuration option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-10-19 18:44:29 +08:00
Lenz Grimmer
86f6a89c1b
Merge pull request #37564 from s0nea/wip-dashboard-nfs-daemons-param
mgr/dashboard/qa: consider `daemons` parameter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
2020-10-19 10:42:22 +02:00
Changcheng Liu
dbdcb2535d common: remove log_early configuration option
After deciding to always enable tracking log in early phase, there's no
need to keep "log_early" option here and remove it directly.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
2020-10-19 14:30:28 +08:00
Deepika Upadhyay
7ef18559cb qa: drop hammer branch qa tests
fixes: https://tracker.ceph.com/issues/47731
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-10-15 17:32:06 +05:30
Kefu Chai
1d15e8c2d1
Merge pull request #37624 from lxbsz/cram
qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-10-15 18:53:38 +08:00
Kefu Chai
2724a73743
Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result
qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2020-10-15 18:12:54 +08:00
Patrick Donnelly
f8f607dbd1
qa: print debug info on mount cleanup
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-13 10:16:40 -07:00
Patrick Donnelly
8e7a89009c
qa: remove redundant rmr
The mount.cleanup method will remove the mount point. This `rm -rf` will
always fail (with exit status 0).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-13 10:16:40 -07:00
Patrick Donnelly
2432871a1b
qa: use null mode to prevent undesired changes to mountpoint
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-13 10:16:40 -07:00
Patrick Donnelly
bc25bd70f2
qa: unmount all clients before deleting the file system
Otherwise we have unnecessary timeout waits.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-13 10:16:40 -07:00
Yan, Zheng
e76abf517b qa/cephfs: update ephemeral pin tests
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-10-13 20:52:40 +08:00
Rishabh Dave
f9f4c78101 vstart_runner: omit result line only when testcases are run one by one
vstart_runner omits the result line ("Ran X tests in Y") generated by
unittest unconditionally. Don't do so when vstart_runner triggers entire
test module at once.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-10-13 09:12:45 +05:30
Rishabh Dave
80ecc017e8 vstart_runner: don't quit running testsuite on a test failure
Add an option that makes it possible to not to quit running testsuite on
a test failure. This way user can get a better idea on the current state
of the testsuite and test its own code patches more effectively and
easily.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-10-13 09:12:45 +05:30
Patrick Donnelly
daa2f85156
Merge PR #29951 into master
* refs/pull/29951/head:
	test: add tests for validating MDS metrics via `perf stats` module
	test: Filesystem class helpers to grow and shrink MDS cluster
	mgr/stats: mds performance stats module
	mds: support sending empty perf metrics to ceph-manager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-12 18:50:33 -07:00
Patrick Donnelly
473d8e9a60
Merge PR #37593 into master
* refs/pull/37593/head:
	qa/tasks/cephfs: Refactor test_volumes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-12 07:42:13 -07:00
Xiubo Li
833eca15f2 qa/tasks: add a 'parallel' option support for the cram task
For the ceph-iscsi test case we need to run the tests sequentially,
because the client test will depend on the gateway ones.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-12 21:46:34 +08:00
Tatjana Dehler
1edaa17181
Merge pull request #37154 from bk201/wip-47397
mgr/dashboard: fix the error when exporting CephFS path "/" in NFS exports

Reviewed-by: Laura Paduano lpaduano@suse.com
Reviewed-by: Stephan Müller smueller@suse.com
Reviewed-by: Varsha Rao varao@redhat.com
2020-10-12 14:32:39 +02:00
Venky Shankar
a95b364e3e test: add tests for validating MDS metrics via perf stats module
Fixes: http://tracker.ceph.com/issues/24285
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-10-12 07:34:51 -04:00
Venky Shankar
5afdf26943 test: Filesystem class helpers to grow and shrink MDS cluster
These routines were test specific. Make them a part of Filesystem
class.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-10-12 07:34:51 -04:00
Nick Janus
d70ca81502 rgw: fixes BucketInfo for missing buckets
The admin api BucketInfo endpoint should now return 404 for buckets that
are not found where only the bucket name is passed as a parameter.

Fixes: https://tracker.ceph.com/issues/45193
Signed-off-by: Nick Janus <njanus@digitalocean.com>
2020-10-12 13:50:06 +03:00
Ramana Raja
d1d5d61884 qa/tasks/cephfs: Refactor test_volumes
... into smaller test classes. This enables breaking up the volumes yaml
into smaller yaml fragments.

Fixes: https://tracker.ceph.com/issues/47160
Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-10-12 13:54:40 +05:30
Patrick Donnelly
07ea8f5bca
Merge PR #37441 into master
* refs/pull/37441/head:
	qa: fix proc exit status check

Reviewed-by: Jos Collin <jcollin@redhat.com>
2020-10-09 18:41:24 -07:00
Lenz Grimmer
16e8df214c
Merge pull request #37064 from bk201/wip-47372
mgr/dashboard: fix error when typing existing paths in the Ganesha form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2020-10-09 13:42:30 +02:00
Patrick Donnelly
2cb09bd4bd
Merge PR #36554 into master
* refs/pull/36554/head:
	mgr/volumes: Make number of cloner threads configurable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Shyamsundar R <srangana@redhat.com>
2020-10-08 13:25:15 -07:00
Patrick Donnelly
5c9f77cb65
Merge PR #36457 into master
* refs/pull/36457/head:
	qa/tasks/cephfs: fix fs authorize cmd args
	mds: add root_squash mode in MDS auth caps
	vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-08 13:23:34 -07:00
Tiago Melo
ed237aa203 mgr/dashboard: Disable RBD clone action when conditions are not met
Disable clone action when RBD snapshot is not protected
and clone format version is 1.

Fixes: https://tracker.ceph.com/issues/37873

Signed-off-by: Tiago Melo <tmelo@suse.com>
2020-10-08 16:45:00 +00:00
Tiago Melo
a7ca0631d6 qa/mgr: Remove _wait_for_expected_get_result
Fixes: https://tracker.ceph.com/issues/47793

Signed-off-by: Tiago Melo <tmelo@suse.com>
2020-10-08 15:58:06 +00:00
Varsha Rao
97959d249a qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr
Fixes: https://tracker.ceph.com/issues/47591
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-10-06 22:56:05 +05:30
Tatjana Dehler
032a39e6f4 mgr/dashboard/qa: consider daemons parameter
Consider the values given by the `daemons` parameter
and do not use hard-coded values only.

Fixes: https://tracker.ceph.com/issues/47759
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2020-10-06 10:38:24 +02:00
Kiefer Chang
ee14ef7c60
mgr/dashboard: fix error when typing existing paths in the Ganesha form
- The `CephFS.ls_dir()` implementation in the backend had changed, the code in the
  UI endpoint `/ui-api/nfs-ganesha/lsdir` needs to adapt.
- Add fs_name as resource_id in `/ui-api/nfs-ganesha/lsdir/<fs_name>` to distinguish FS.
- Add more checks and unit tests.

Fixes: https://tracker.ceph.com/issues/47372
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2020-10-06 11:25:55 +08:00
Kefu Chai
fbb2cc49bf
Merge pull request #36732 from rishabh-d-dave/vr-rotate-logs
vstart_runner: rotate logs between testcases

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-04 13:28:39 +08:00
Rishabh Dave
864a15e7ca vstart_runner: rotate logs between testcases optionally
Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-29 14:52:47 +05:30
Rishabh Dave
6673531804 vstart_runner: rearrange exec_test()
Rearrange the code that triggers the test runner so that in the later
commits the way tests are triggered can be changed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-29 14:49:51 +05:30
Patrick Donnelly
eb95dabc3f
qa: fix proc exit status check
Fixes: f30a84b6a7
Fixes: https://tracker.ceph.com/issues/47677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-28 11:03:21 -07:00
Ramana Raja
525c0e699e qa/tasks/cephfs: fix fs authorize cmd args
... in test_admin.TestSubCmdFsAuthorize.setup_for_multiple_paths().

Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-09-25 15:54:12 +05:30
Ramana Raja
e1c5220780 mds: add root_squash mode in MDS auth caps
Implement a root_squash mode in MDS auth caps to deny operations for
clients with uid=0 or gid=0 that need write access. It's mainly to
prevent operations such as accidental `sudo rm -rf /path`.

The root squash mode can be enforced in one of the following ways in
the MDS caps,
  'allow rw root_squash'
  (across file systems)
          or
  'allow rw fsname=a root_squash'
  (on a file system)
          or
  'allow rw fsname=a path=/vol/group/subvol00 root_squash'
  (on a file system path)

Fixes: https://tracker.ceph.com/issues/42451
Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-09-25 14:34:35 +05:30
Ramana Raja
a177b470aa vstart_runner: add write_file and sudo_write_file
... methods in LocalRemote class. These methods are called in some of
the recently added cephfs tests. They were implemented in teuthology's
Remote class, but not in vstart_runner's LocalRemote class. Hence some
cephfs tests couldn't be run locally using vstart_runner without this
change.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-09-25 14:34:35 +05:30
Patrick Donnelly
cc2c25b9fe
Merge PR #37266 into master
* refs/pull/37266/head:
	qa: ignore logrotate state rename error

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-09-24 14:37:08 -07:00
Lenz Grimmer
a433ed1282
Merge pull request #37244 from ceph/47543-bucket-list-perf
mgr/dashboard: fix perf. issue when listing large amounts of buckets

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-09-23 16:26:09 +02:00
Lenz Grimmer
fac25773b6
Merge pull request #37192 from rhcs-dashboard/fix-46898-master
mgr/dashboard/api: move/create OSD histogram in separate endpoint

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-09-23 11:01:06 +02:00
Jason Dillaman
78d630d4a5
Merge pull request #37110 from CongMinYin/qa_immutable_object_cache_check
qa: check immutable object cache content

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-09-22 19:12:42 -04:00
Patrick Donnelly
6547276ec3
Merge PR #37227 into master
* refs/pull/37227/head:
	qa/cephfs: don't pass args to destroy() in recreate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-21 14:04:06 -07:00
Patrick Donnelly
c46bf97646
Merge PR #37233 into master
* refs/pull/37233/head:
	qa/mgr: revert a patch from commit 04ed58f

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-09-21 14:03:32 -07:00
Lenz Grimmer
529ff10c1d
Merge pull request #35785 from rhcs-dashboard/wip-45957-consolidate_Osd_Endpoints
mgr/dashboard: Consolidate Osd mark endpoints

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2020-09-21 13:59:48 +02:00
Patrick Donnelly
f30a84b6a7
qa: ignore logrotate state rename error
This is hopefully a transient issue that can be ignored.

Fixes: https://tracker.ceph.com/issues/42433
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-20 14:05:13 -07:00
Yin Congmin
f18aca8746 qa: check immutable object cache content
fix check immutable cache does not work

Signed-off-by: Yin Congmin <congmin.yin@intel.com>
2020-09-21 01:36:56 +08:00
Patrick Donnelly
269667b3a2
Merge PR #37218 into master
* refs/pull/37218/head:
	qa: spawn MDS daemons before creating fs

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-18 16:25:59 -07:00
Neha Ojha
8ba0a61a51
Merge pull request #35906 from gregsfortytwo/wip-stretch-mode
Add a new stretch mode for 2-site Ceph clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-18 14:31:45 -07:00
Alfonso Martínez
924368e1d0 mgr/dashboard: fix performance issue when listing large amounts of buckets
Fixes: https://tracker.ceph.com/issues/47543
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
2020-09-18 17:16:34 +02:00
Rishabh Dave
f07ccd64d4 qa/mgr: revert a patch from commit 04ed58f
mds_cluster.mds_fail() runs command "mds fail" not "fs fail". The reason
for failure was PR #32581 which accidentally changed the return code
from 0 to EINVAL. Since this was reversed in PR #37159, the change
introduced by 04ed58f is not only incorrect but also redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-18 16:33:41 +05:30
Aashish Sharma
1c812e0d61 mgr/dashboard/api: move/create OSD histogram in separate endpoint
Added a separate endpoint for osd/histogram - api/osd/{svc_id}/histogram

Fixes:https://tracker.ceph.com/issues/46898
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2020-09-18 16:11:04 +05:30
Rishabh Dave
d7ec8d56aa qa/cephfs: don't pass args to destroy() in recreate()
In filesystem.py, don't set value of reset_obj_attrs to False.

Fixes: https://tracker.ceph.com/issues/47526
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-18 14:10:10 +05:30
David Zafman
a97a1cd858
Merge pull request #36397 from dzafman/wip-39012
distinguish unfound + impossible to find, vs start some down OSDs to get

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-17 17:30:43 -07:00
Patrick Donnelly
61db7a9c2e
qa: spawn MDS daemons before creating fs
This avoids unnecessary MDS_ALL_DOWN messages because the MDS daemons
have not yet been spawned.

Fixes: https://tracker.ceph.com/issues/47518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-17 09:26:38 -07:00
Patrick Donnelly
23f9da70a7
Merge PR #37199 into master
* refs/pull/37199/head:
	qa/cephfs: add a note to test_nfs.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-16 16:44:38 -07:00
Rishabh Dave
aa08a38b61 qa/cephfs: add a note to test_nfs.py
A note that tells that test_nfs.py is not yet compatible with
vstart_runner.py.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-16 22:10:42 +05:30
Neha Ojha
ca43c6e8ee
Merge pull request #37167 from neha-ojha/wip-47239
qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-09-16 09:30:11 -07:00
Neha Ojha
e7eddec5a0 qa/tasks/ceph_manager.py: remove redundant check in raw_cluster_cmd_result
Fixes 530982129e. The check for cephadm is no
longer needed since it was moved to run_cluster_cmd.

Fixes: https://tracker.ceph.com/issues/47239
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-09-15 17:56:33 +00:00
Kiefer Chang
5d0d7f8928
mgr/dashboard: fix the error when exporting CephFS path "/" in NFS exports
Fixes: https://tracker.ceph.com/issues/47397
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2020-09-15 17:24:43 +08:00
Kalpesh Pandya
60904c907e qa : STS Testing Files (AssumeRole and GetSessionToken)
Added required files for testing of AssumeRole and GetSessionToken API's and modified s3tests.py to handle the same.

(cherry picked from commit c2c90eaf52)

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2020-09-15 11:58:59 +05:30
Greg Farnum
9506d09e3b Merge remote-tracking branch 'origin/master' into wip-stretch-mode
Conflicts:
	src/include/ceph_features.h

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2020-09-15 02:25:07 +00:00
David Zafman
12b495b370 test: Test list_unfound with asserts to check for new fields
Signed-off-by: David Zafman <dzafman@redhat.com>
2020-09-14 13:06:29 -07:00
David Zafman
4a655868a2 test: Fix typos in descriptions
Signed-off-by: David Zafman <dzafman@redhat.com>
2020-09-14 13:06:29 -07:00
Greg Farnum
d02625331c Merge remote-tracking branch 'origin/master' into wip-stretch-mode 2020-09-14 02:32:19 +00:00
Sebastian Wagner
0f90e60faa
Merge pull request #35543 from sebastian-philipp/qa-cephadm-iscsi
qa/cephadm: Add iSCSI

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Georgios Kyratsas <gkyratsas@suse.com>
Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-09-11 21:10:37 +02:00
Ramana Raja
4b2911fc70
Merge pull request #32581 from rishabh-d-dave/wip-djf-15070
mon, cephfs: Add auth caps for CephFS fsids

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2020-09-12 00:00:52 +05:30
Rishabh Dave
995c73639b qa/cephfs: add tests for "fs authorize" subcommand
Make caps FS-specific affects "fs authorize" subcommand. Let's add few
tests to verify its behaviour.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-11 18:04:23 +05:30
Rishabh Dave
4e0f090861 qa/cephfs: backup mount object before running tests
Right now, only client IDs are stashed and restored but with the recent
changes (addition of more attributes to mount objects, specifically),
this is not enough. Saving and restoring these details before and after
tests respectively ensures that mount commands rus smoothly. Not doing
this typically leads to mount command failure for the second test in the
testsuite under execution since the client IDs are saved and restored in
CephFSTestCase.setUp and CephFSTestCase.tearDown respectively but the
rest of the details are not.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-11 18:04:23 +05:30
Rishabh Dave
352a41d1b3 qa/cephfs: add tests for multi-FS auth tests
Add testsuite for testing authorization on Ceph cluster with multiple
file systems and enable it to be executable with Teuthology framework.

Also add helper methods required to setup the test environment for
multi-FS tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-11 18:02:48 +05:30
Sebastian Wagner
130c7a438e qa/cephadm: Also fetch iscsi logs
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-09-11 14:17:09 +02:00
Sebastian Wagner
430f2c5132 qa/cephadm: Add iSCSI
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-09-11 14:17:09 +02:00
Kefu Chai
75bbdeb0c3
Merge pull request #36211 from kshtsk/wip-remote-write-file
qa/tasks: use remote write file

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-11 17:11:10 +08:00
Kotresh HR
83c4442c76 mgr/volumes: Make number of cloner threads configurable
The number of cloner threads is set to 4 and it can't be
configured. This patch makes the number of cloner threads
configurable via the mgr config option "max_concurrent_clones".

On an increase in number of cloner threads, it will just
spawn the difference of threads between existing number of
cloner threads and the new configuration. It will not cancel
the running cloner threads.

On decrease in number of cloner threads, the cases are as follows.

1. If all cloner threads are waiting for the job:
      In this case, all threads are notified and required number
   threads are terminated.

2. If all the cloner threads are processing a job:
      In this case, the condition is validated for each thread after
   the current job is finished and the thread is termianted if the
   condition for required number of cloner threads is not satisified.

3. If few cloner threads are processing and others are waiting:
      The threads which are waiting are notified to validate the
   number of threads required. If terminating those doesn't satisfy the
   required number of threads, the remaining threads are terminated
   upon completion of existing job.

Fixes: https://tracker.ceph.com/issues/46892
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2020-09-11 14:36:37 +05:30
Rishabh Dave
3f0284f272 qa/cephfs: add methods to read/write on CephFS mounts
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 23:57:15 +05:30
Rishabh Dave
04ed58fec2 qa/cephfs: modify delete_all_filesystems() in filesystem.py
Modify filesystem.Filesystem.delete_all_filesystems() method to make it
more succinct, move it to class MDSCluster instead and update every call
to it accordingly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 23:56:59 +05:30
Rishabh Dave
a7eaec9e26 qa/cephfs: modify recreate() in filesystem.py
Modify cephfs.filesystem.Filesystem.recreate() method to delete only the
FS represented by the object instead of deleting the every FS on the
Ceph cluster.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
ee6e2297a1 qa/cephfs: add a method to destroy fs in filesystem.py
And reset_obj_attrs parameter to it so that the caller of the method can
choose to destroy the Ceph FS represented by the object without
disturbing the object attributes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
9a36a0abd0 qa/cephfs: allow not aborting execution when mount command fails
This commit adds a new argument check_status to mount methods of
KernelMount, FuseMount, LocalKernelMount and LocalFuseMount. When value
of this argument is False, these methods would catch the
CommandFailedError exception and would return a tuple consisting of the
exception itself, and stdout and stderr of the mount command. This
allows reusing these mount methods while running negative tests for
commands.

The name "check_status" is selected so since teuthology's run() and
vstart_runner's run() use a variable with same name for the very same
purpose.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
07e493ffb5 qa/cephfs: allow reusing mount objects and add remount method
This commit introduces following two set of changes -

First, make client keyring path, mountpoint on host FS and CephFS and
CephFS's name attributes of the object representing the mount
and update all the mount object creation calls accordingly. Also,
rewrite all the mount object creation to use keyword arguments instead
of positional arguments to avoid mistakes, especially since a new
argument was added in this commit.

Second, add remount method to mount.py so that it's possible to unmount
safely, modify the attributes of the object representing the mount and
mount again based on new state of the object *in a single call*. The
method is placed in mount.py to avoid duplication.

This change has two leads to two more changes: upgrading interface of
mount() and mount_wait() and upgrading testsuites to adapt to these
change.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Kefu Chai
e5b9b08cc4
Merge pull request #36962 from tchaikov/wip-qa-py3-cleanup
qa: py3 cleanups

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-10 09:39:20 +08:00
Fabrizio D'Angelo
ff92dd4776 mgr/dashboard: Consolidate Osd mark endpoints
Instead of 4 different GET requests, there is now one PUT request
that parses the desired action in the request body.
A description was added to describe the usage.

* Osd flag endpoints were grouped with Osd endpoints
* A few mypy issues were also resolved

Fixes: https://tracker.ceph.com/issues/46181

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit f67d5aa8fa30859d15d314ef5fbb3d4cda87cd5b)
(cherry picked from commit dcc2ceb0a0ce30f67aee8d8096d6e1714d3e56e7)
(cherry picked from commit 5ee7f8c9b731fe249daeec6f114cf6e9c998af66)
2020-09-04 19:31:24 +05:30
Kyr Shatskyy
4eba42382d qa/tasks/restart: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:16 +02:00
Kyr Shatskyy
e5b06d6431 qa/tasks/workunit: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:16 +02:00
Kyr Shatskyy
e510687f1f qa/tasks/devstack: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:16 +02:00
Kyr Shatskyy
5e9cd8624e qa/tasks/cephadm: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:16 +02:00
Kyr Shatskyy
0f54b4784f qa/tasks/ceph_deploy: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
cea546f3b0 qa/tasks/ceph: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
3aaec3fe91 qa/tasks/cephfs: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
122113f9ea qa/tasks/s3a_hadoop: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
7f9c3308ef qa/tasks/s3tests_java: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
9bec5e5f01 qa/tasks/rbd_fio: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
c213caf8e6 qa/tasks/samba: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
36ab658ba7 qa/tasks/ragweed: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
6179f2a431 qa/tasks/qemu: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
bfd5d8b097 qa/tasks/dnsmasq: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
0ea7446b19 qa/tasks/cephfs: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
6a24218a85 qa/tasks/ceph_objectstore_tool: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
b4a0ef0ed0 qa/tasks/ceph: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
85f4747133 qa/tasks/cbt: user remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
ba4aff144e qa/tasks/autotest: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
bdb3c8a23d qa/tasks/cephadm: deprecate misc.sudo_write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
a44f2ff646 qa/tasks/devstack: use remote.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
552e6a771b qa/tasks/s3tests: use remote.write_file instead of misc.write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
af0f2d20eb qa/tasks/cephfs: use remote.write_file instead of misc.sudo_write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
e80f66c14a tasks.ceph_deploy: deprecate misc.append_lines_to_file
Use remote context orchestra.remote.sudo_write_file() method
instead of misc.append_lines_to_file()

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Kyr Shatskyy
442805277f tasks.ceph_deploy: deprecate misc.sudo_write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Patrick Donnelly
465eba6d14
Merge PR #36546 into master
* refs/pull/36546/head:
	vstart_runner: log commands in a more usable form

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-09-03 10:08:42 -07:00
Sebastian Wagner
71da1d457e
Merge pull request #36894 from pcuzner/device-ls-with-lsm
orchestrator: intergrate LSM information with device ls
2020-09-03 14:45:06 +02:00
Kefu Chai
eda90040ad qa: always use subprocess.{DEVNULL,check_output}
no need to check for their existence, and prepare a replacement.
because we've migrated to python3. and we only support python3.6 and up.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-03 13:09:16 +08:00
Paul Cuzner
8691f94104 mgr/orchestrator: add test command testcase for "device ls"
Quick test to ensure changes that break wide mode are picked
up

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2020-09-03 14:53:22 +12:00
Rishabh Dave
464ec3dbdc vstart_runner: log commands in a more usable form
Currently commands are printed as ['arg1', 'arg2', 'arg3']. Instead, log
them as '> arg1 arg2 arg3' so that it's simpler to copy and run them
manually. The reason behind prepending '> ' to these logs entries is
just to follow the practice followed by teuthology logs.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-02 22:52:27 +05:30
Lenz Grimmer
4e71a942ce
Merge pull request #35802 from rhcs-dashboard/wip-46160-fix_cephfs_api_inconsistency
mgr/dashboard: Address API inconsistencies in Cephfs endpoint

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-09-02 12:29:59 +02:00
Lenz Grimmer
9e76d73a13
Merge pull request #34644 from rhcs-dashboard/26954-histogram
mgr/dashboard: Remove Histogram Component

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2020-09-02 08:28:49 +02:00
Fabrizio D'Angelo
871df3671c mgr/dashboard: Address API inconsistencies
Cephfs endpoints are inconsistencies with other endpoints.

* Change rm endpoints from GET to DELETE requests
* Change set_quotas from POST to PUT
* Renames endpoints accordingly
* Changes related angular files
* update a few lines related to mypy
* remove future import as the python2 compatibility is not needed

Fixes: https://tracker.ceph.com/issues/46160
Signed-off-by: Fabrizio D'Angelo <fdangelo@redhat.com>
(cherry picked from commit 222286b08bde3600d50a6deb25e7425c17e9e482)
2020-08-31 23:45:49 +05:30
Patrick Donnelly
d33c6157a3
qa: test for statfs on deleted file system
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-29 10:22:12 -07:00
Patrick Donnelly
72d6f0413e
qa: move run_shell args to kwargs
To avoid listing all run args.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-29 10:22:12 -07:00
Kefu Chai
786b87cdb2
Merge pull request #36718 from ShraddhaAg/add-dir-to-archive-in-logs
qa/tasks/ceph.py: add ceph logs directory in job's info.yaml

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
2020-08-27 22:26:01 +08:00
Laura Paduano
836e1d4202
Merge pull request #36736 from ceph/47052-fix-base-uri-api-tests
mgr/dashboard: fix API tests: Invalid URL: None/api/auth

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-08-27 14:27:08 +02:00
anurag
ff7c7ae40d mgr/dashboard: Remove histogram component
This commit is intended for code cleanup of Histogram component.

Fixes: https://tracker.ceph.com/issues/26954
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2020-08-27 14:10:42 +05:30
Patrick Donnelly
d2a3fcc728
Merge PR #36773 into master
* refs/pull/36773/head:
	mgr/volumes: Prevent subvolume recreate if trash is not-empty
	mgr/volumes: Disallow subvolume group level snapshots
	mgr/volumes: Add test case to ensure subvolume is marked
	mgr/volumes: handle idempotent subvolume marks
	mgr/volumes: Tests amended and added to ensure subvolume trash functionality
	mgr/volumes: Mark subvolume root with the vxattr ceph.dir.subvolume
	mgr/volumes: Move incarnations for v2 subvolumes, to subvolume trash
	mgr/volumes: maintain per subvolume trash directory
	mgr/volumes: make subvolume_v2::_is_retained() object property

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-26 10:46:07 -07:00
Shyamsundar Ranganathan
00e4ab054f mgr/volumes: Prevent subvolume recreate if trash is not-empty
Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-08-26 13:41:33 -04:00
Shyamsundar Ranganathan
f97e57c28c mgr/volumes: Disallow subvolume group level snapshots
Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-08-26 13:41:26 -04:00
Shyamsundar Ranganathan
3527711a4a mgr/volumes: Add test case to ensure subvolume is marked
with the vxattr ceph.dir.subvolume set to true.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-08-26 13:41:21 -04:00
Shyamsundar Ranganathan
88cffe6ea9 mgr/volumes: Tests amended and added to ensure subvolume trash functionality
Amended a few test cases to ensure created subvolumes and snaps
are removed, and trash stays empty at the end of the test.

Further added one test case for create errors in a retained
v2 subvolume, to ensure metadata is sane, and created incarnation
is not present.

Fixes: https://tracker.ceph.com/issues/47154
Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-08-26 13:41:08 -04:00
Neha Ojha
fb44c2f500
Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist
Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2020-08-25 12:24:36 -07:00
Sebastian Wagner
f82058f06b
Merge pull request #36738 from ShraddhaAg/cephadm-archive-logs
qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-08-25 15:18:58 +02:00
Patrick Donnelly
d83dc481dc
Merge PR #36684 into master
* refs/pull/36684/head:
	qa/tasks/nfs: Test mounting of export created with nfs command
	qa/tasks/nfs: Add helper method to check nfs cluster status
	qa/tasks/nfs: Cleanup created filesystem
	qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-24 13:18:13 -07:00
Patrick Donnelly
c4f5de0058 *: convert EBLACKLISTED to EBLOCKLISTED
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-24 19:53:08 +00:00
Sage Weil
2ee9365d0b qa: log-whitelist -> log-ignorelist
Signed-off-by: Sage Weil <sage@newdream.net>
2020-08-24 19:53:08 +00:00
Sage Weil
dfd01d7653 blacklist -> blocklist
Signed-off-by: Sage Weil <sage@newdream.net>
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-08-24 19:53:08 +00:00
Patrick Donnelly
02813988a6
Merge PR #36394 into master
* refs/pull/36394/head:
	qa: add method run ceph cluster command with better interface
	cephfs: allow only "good" characters

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-24 11:33:03 -07:00
Kefu Chai
103fd4af21
Merge pull request #36760 from tchaikov/wip-qa-optional-args
qa/tasks/workunit: allow passing optional args to workunit

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-24 20:03:04 +08:00
Varsha Rao
f56a83f806 qa/tasks/nfs: Test mounting of export created with nfs command
Fixes: https://tracker.ceph.com/issues/46989
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-24 17:04:45 +05:30
Varsha Rao
5d6742b6b4 qa/tasks/nfs: Add helper method to check nfs cluster status
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-24 17:04:40 +05:30
Varsha Rao
e94a250cb0 qa/tasks/nfs: Cleanup created filesystem
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-24 17:00:42 +05:30
Varsha Rao
91e89a4df5 qa/tasks/nfs: Remove unused port status function and 'stdin' keyword argument
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-24 17:00:42 +05:30
Kefu Chai
12212a40db
Merge pull request #36751 from tchaikov/wip-redirect-stderr
qa/tasks/ceph: create a log file before redirecting to it

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-22 16:59:14 +08:00
Kefu Chai
56c86bbb23 qa/tasks/workunit: allow passing optional args to workunit
* add comment to _run_tests()
* use `os.path.commonpath()` instead using string matching directly
  for matching given workunit spec with executables.
* allow passing optional args to workunit

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-22 15:26:54 +08:00
Kefu Chai
e6eabeeeb2 qa/tasks/ceph: create a log file before redirecting to it
as it is shell who interprets ">>" and redirect the stderr to given
file, but the shell process is launched by ubuntu:ububunt without using
sudo, so the command fails with "Permission denied" failure. to address
this issue, in this change, a file with proper priviledges is created
beforehand using `install`, so shell is able to write to it.

also, instead of creating this file in `maybe_redirect_stderr()`, it
returns the command to create the log file.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-22 11:33:12 +08:00
Patrick Donnelly
3fb857cd04
Merge PR #36131 into master
* refs/pull/36131/head:
	doc: document cephfs mirroring dev work
	test: add tests for `ceph fs mirror` family of commands
	mds: track filesystem mirror peers in fsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-21 18:40:42 -07:00
Rishabh Dave
530982129e qa: add method run ceph cluster command with better interface
This new method should allow better control on the process launched by
the passed command. This is achieved by allowing arguments provided by
teuthology.orchestra.run.run().

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-21 22:16:21 +05:30
Alfonso Martínez
28ecbbde65 mgr/dashboard: fix API tests: Invalid URL: None/api/auth
Fixes: https://tracker.ceph.com/issues/47052
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
2020-08-21 17:22:21 +02:00
Sebastian Wagner
fc9da48226
Merge pull request #36740 from varshar16/wip-fix-nfs-bug-47009
qa/tasks/nfs: Change the mds name and delete created nfs clusters

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.
2020-08-21 09:58:12 +02:00
Kefu Chai
ec73fea99e
Merge pull request #36680 from tchaikov/wip-mgr-disk-predict
pybind/mgr/diskprediction_cloud: remove mgr-diskprediction-cloud

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-08-21 09:46:54 +08:00
Varsha Rao
5c4e659067 qa/tasks/nfs: Instead of 'orch ls' use 'orch ps'
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-20 15:18:31 +00:00
Varsha Rao
049afe5a7e qa/tasks/nfs: Delete nfs cluster after exports are created
Fixes: https://tracker.ceph.com/issues/47009
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-20 14:27:25 +00:00
Ramana Raja
f84d22832a
Merge pull request #35756 from ShyamsundarR/tr-46163
mgr/volumes: Use snapshot root directory attrs when creating clone root

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
2020-08-20 16:53:11 +05:30
Rishabh Dave
4022e2ba7d cephfs: allow only "good" characters
Allow only [A-Za-z0-9-_.] characters for FS, volume, subvolume and
subvolume group names and add test for the same.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-20 15:34:02 +05:30
Ernesto Puerta
64ec1112bb
Revert "vstart_runner: rotate logs between testcases"
Fixes: https://tracker.ceph.com/issues/47032
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2020-08-19 19:25:22 +02:00
Shraddha Agrawal
d2ff5c5eb6 qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml
This commit adds the file path of ceph log directories to the job's
info.yaml log file. The motivation behind this is, in case of job
timeout, the logs would still be tranferred to teuthology host
before nuking test machines using these ceph log directory paths in
job's info.yaml log file.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
2020-08-19 17:52:06 +05:30
Shraddha Agrawal
e991f04b59 qa/tasks/ceph.py: add ceph logs directory in job's info.yaml
This commit adds the file path of ceph log directories to the job's
info.yaml log file. The motivation behind this is, in case of job
timeout, the logs would still be tranferred to teuthology host
before nuking test machines using these ceph log directory paths in
job's info.yaml log file.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
2020-08-19 17:34:26 +05:30
Kefu Chai
dc977d543d qa/tasks/mgr: drop commented code
the test for diskprediction_cloud is never enabled, and the used
cloud-based service is not reachable anymore. let's just remove the dead
code.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-19 11:08:39 +08:00
Rishabh Dave
bb325e332c vstart_runner.py: rotate logs between testcases
Fixes: https://tracker.ceph.com/issues/46059
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-18 14:47:21 +05:30
Rishabh Dave
90ff87b1e8 vstart_runner: run one test method at a time...
instead of running entire testsuite at once; this would allow us more
control on the test runner.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-08-18 14:46:33 +05:30
Ernesto Puerta
88865cbfb1
mgr/dashboard/api: reduce amount of daemon logs
Fixes: https://tracker.ceph.com/issues/46920
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2020-08-13 13:45:17 +02:00
Venky Shankar
e2c5abb85d test: add tests for ceph fs mirror family of commands
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-08-12 09:28:12 -04:00
Shyamsundar Ranganathan
0eaf9b0bd2 mgr/volumes: Use snapshot root directory attrs when creating clone root
If a subvolumes mode or uid/gid values are changed post a snapshot,
and a clone of a snapshot prior to the change is initiated, the clone
inherits the current source subvolumes attributes, rather than the
snapshots attributes.

Fixing this by using the snapshots subvolume root attributes to create
the clone subvolumes root.

Following attributes are picked from the source subvolume snapshot:
- uid, gid, mode, data pool, pool namespace, quota

Fixes: https://tracker.ceph.com/issues/46163

Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-08-10 20:15:45 -04:00
Varsha Rao
bba2d66176 qa/tasks/cephfs/nfs: Add tests for cluster config set and reset
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-10 10:54:31 +00:00
Kefu Chai
9e2f494cf4
Merge pull request #36499 from tchaikov/wip-redirect-stderr
qa/tasks/ceph: redirect stderr for crimson flavor

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-08-10 17:34:44 +08:00
Lenz Grimmer
c6fdae1842
Merge pull request #36423 from bk201/wip-46812
mgr/dashboard: fix tasks.mgr.dashboard.test_rbd.RbdTest.test_move_image_to_trash error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2020-08-07 13:53:06 +02:00
Patrick Donnelly
f4fc138849
qa: add tests for mds_min_caps_working_set
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:03 -07:00
Patrick Donnelly
edc5c14d1c
qa: use config_set/config_get
It's simpler and does not require MDS restarts.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:03 -07:00
Patrick Donnelly
ac6c150eb0
qa: do not append file names to dirname
Otherwise the files generated are not actually under the sub-directory!
This is correcting a confusing aspect of the test infrastructure but
doesn't actually require any changes to the tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:02 -07:00
Patrick Donnelly
dbc44e5e34
qa: add exception for test timeouts
To make this easier to catch. It is still a RuntimeError so it should
not affect current tests by default.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 11:21:54 -07:00
Kefu Chai
e0620eefbd qa/tasks/ceph: redirect stderr for crimson flavor
we should redirect stderr for crimson instead for default flavor. this
change addresses a regression introduced by
da76f46461

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-07 01:11:57 +08:00
Kiefer Chang
d208d67f9f
mgr/dashboard: fix tasks.mgr.dashboard.test_rbd.RbdTest.test_move_image_to_trash error
The API call is a task and the response status is determined by whether
the call is completed within a pre-defined duration (2 seconds) or not.
We should also allow the status when the call takes longer.

Fixes: https://tracker.ceph.com/issues/46812
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2020-08-04 10:12:14 +08:00
Patrick Donnelly
bc7571cc7e
Merge PR #36155 into master
* refs/pull/36155/head:
	qa: Fix traceback during fs cleanup between tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-07-31 14:15:22 -07:00
Patrick Donnelly
ca7be68eff
Merge PR #35944 into master
* refs/pull/35944/head:
	qa: defer cleaning the mountpoint's netnses and the bridge
	qa/tasks/cephfs/mount.py: remove the stale netnses and bridge
	qa/tasks/cephfs/mount.py: try to flush the stale ceph-brx dev info
	qa/tasks/cephfs/mount.py: switch to run_shell_payload() helper
	qa/tasks/cephfs/mount.py: clean up the none used code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-07-31 14:12:53 -07:00
Patrick Donnelly
cd8457cbfd
Merge PR #35647 into master
* refs/pull/35647/head:
	mgr/volumes: Add documentation regarding --retain-snapshots option
	mgr/volumes: Avoid trashing retained subvolume on create errors
	mgr/volumes: Add subvolume v2 test cases
	mgr/volumes: Derive v2 from v1 to leverage common methods
	mgr/volumes: Introduce v2 subvolumes
	mgr/volumes: Use operation type during subvolume open

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
2020-07-31 14:11:27 -07:00
Kefu Chai
47d2822969
Merge pull request #36337 from tchaikov/wip-qa-ceph-waiting
qa/tasks/ceph: do not print out empty list of pg

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-07-31 20:32:13 +08:00
Sebastian Wagner
d688926b8a
Merge pull request #36338 from pcuzner/fix-daemon-actions
mgr/cephadm: fix call to cephadm for daemon restarts etc

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2020-07-31 13:05:18 +02:00
Kefu Chai
2225deb989
Merge pull request #36376 from tchaikov/wip-crimson-log
qa/tasks/ceph: redirect stderr to log file

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-07-31 17:45:35 +08:00
Kefu Chai
87e61e1d6e qa/tasks/ragweed: always set ragweed_repo
* extract get_ragweed_branch() out of download() task, for better
  readablity.
* use a loop for retry when the first clone fails
* drop the `raise ValueError()` clause as it never happens. we could use
  an assert() here, but i don't think it is necessary anyway.
* use sh() instead of run() for better readablity.
* always set ragweed_repo. before this change this variable is
  unbounded if `force-branch` is set.

Fixes: https://tracker.ceph.com/issues/46771
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-31 13:08:56 +08:00
Kefu Chai
da76f46461 qa/tasks/ceph: redirect stderr to log file
crimson write log to stderr, let's redirect it to log file for a more
peaceful teuthology.log.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-31 00:51:30 +08:00
Kotresh HR
381925d219 qa: Fix traceback during fs cleanup between tests
The 'mon_allow_pool_delete' option is set to 'True'
in 'setUp' of 'TestVolumes' and is cleared of in
corresponding 'tearDown' function. Hence, any pool
deletion in parent classes such as 'CephFSTestCase'
would fail. This patch fixes the same by setting
the config 'mon_allow_pool_delete' option in the
'CephFSTestCase'.

Fixes: https://tracker.ceph.com/issues/46597
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2020-07-30 13:09:24 +05:30
Patrick Donnelly
cd8571ea4a
Merge PR #36064 into master
* refs/pull/36064/head:
	mgr/volumes: Fix traceback of ops when volume doesn't exist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-07-29 19:58:16 -07:00
Shyamsundar Ranganathan
63599e27f5 mgr/volumes: Add subvolume v2 test cases
Added test cases for v2 subvolumes

Signed-off-by: Shyamsundar Ranganathan <srangana@redhat.com>
2020-07-29 21:14:40 -04:00
Xiubo Li
47600028ae qa: defer cleaning the mountpoint's netnses and the bridge
The netnses maybe created/deleted many times in the whole test cases,
we can defer cleaning them untile the last mountpoint is unmounted
or when the test is exiting.

Fixes: https://tracker.ceph.com/issues/46282
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 20:58:36 -04:00
Paul Cuzner
5ff0a1d2da mgr/cephadm: add qa tests for daemon restart
Added tests to cove daemon stop/start and restart

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2020-07-30 09:02:04 +12:00
Kefu Chai
45c3bed0ea
Merge pull request #36232 from mgfritch/cephadm-ok-to-stop
mgr/cephadm: add `orch ok-to-stop` commands

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
2020-07-29 20:29:01 +08:00
Kefu Chai
735aa04105 qa/tasks/ceph: do not print out empty list of pg
we could have following logging messages:

tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['2.6', '2.5', '1.0', '2.4'] to go clean and/or [] to split/merge

if the cluster has non-active+clean pgs when the "ceph" is about to
end. but this message is a little bit confusing in the sense it
lists "[]" in it.

in this change, only PGs being waited are listed. also, added some
cleanups:

* use "else" to check if the loop is terminated by a break
* remove "0" from the range() call

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-29 15:07:26 +08:00
Xiubo Li
c0c5550ed7 qa/tasks/cephfs/mount.py: remove the stale netnses and bridge
If the previous test cases failed, the netnses and bridge will be
left. Here will remove them when new test cases begin.

Fixes: https://tracker.ceph.com/issues/45806
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 08:41:18 +08:00
Xiubo Li
438aaf851f qa/tasks/cephfs/mount.py: try to flush the stale ceph-brx dev info
Once we have run the test cases and the ceph-brx bridge is setup,
it will save the config in "/etc/sysconfig/network-scripts/ifcfg-ceph-brx"
or somewhere else. It will be kept after the ceph-brx bridge removed.
So next time once the ceph-brx bridge is created or added, it will
read the config from it, then when we config it again we will get
error like:

    "RTNETLINK answers: File exists"

Here we need to flush it before config it.

Fixes: https://tracker.ceph.com/issues/45817
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 08:41:02 +08:00
Xiubo Li
f8d8e3083a qa/tasks/cephfs/mount.py: switch to run_shell_payload() helper
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-07-29 08:40:43 +08:00
Michael Fritch
60b99dcdc7
mgr/orch: add errno to OrchestratorError
add errno to OrchestratorError and ServiceSpecValidationError exceptions

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-07-28 15:54:46 -06:00
Tatjana Dehler
739b365a3f mgr/dashboard: wait longer for health status to be cleared
Because of reasons the cluster needs more time to recover from
HEALTH_WARN while changes are made by `test_pool_update_metadata`.
Lets wait several times for the cluster status to be HEALTH_OK
again.

Fixes: https://tracker.ceph.com/issues/46573
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2020-07-28 14:47:27 +02:00
Varsha Rao
801047f6cb qa/tasks/nfs:Add test for relative and just '/' pseudo path
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-07-23 13:03:38 +05:30
Laura Paduano
63a56b4b52
Merge pull request #35694 from rhcs-dashboard/wip-45901-master
mgr/dashboard: increase API test coverage in API controllers

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-07-22 16:58:44 +02:00
Kefu Chai
eb4e5de14e
Merge pull request #35866 from tchaikov/wip-qa-tempest-pin-openstackclients
qa/keystone: pin python-openstackclient

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-07-22 20:38:54 +08:00
Kefu Chai
75570ce6a4 qa/keystone: pin python-openstackclient and osc-lib
keystone's dependencies are installed using its tox.ini,
which in turn uses a constraints file of
https://releases.openstack.org/constraints/upper/ussuri,
and it pins cliff to 3.1.0, which is not able to fulfill the requirement
of osc-lib 2.2.0. as it needs needs cliff>=3.2.0. per
https://releases.openstack.org/ussuri/, the latest osc-lib for
ussuri is 2.0.0. and osc-lib>=2.0.0 is required by
python-openstackclient 2.5.1, so let's use it instead of using the latest
one.

if we install cliff==3.1.0 along with python-openstackclient==5.2.1,
we will have following error, as `CommandManager.add_command_group()`
method was added to cliff in 3.2.0. see
8477c4dbd0,
so cliff failed to work with the latest openstackclient, like:

2020-06-29T17:26:23.402 INFO:teuthology.orchestra.run.smithi039.stderr:'CommandManager' object has no attribute 'add_command_group'
2020-06-29T17:26:23.402 INFO:teuthology.orchestra.run.smithi039.stderr:Traceback (most recent call last):
2020-06-29T17:26:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/cliff/app.py", line 264, in run
2020-06-29T17:26:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:    self.initialize_app(remainder)
2020-06-29T17:26:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/openstackclient/shell.py", line 133, in
initialize_app
2020-06-29T17:26:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:    super(OpenStackShell, self).initialize_app(argv)
2020-06-29T17:26:23.403 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/osc_lib/shell.py", line 442, in initialize_app
2020-06-29T17:26:23.404 INFO:teuthology.orchestra.run.smithi039.stderr:    self._load_plugins()
2020-06-29T17:26:23.404 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/openstackclient/shell.py", line 104, in
_load_plugins
2020-06-29T17:26:23.404 INFO:teuthology.orchestra.run.smithi039.stderr:    self.command_manager.add_command_group(cmd_group)
2020-06-29T17:26:23.404 INFO:teuthology.orchestra.run.smithi039.stderr:AttributeError: 'CommandManager' object has no attribute 'add_command_group'
2020-06-29T17:26:23.404 INFO:teuthology.orchestra.run.smithi039.stderr:Traceback (most recent call last):
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/osc_lib/shell.py", line 134, in run
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:    ret_val = super(OpenStackShell, self).run(argv)
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/cliff/app.py", line 264, in run
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:    self.initialize_app(remainder)
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/openstackclient/shell.py", line 133, in
initialize_app
2020-06-29T17:26:23.405 INFO:teuthology.orchestra.run.smithi039.stderr:    super(OpenStackShell, self).initialize_app(argv)
2020-06-29T17:26:23.406 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/osc_lib/shell.py", line 442, in initialize_app
2020-06-29T17:26:23.406 INFO:teuthology.orchestra.run.smithi039.stderr:    self._load_plugins()
2020-06-29T17:26:23.406 INFO:teuthology.orchestra.run.smithi039.stderr:  File "/home/ubuntu/cephtest/keystone/.tox/venv/lib/python3.6/site-packages/openstackclient/shell.py", line 104, in
_load_plugins
2020-06-29T17:26:23.406 INFO:teuthology.orchestra.run.smithi039.stderr:    self.command_manager.add_command_group(cmd_group)
2020-06-29T17:26:23.406 INFO:teuthology.orchestra.run.smithi039.stderr:AttributeError: 'CommandManager' object has no attribute 'add_command_group'

in this change the openstackclients version is pin'ed to the
latest stable of 5.2.1. will have a separated PR to bump up
the cliff version on teuthology side.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-22 19:20:00 +08:00
Kefu Chai
6048c0320f
Merge pull request #36198 from tchaikov/wip-test-drop-py2
rpm,deb,qa,python-common,test: drop python2 support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-07-22 17:38:01 +08:00
Kefu Chai
24e7acc261 qa/tasks: drop python2 support from rgw related tests
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-22 11:55:49 +08:00
Kefu Chai
b36f98d388 qa/tasks/samba: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-22 11:55:49 +08:00
Kefu Chai
4aa86f7a76 qa/tasks/rbd: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-22 11:55:49 +08:00
Kefu Chai
877a0cd871 qa/tasks/mgr/dashboard: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-22 11:55:49 +08:00
Kefu Chai
56d2ae8ccf qa/tasks/fs: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-21 19:52:11 +08:00
Kefu Chai
4aa537b739 qa/tasks: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-21 19:52:11 +08:00
Kefu Chai
76c4e68be4 qa/tasks/cephfs: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-21 19:52:11 +08:00
Kefu Chai
2a377b01fd qa/tasks/cephadm: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-21 19:52:11 +08:00