Commit Graph

3260 Commits

Author SHA1 Message Date
Sage Weil
99cdaaba70 qa/tasks/ceph_manager: parallelize flush_pg_stats()
Signed-off-by: Sage Weil <sage@newdream.net>
2021-11-25 07:52:55 -06:00
Matt Benjamin
0046803534 qa/rgw: use local runner with cmdline radosgw_admin.py
Restore ability to run radosgw_admin.py unit standalone--improved
to use vstart_runner hooks.

Local rgwadmin(...) wrapper suggested as a cleanup in review by Casey.

Fixes: https://tracker.ceph.com/issues/52837

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2021-11-17 11:22:32 -05:00
Laura Flores
92fcfbb464
Merge pull request #43411 from ljflores/wip-mgr-command-cleanup
mon: simplify 'mgr module ls' output
2021-11-10 14:09:51 -06:00
Patrick Donnelly
b4980dd1ed
Merge PR #43767 into master
* refs/pull/43767/head:
	qa: increase the timeout value to wait a litte longer

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-11-10 14:00:19 -05:00
Venky Shankar
3a4dd30a1e test: add cephfs_mirror thrasher
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-11-09 00:08:05 -05:00
Venky Shankar
087d7aa8ca tasks/cephfs_mirror: optionally run in foreground
cephfs mirror damon thrasher needs to send SIGTERM to mirror
daemons. The mirror daemon needs to run in foreground for
it to receive signal via `daemon.signal`.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-11-09 00:08:05 -05:00
Patrick Donnelly
93cdc800e2
Merge PR #43666 into master
* refs/pull/43666/head:
	qa/vstart_runner: add "managers" to LocalContext instances

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-11-04 16:54:14 -04:00
Patrick Donnelly
373b750bfe
Merge PR #43638 into master
* refs/pull/43638/head:
	qa: pass subdir arg when executing workunit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-11-04 16:53:20 -04:00
Patrick Donnelly
10d8c7a4a5
Merge PR #43613 into master
* refs/pull/43613/head:
	qa: lengthen health warning wait

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2021-11-04 16:52:23 -04:00
Sage Weil
d87d2bd146 Merge PR #43611 into master
* refs/pull/43611/head:
	doc/mgr/nfs: document rgw user and bucket exports
	PendingReleaseNotes: add note about nfs CLI change(s)
	qa/suites/orch/cephadm/smoke-roleless: add rgw user nfs export case
	mgr/nfs: take user-id and/or bucket for 'nfs export create rgw'
	mgr/nfs: reorder 'nfs export creage rgw' arguments
	mgr/nfs: reorder 'nfs export create cephfs' arguments
	mgr/nfs: use keyword args for 'nfs export create rgw'
	mgr/nfs: document and use keyword args for 'nfs export create cephfs'
	qa/tasks/cephfs/test_nfs: use keyword args
	pybind/ceph_argparse: handle misordered keyword arguments

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-11-04 14:33:45 -04:00
Sage Weil
fa4ee0f3c6 qa/tasks/cephfs/test_nfs: use keyword args
Signed-off-by: Sage Weil <sage@newdream.net>
2021-11-02 17:06:58 -04:00
Xiubo Li
8795d33185 qa: increase the timeout value to wait a litte longer
Sometimes the OpenFileTable::commit() will just come after the 30
seconds' waiting.

Fixes: https://tracker.ceph.com/issues/52887
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-11-02 11:06:14 +08:00
Sage Weil
38b6a8e8d0 Merge PR #43101 into master
* refs/pull/43101/head:
	mgr/rook: implement apply rbd-mirror

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
2021-11-01 15:27:13 -04:00
Patrick Donnelly
e0c19acbf1
Merge PR #43590 into master
* refs/pull/43590/head:
	qa: test that new mounts of same fs function after old mount is evicted
	qa: remove REQUIRE_KCLIENT_REMOTE

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2021-11-01 12:34:14 -04:00
Casey Bodley
74565072f7
Merge pull request #43625 from alimaredia/wip-marcus-teuthvault-2
qa/rgw: Fix vault token file access.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-29 15:14:03 -04:00
Joseph Sawaya
1b808345ee mgr/rook: implement apply rbd-mirror
This commit implements `orch apply rbd-mirror` in the rook orchestrator,
it creates a CR with a default name if the service_id isn't specified in
the spec, else it sets the name of the CR to the service_id in the spec.
This commit also adds `orch apply rbd-mirror` to the rook QA. This commit
also implements `orch rm rbd-mirror`.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
2021-10-28 15:47:13 -04:00
Marcus Watts
454cc8a18c Fix vault token file access.
Put the vault token file in a location that ceph can read.
Make it readable only by ceph.

On rhel8 (and indeed, any vanilla rhel machine), $HOME is liable to be
mode 700.  This means the ceph user can't read things in that user's
directory.  This causes radosgw to emit the confusing message "ERROR:
Vault token file ... not found" even though the teuthology log will
plainly show it was created and made readable by ceph.

Fixes: http://tracker.ceph.com/issues/51539
Signed-off-by: Marcus Watts <mwatts@redhat.com>
2021-10-28 14:14:10 -04:00
Sage Weil
d7acc16860 qa/tasks/cephfs/test_nfs: wait for fs to come up before exporting
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-28 13:28:35 -04:00
Patrick Donnelly
c8810e46e8
qa: lengthen health warning wait
It's just a little too short!

Fixes: https://tracker.ceph.com/issues/52995
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-26 14:26:19 -04:00
Rishabh Dave
e650bc6e87 qa/vstart_runner: add "managers" to LocalContext instances
Without this plenty tests become incompatible with vstart_runner.py.
Ideally, vstart_runner.py should've been updated in commit 7812cfb674.

Fixes: https://tracker.ceph.com/issues/53043
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-10-26 20:51:09 +05:30
Venky Shankar
01154fc41c qa: pass subdir arg when executing workunit
`_run_tests()` accepts subdir argument (to run a workunit with
the passed in sub-directory as cwd). One invocation was missing
the subdir argument causing `subdir` tag in yaml to be ineffective.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-10-25 12:40:53 +05:30
Patrick Donnelly
04aabf8bee
Merge PR #38752 into master
* refs/pull/38752/head:
	qa: enable dynamic debug support to kclient

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-20 22:24:32 -04:00
Jeff Layton
242585656c qa: test that new mounts of same fs function after old mount is evicted
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2021-10-20 10:34:01 -04:00
Neha Ojha
9fda06aa6a
Merge pull request #43572 from trociny/wip-qa-backfill-toofull-compress
qa/tasks/backfill_toofull: make test work when compression on

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-10-20 06:57:06 -07:00
Jeff Layton
5ab91d53a1 qa: remove REQUIRE_KCLIENT_REMOTE
Nothing references this variable anymore since commit 2df7caae4b (qa:
remove obsolete test).

Signed-off-by: Jeff Layton <jlayton@redhat.com>
2021-10-19 11:07:09 -04:00
Ernesto Puerta
f5fddd6121
Merge pull request #42526 from liewegas/dashboard-nfs
mgr/dashboard: consume mgr/nfs

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Varsha Rao <rvarsha016@gmail.com>
2021-10-19 11:17:17 +02:00
Ilya Dryomov
cf8b6dc972
Merge pull request #42760 from ideepika/wip-iscsi-testing
qa/suites/rbd: switch iscsi tests to cephadm

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-18 14:25:36 +02:00
Alfonso Martínez
58a6ab2147 mgr/dashboard: NFS exports: API + UI: integration with mgr/nfs; cleanups
mgr/dashboard: move NFS_GANESHA_SUPPORTED_FSALS to mgr_module.py

Importing from nfs module throws AttributeError because as a side effect the dashboard module is impersonating the nfs module.
https://gist.github.com/varshar16/61ac26426bbe5f5f562ebb14bcd0f548

mgr/dashboard: 'Create NFS export' form: list clusters from nfs module

mgr/dashboard: frontend+backend cleanups for NFS export

Removed all code and references related to daemons. UI cleanup and adopted unit-testing for
nfs-epxort create form for CEPHFS backend. Cleanup for export list/get/create/set/delete endpoints.

mgr/dashboard: rm set-ganesha ref + update docs

Remove existing set-ganesha-clusters-rados-pool-namespace references as
they are no longer required. Moreover, nfs doc in dashboard doc is
updated accordingly to the current nfs status.

mgr/dashboard: add nfs-export e2e test coverage

mgr/dashboard: 'Create NFS export' form: remove RGW user id field.

- Improve bucket typeahead behavior.
- Increase version for bucket list endpoint.
- Some refactoring.

mgr/dashboard: 'Create NFS export' form: allow RGW backend only when default realm is selected.

When RGW multisite is configured, the NFS module can only handle buckets in the default realm.

mgr/dashboard: 'Create service' form: fix NFS service creation.

After https://github.com/ceph/ceph/pull/42073, NFS pool and namespace are not customizable.

mgr/dashboard: 'Create NFS export' form: add bucket validation.

- Allow only existing buckets.
- Refactoring:
  - Moved bucket validator from bucket form to cd-validators.ts
  - Split bucket validator into 2: bucket name validator and bucket existence (that checks either existence or non-existence).

mgr/dashboard: 'Create NFS export' form: path validation refactor: allow only existing paths.

Fixes: https://tracker.ceph.com/issues/46493
Fixes: https://tracker.ceph.com/issues/51479
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
2021-10-18 12:58:54 +02:00
Deepika Upadhyay
cbd2c71398 qa/tasks: adapt ceph_iscsi.py task to ceph_iscsi_client
* we use setup_iscsi_client.py to deploy iscsi client services,
  configuring intiator and mulitpath this is done by qa task
  ceph_iscsi_client
* qa/cephadm: adds remotes ip addresses to iscsi gateway,
* rename poolname: iscsi >> datapool, which we usually use for tests and
  expresses type of pool more clearly.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-10-18 13:21:50 +05:30
Mykola Golub
429ac06cbb qa/tasks/backfill_toofull: make test work when compression on
The osd backfill reservation does not take compression into account so
we need to operate with "uncompressed" bytes when calculating nearfull
ratio.

Signed-off-by: Mykola Golub <mgolub@suse.com>
2021-10-16 15:28:53 +03:00
Patrick Donnelly
5af9882f94
Merge PR #43426 into master
* refs/pull/43426/head:
	qa/cephfs: update xfstests_dev for centos stream

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-15 11:15:41 -04:00
Patrick Donnelly
a8e77365b1
Merge PR #43425 into master
* refs/pull/43425/head:
	qa: import CommandFailedError from exceptions not run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
2021-10-15 11:13:55 -04:00
Patrick Donnelly
7aea7f48ba
Merge PR #43420 into master
* refs/pull/43420/head:
	qa: skip internal metadata directory when scanning ceph debugfs directory

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-15 11:13:05 -04:00
Laura Flores
bb0a39b7d9 qa/tasks/mgr: update tests that use mgr module ls to specify the json format
Signed-off-by: Laura Flores <lflores@redhat.com>
2021-10-14 23:59:47 +00:00
Kefu Chai
70b049ffdb
Merge pull request #43239 from trociny/wip-48959
osd: handle inconsistent hash info during backfill and deep scrub gracefully

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-10-14 22:43:16 +08:00
Kefu Chai
b5d2548ceb
Merge pull request #43463 from Zhiwei-Dai/wip-enhance-qa-python3-compatibility
qa/tasks: replace iterkeys() with keys() for Python 3

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 22:38:39 +08:00
Sebastian Wagner
d4783f5a65
Merge pull request #43214 from batrick/i52654
pybind/mgr/cephadm: set allow_standby_replay during CephFS upgrade

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 16:28:30 +02:00
Avan Thakkar
6644a00a2c mgr/dashboard: introduce gather facts in host list
Fixes: https://tracker.ceph.com/issues/52017
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
2021-10-13 16:02:51 +05:30
Avan Thakkar
b9f38cadc4 mgr/dashboard: Create Cluster Workflow welcome screen and e2e tests
A module option called CLUSTER_STATUS has two option. INSTALLED
AND POST_INSTALLED. When CLUSTER_STATUS is INSTALLED it will allow to show the
create-cluster-wizard after login the initial time.  After the cluster
creation is succesfull this option is set to POST_INSTALLED
Also has the e2e codes for the Review Section

Fixes: https://tracker.ceph.com/issues/50336
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>
2021-10-13 15:52:14 +05:30
Sebastian Wagner
1be6dc174f
Merge pull request #43455 from liewegas/qa-nvme-loop
qa: use nvme_loop devices for (some) cephadm tests

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-11 16:43:10 +02:00
Dai Zhiwei
00e5e5d5cd qa/tasks: replace iterkeys() with keys() in Python 3
Python 2.7 reached the end of its lifetime, the pr fixes teuthology task
error in Python 3.x

Fixes: https://tracker.ceph.com/issues/52878
Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>
2021-10-09 23:10:03 +08:00
Sage Weil
d4a1ec2d06 qa/tasks/nvme_loop: loop until 'nvme list' shows new devs
Sometimes this doesn't happen immediately.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-08 16:06:28 -05:00
Sage Weil
dda15a7924 qa/tasks/cephadm: wait for osds to start explicitly
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-08 16:06:28 -05:00
Sage Weil
d3c9486ed9 qa/tasks/cephadm: if no osd roles, --all-available-devices
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-08 16:06:28 -05:00
Sage Weil
65cf69c6ff qa/tasks/nvme_loop: set up nvme_loop on scratch_devs
Using an nvme loop device makes the LVs look like "real" disks,
which means we can exercise all of the normal code paths for
provisioning, deprovisioning, and zapping.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-08 16:06:28 -05:00
Patrick Donnelly
d33debc643
qa: fsync dir for asynchronous creat on stray tests
Use the enhanced create_n_files to dedup code. Also split the large test
into three.

Fixes: https://tracker.ceph.com/issues/52606
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-07 15:08:34 -04:00
Patrick Donnelly
395d20a2b7
qa: refactor and generalize create_n_files
Few things:

- Allow calling fsync on directory (to support async create kernel).
- Allow immediately unlinking the created file (for stray testing).
- Close any file descriptors created.
- Write unique content (the i variable) to each file.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-07 15:08:34 -04:00
Matt Benjamin
103b6cc355
Merge pull request #43442 from linuxbox2/wip-rgwadmin-logfix
qa/rgw: fix ops log tests to handle non-bucket ops (which are now valid)
2021-10-07 11:04:16 -04:00
Patrick Donnelly
2363078751
Merge PR #43231 into master
* refs/pull/43231/head:
	qa: fix promotion test

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-10-07 09:16:34 -04:00
Venky Shankar
ff88d7de52 qa: skip internal metadata directory when scanning ceph debugfs directory
kclient patchset

        https://patchwork.kernel.org/project/ceph-devel/list/?series=556049

introduces `meta` directory to add debugging entries. This needs to be filtered
when scanning ceph debugfs directory.

Fixes: https://tracker.ceph.com/issues/52824
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-10-07 00:40:13 -04:00