Commit Graph

921 Commits

Author SHA1 Message Date
Patrick Donnelly
e2b39f6c8c
qa: avoid using sudo for regular test artifacts
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-01 10:56:10 -07:00
Patrick Donnelly
4fc4ccb7a1
qa: convert mount calls to mount_wait
These tests want to immediately use the mount anyway. But the main
problem is, without waiting for the mount to complete, the command:

    chmod 1777 /path/to/mount

is not run so the mount cannot be written to by normal users without
sudo.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-01 10:56:10 -07:00
Patrick Donnelly
8a1ee83b1a
qa: use run_shell_payload to avoid sudo
"run_shell" adds 'sudo' which runs afoul of new security protections on
Ubuntu 20.04.

Fixes: https://tracker.ceph.com/issues/51417
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-01 10:56:10 -07:00
Sage Weil
4537b14b34 Merge PR #41574 into master
* refs/pull/41574/head:
	qa/tasks/vstart_runner: add LocalCluster.run
	qa/tasks/cephfs/test_nfs: fiddle with sudo
	mgr/nfs/export: some cleanup, minor refactoring
	mgr/nfs/cluster: remove unused @cluster_setter
	nfs/mgr: fix help message case
	doc/cephfs/fs-nfs-export: add note about export update behavior
	mgr/nfs: move user create/delete into helper
	mgr/nfs: refactor _delete_user helper
	mgr/nfs: refactor create_export_from_dict() helper
	mgr/nfs: keep 'nfs export get' around for backward-compat
	mgr/nfs: rename method
	qa/tasks/cephfs/test_nfs: test new export via apply
	doc/cephfs/fs-nfs-export: be consistent with cluster_id and _ vs -
	mgr/nfs: addr -> client_addr for 'nfs export create ...'
	mgr/nfs: fix tests
	mgr/nfs: 'nfs export get' -> 'nfs export info'
	mgr/nfs: binding -> pseudo_path
	mgr/nfs: more revisions based on review
	mgr/nfs: adjust NFSExceptoin errno arg
	doc/cephfs: update 'nfs export {get,apply}' docs
	mgr/nfs: merge FSExport back into ExportMgr
	doc/radosgw/nfs: document mgr/nfs way to add/remove rgw exports
	mgr/nfs: merge 'nfs export {update,import}' -> 'nfs export apply'
	mgr/nfs: test export creation and list
	mgr/nfs: test export_update (+ fixes)
	mgr/nfs: test Export.validate(); several fixes
	mgr/nfs: test that export <-> block+dict conversions go both ways
	mgr/nfs: clean up test a bit
	mgr/nfs/export: fix export validation
	mgr/nfs/export: fix tests
	mgr/nfs: handle option addr/client block in create_export()
	mgr/nfs: allow multiple addrs for new exports
	mgr/nfs: fix/finish rgw export
	mgr/nfs/module: clusterid -> cluster_id
	mgr/nfs/export: fix export_update_1 to type check
	mgr/nfs/cluster: fix type error
	mgr/nfs/export: wrap long lines
	mgr/nfs: ExportMgr._delete_export only works for cephfs for now
	mgr/nfs: Remove pool_ns from NFSCluster
	mgr/nfs: Remove ExportMgr.rados_namespace
	mgr/nfs: flake8
	mgr/nfs: Add type checking
	mgr/nfs: Add __eq__ method to Export
	mgr/nfs: Add some compatibility to mgr/dashboard
	mgr/nfs: Fix whitespace handling
	mgr/nfs: Copy unit tests from mgr/dashboard
	mgr/nfs: partially implement rgw export support
	mgr/nfs: abstract FSAL; add RGWFSAL
	mgr/nfs: refactor to merge 'update' and 'import' code
	mgr/nfs: add 'nfs export import' command
	mgr/nfs: refactor 'nfs export update' and export validation
	mgr/nfs: fix _fetch_export to distinguish between clusters
	mgr/nfs: move export ganesha conf translation into caller
	mgr/nfs: name nfs cephfs client key 'nfs.{cluster_id}.{export_id}'
	mgr/nfs: add --addr to 'nfs export create'
	mgr/nfs: add --squash to 'nfs export create'
	mgr/nfs/export_utils: include false but non-None items in config
	vstart.sh: enable nfs module
	mgr/cephadm: nfs: drop attr_expiration_time from top-level config
	mgr/cephadm: remove Dir_Chunk = 0

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-06-26 10:41:27 -04:00
Sage Weil
8788af5663 qa/tasks/cephfs/test_nfs: fiddle with sudo
- no sudo for 'ceph' commands
- explicit sudo for _sys_cmd (things like 'rados' don't need sudo!)

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-26 08:47:06 -04:00
Sage Weil
10786a6380 mgr/nfs: move user create/delete into helper
- Do user create or delete via a helper
- Defer until after we have validated the Export (on create or update)
- Support updates to user_id, which is needed to keep the naming consistent
and to also support changing the bucket, since the user_id is derived
from that.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-25 19:13:09 -04:00
Patrick Donnelly
e590f91e84
Merge PR #41908 into master
* refs/pull/41908/head:
	qa: always format the pgid in hex

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-22 19:38:16 -07:00
Patrick Donnelly
a90da2d9c5
Merge PR #41385 into master
* refs/pull/41385/head:
	mon/FSCommands: add command to rename a file system
	qa/cephfs: split test_admin.TestAdminCommands
	mds: remove 'fs_name' from MDSRank

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-22 19:34:01 -07:00
Patrick Donnelly
f27ec02a61
Merge PR #41860 into master
* refs/pull/41860/head:
	qa: log messages when falling back to force/lazy umount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2021-06-21 16:54:59 -07:00
Sage Weil
6fb9f0a391 qa/tasks/cephfs/test_nfs: test new export via apply
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Sage Weil
a2679b3884 mgr/nfs: fix tests
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Sage Weil
2400937ce2 mgr/nfs: 'nfs export get' -> 'nfs export info'
This is a breaking change, but aligns the CLI verbs with the rest of ceph.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Sage Weil
b276adbd47 mgr/nfs: more revisions based on review
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Sage Weil
473097a36f mgr/nfs/export: fix tests
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Patrick Donnelly
efea9ecac7
Merge PR #40997 into master
* refs/pull/40997/head:
	test: add test to verify adding an active peer back to source
	pybind/mirroring: disallow adding a active peer back to source
	pybind/cephfs: interface to fetch file system id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-18 19:49:15 -07:00
Patrick Donnelly
c4de4b3df8
Merge PR #36823 into master
* refs/pull/36823/head:
	qa : add a test for the cmd, dump cache
	mds : add timeout to the command, dump cache, to prevent it from running too long and affecting the service

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-18 19:47:53 -07:00
Sage Weil
9b1372dc38 mgr/nfs: name nfs cephfs client key 'nfs.{cluster_id}.{export_id}'
Better than '{cluster_id}{export_id}', which is confusing and ambiguous.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-17 16:18:06 -04:00
Ramana Raja
bd83fac803 mon/FSCommands: add command to rename a file system
The fs_name of the relevant MDSMap is set to the new name. Also,
the application tags of the data pools and the meta data pool of
the file system is set to the new name.

Fixes: https://tracker.ceph.com/issues/47276
Signed-off-by: Ramana Raja <rraja@redhat.com>
2021-06-17 13:59:08 -04:00
Rishabh Dave
c8e2db190c qa/cephfs: split test_admin.TestAdminCommands
Splitting TestAdminCommands makes it easier to exercise a particular
"ceph fs" subcommand. Also, rename class TestSubCmdFsAuthorize to
TestFsAuthorize.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2021-06-17 13:59:05 -04:00
Xiubo Li
2b9b2a3fbd qa: always format the pgid in hex
If the pg number is larger than 9, this won't match the array index,
which was in dec just before this.

Fixes: https://tracker.ceph.com/issues/50808
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-06-17 13:55:09 +08:00
Patrick Donnelly
174b8ad30b
Merge PR #41840 into master
* refs/pull/41840/head:
	qa: update cli syntax to conventional

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-06-15 10:34:18 -07:00
Patrick Donnelly
8cb34b3849
Merge PR #41771 into master
* refs/pull/41771/head:
	qa: update scrub start code to use comma sep scrubopts

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-15 10:32:06 -07:00
Jeff Layton
53863f3654 qa: log messages when falling back to force/lazy umount
We suspect that we may be hitting races against lazy umounts that are
causing problems when enumerating debugfs files. Currently though we
don't log anything special when falling back to a lazy unmount. Log
a message when killing processes on a mount and when force+lazy
unmounting.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
2021-06-15 07:16:00 -04:00
Patrick Donnelly
a402b23c84
qa: update cli syntax to conventional
This was using an obscure syntax that worked at one time and wasn't
documented (AFAIK).

Fixes: https://tracker.ceph.com/issues/51182
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-14 10:21:43 -07:00
Patrick Donnelly
7e320919f7
Merge PR #41482 into master
* refs/pull/41482/head:
	qa: remove obsolete deactivate routines

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-06-13 19:56:34 -07:00
Patrick Donnelly
6a095654f4
Merge PR #41422 into master
* refs/pull/41422/head:
	qa/tasks/cephfs/test_sessionmap: reap connections immediately
	msg/async: configurable threshold for reaping dead connections

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-13 19:52:48 -07:00
Patrick Donnelly
0441b3d60f
Merge PR #41403 into master
* refs/pull/41403/head:
	mgr/volumes: Add config to insert delay at the beginning of the clone

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-13 19:51:52 -07:00
Patrick Donnelly
0f505dc299
qa: update scrub start code to use comma sep scrubopts
The documentation specifies this in [1] and yet we were using (I
believe) an older syntax:

    ceph tell mds.foo:0 scrub start / recursive force

instead of

    ceph tell mds.foo:0 scrub start / recursive,force

Oddly the former works at least as recently as in [2]:

    2021-06-03T07:11:42.071 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / recursive force
    ...
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:{
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:    "return_code": 0,
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:    "scrub_tag": "cf7a74b2-3eb2-4657-9274-ea504b1ebf8f",
    2021-06-03T07:11:42.269 INFO:teuthology.orchestra.run.smithi025.stdout:    "mode": "asynchronous"
    2021-06-03T07:11:42.269 INFO:teuthology.orchestra.run.smithi025.stdout:}

[1] https://docs.ceph.com/en/latest/cephfs/scrub/
[2] /ceph/teuthology-archive/pdonnell-2021-06-03_03:40:33-fs-wip-pdonnell-testing-20210603.020013-distro-basic-smithi/6148097/teuthology.log

Fixes: https://tracker.ceph.com/issues/51146
See-also: https://tracker.ceph.com/issues/51145
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-09 07:23:05 -07:00
Sage Weil
b18427da4b Merge PR #41509 into master
* refs/pull/41509/head:
	common/cmdparse: fix CephBool validation for tell commands
	mgr/nfs: fix 'nfs export create' argument order
	common/cmdparse: emit proper json
	mon/MonCommands: add -- seperator to example
	qa/tasks/cephfs/test_nfs: fix export create test
	mgr: make mgr commands compat with pre-quincy mon
	doc/_ext/ceph_commands: handle non-positional args in docs
	mgr: fix reweight-by-utilization cephbool flag
	mon/MonCommands: convert some CephChoices to CephBool
	mgr/k8sevents: fix help strings
	pybind/mgr/mgr_module: fix help desc formatting
	mgr/orchestrator: clean up 'orch {daemon add,apply} rgw' args
	mgr/orchestrator: add end_positional to a few methods
	mgr/orchestrator: reformat a few methods
	pybind/ceph_argparse: stop parsing when we run out of positional args
	pybind/ceph_argparse: remove dead code
	pybind/mgr/mgr_module: infer non-positional args
	pybind/mgr/mgr_module: add separator for non-positional args
	command/cmdparse: use -- to separate positional from non-positional args
	pybind/ceph_argparse: adjust help text for non-positional args
	pybind/ceph_argparse: track a 'positional' property on cli args

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-07 10:02:52 -04:00
Sage Weil
8683cccd06 qa/tasks/cephfs/test_nfs: fix export create test
Everything after --readonly is non-positional.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-04 16:56:17 -04:00
Sage Weil
c8c5071dcd qa/tasks/cephfs/test_sessionmap: reap connections immediately
We have to reap connections promptly for this test to work.

This test was broken indirectly by d51d80b323,
which moved the counter decrement to reap time instead of mark_down/stop
time.

The reaping is asynchronous, so allow for a delay in the count change.

Fixes: https://tracker.ceph.com/issues/50622
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-04 11:02:29 -04:00
Patrick Donnelly
1c40ee32f9
qa: increase fragmentation to improve uniform distribution
Fixes: https://tracker.ceph.com/issues/51060
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-01 20:25:17 -07:00
Venky Shankar
074f55006e test: add test to verify adding an active peer back to source
Fixes: http://tracker.ceph.com/issues/50447
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-31 00:32:05 -04:00
Milind Changire
26fbbefa82
Merge pull request #40831 from vshankar/wip-cephfs-mirror-incremental-sync
cephfs-mirror: incremental sync

Reviewed-by: Milind Changire <mchangir@redhat.com>
2021-05-27 13:39:23 +05:30
Sage Weil
9ab9cc26e2 Merge PR #41007 into master
* refs/pull/41007/head:
	qa/tasks/cephfs/test_nfs: fix info test
	doc/cephfs/fs-nfs-exports: document --ingress --virtual-ip
	mgr/nfs: move ingress vs virtual_ip check to cluster interface
	PendingReleaseNotes: clarify deprecated
	PendingReleaseNotes: note breaking CLI changes
	doc/cephadm/nfs: document nfs+ingress
	qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
	mgr/nfs: take --ingress argument to 'nfs cluster create'
	mgr/cephadm: adjust debug output for device refresh
	mgr/cephadm: ingress: fix log msg
	mgr/cephadm: fix logging of config/placement errors
	common/options: enable nfs module for new clusters
	cephadm: --stop-signal=SIGTERM
	mgr/orchestrator: default nfs pool, namespaces
	mgr/cephadm: nfs: create pool if it doesn't yet exist
	doc/cephadm/nfs: update
	mgr/nfs: change 'nfs cluster info'
	mgr/nfs: take optional virtual_ip for deploying ingress
	mgr/nfs: remove 'nfs cluster update'
	mgr/nfs: factor out ganesha pool creation
	mgr/nfs: delete -> rm for CLI
	mgr/nfs: add some type annotations
	python-common: fix IngressSpec yaml dump
	mgr/cephadm: ingress: remove eth0 default
	qa/tasks/cephadm: allow mounting volumes in shell
	cephadm: add -v arg to shell
	qa/tasks/vip: add 'vip.exec' task
	mgr/orchestrator: add --port arg to 'orch apply nfs'
	mgr/cephadm: nfs: add purge
	mgr/cephadm: ingress: support nfs
	mgr/cephadm: do not reconfigure daemons on deleted services
	mgr/cephadm: nfs: shell out to rados tool for conf creation
	mgr/cephadm: nfs: add rank to grace file from mgr module
	mgr/cephadm: nfs: bind ganesha to appropriate ip:port
	mgr/cephadm: enable ranked daemons for nfs
	mgr/cephadm: support creation of daemons with ranks
	mgr/cephadm: make _plan show removed daemon names
	mgr/cephadm/schedule: assign/map ranks
	mgr/cephadm: add rank[_generation] properties
	mgr/cephadm/inventory: store optional rank_map along with specs
	mgr/cephadm: include service_name is generated DaemonDescription
	mgr/orchestrator: include service_name in DaemonDescription dump
	mgr/cephadm/inventory: fix deleted check
	mgr/cephadm: simplify
	mgr/cephadm/schedule: make placement shuffle deterministic
	mgr/cephadm: document CephadmService flags

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2021-05-25 16:17:44 -04:00
Sage Weil
218eec938d qa/tasks/cephfs/test_nfs: fix info test
Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-25 10:15:45 -04:00
Venky Shankar
da9788a9b5 test: add test to verify incremental snapshot updates
Fixes: http://tracker.ceph.com/issues/49939
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-25 08:45:44 -04:00
Simon Gao
430d93222a qa : add a test for the cmd, dump cache
Signed-off-by: Simon Gao <simon29rock@gmail.com>
2021-05-23 14:43:02 +08:00
Patrick Donnelly
285e1547de
qa: remove obsolete deactivate routines
This is handled automatically since Mimic.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-21 12:52:10 -07:00
Kotresh HR
7588f98505 mgr/volumes: Add config to insert delay at the beginning of the clone
Added the config 'delay_snapshot_clone' to insert delay at the beginning
of the clone to avoid races in tests. The default value is set to 0.

Fixes: https://tracker.ceph.com/issues/48231
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-05-21 12:35:07 +05:30
Sage Weil
ad0a62dc9d mgr/nfs: delete -> rm for CLI
The rest of the CLI uses 'rm' in place of 'remove' or 'delete', so let's
deprecate 'delete' and add 'rm'.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-20 18:38:18 -04:00
Patrick Donnelly
21b69fda3c
Merge PR #41084 into master
* refs/pull/41084/head:
	test: test to verify dir path removal when no mirror daemons are running
	pybind/mirroring: advance state machine from stalled state
	pybind/mirroring: start from correct state during policy init

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-20 12:42:19 -07:00
Patrick Donnelly
e9f7fafe52
Merge PR #41171 into master
* refs/pull/41171/head:
	test: disable mirroring module for certain tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-18 13:41:28 -07:00
Varsha Rao
94644d1e5f qa/tasks/nfs: add test to check if cmds fail on not passing required arguments
Signed-off-by: Varsha Rao <varao@redhat.com>
2021-05-14 21:03:50 +05:30
Venky Shankar
f4ed3ae5f3 test: test to verify dir path removal when no mirror daemons are running
Fixes: http://tracker.ceph.com/issues/50523
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-11 08:39:45 -04:00
Venky Shankar
0676409bec test: disable mirroring module for certain tests
Without this a traceback is seen in mgr logs. Also, this solves
one part of the issue. The other half (failing tests) will be
resolved by PR #40885.

Fixes: http://tracker.ceph.com/issues/50224
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-05 08:37:35 -04:00
Patrick Donnelly
a52cc67249
Merge PR #40962 into master
* refs/pull/40962/head:
	test: add test to validate snap synchronization with parent directory snapshots
	cephfs-mirror: ignore parent directory snapshots when building snap map

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-03 17:43:47 -07:00
Venky Shankar
9dcf6f7c8c
Merge pull request #40157 from leseb/do-not-fail-when-nosnap
mgr/pybind/snap_schedule: do not fail when no fs snapshots are available

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-04-27 18:51:57 +05:30
Venky Shankar
8343471a0f
Merge pull request #40933 from vshankar/wip-50266
cephfs-mirror: sanitize `daemons status` JSON

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-04-27 12:22:37 +05:30
Sébastien Han
74a7cdda6b mgr/pybind/snap_schedule: do not fail when no fs snapshot schedules are available
When listing for available snapshot schedules, we should not an error in case
there is none. We should just return 0 with an empty dict.

Fixes: https://tracker.ceph.com/issues/49837
Signed-off-by: Sébastien Han <seb@redhat.com>
2021-04-26 08:57:28 +02:00