Commit Graph

479 Commits

Author SHA1 Message Date
Gregory Farnum
60504bd048
Merge pull request #33279 from rishabh-d-dave/fs-move-run_shell-and-related-methods
qa/cephfs: move run_shell to mount.py and add methods for negative testing
2020-03-27 12:59:43 -07:00
Kefu Chai
4b4f393aec qa/tasks/cephfs: fix print() to be py3 compatible
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-03-27 14:51:24 +08:00
Yan, Zheng
4b79a2ecce
Merge pull request #29443 from simon-rock/metatool_dev
tool: add cephfs-meta-injection to show and amend info of inode
2020-03-26 17:10:28 +08:00
Kefu Chai
496d7e2132 qa/tasks/cephfs: fix syntax error
it's a regression introduced by 9f6c764f10

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-03-24 18:27:55 +08:00
Kefu Chai
947a74349d qa: import with full path
to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-03-24 18:27:55 +08:00
simon gao
53ba9089d8 qa : add case to test cephfs-meta-injection
Signed-off-by: simon gao <simon29rock@gmail.com>
2020-03-24 16:39:02 +08:00
Rishabh Dave
48e0e1d703 vstart_runner.py: add methods for negative testing a cmd
Methods like run_shell effectively conduct positive test on the given
command. Add methods that runs given command expecting failure and then
verifies return value and error message with given one. Rewrite testcmd,
testcmd_as_user and testcmd_as_root to create these new methods for
negative testing since run_shell, run_as_user and run_as_root is
equivalent of running positive test.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-03-20 02:11:58 +05:30
Rishabh Dave
751d432a86 qa/cephfs: move run_shell and related methods to mount.py
LocalFuseMout and LocalKernelMount can directly inherit these methods
from CephFSMount via FuseMount and KernelMount respectively. Moving
would avoid duplication and would make these methods more accessible
for reusing via inheritance.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-03-20 02:11:54 +05:30
Sage Weil
6cb39e4aa8 Merge PR #34060 into master
* refs/pull/34060/head:
	Merge PR #34027 into octopus
	Merge PR #34045 into octopus
	Merge pull request #34035 from dillaman/wip-rbd-permissions
	mgr/progress: fix duration strings
	Merge PR #34014 into octopus
	Merge PR #34001 into octopus
	Merge PR #34011 into octopus
	qa/workunits/rbd: use context managers to control Rados lifespan
	Merge pull request #34032 from dillaman/wip-rbd-octopus-docs
	doc/releases/octopus: add additional RBD improvements
	qa/workunits/cephadm/test_cephadm: mark services unmanaged for test
	mgr/cephadm: do not reconfig unmanaged services
	Merge PR #33981 into octopus
	Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel
	qa/workunits/cephadm/test_cephadm: output file for pub key
	Merge PR #33866 into octopus
	Merge PR #34005 into octopus
	Merge PR #34013 into octopus
	mgr/cephadm: pytest: Enable SpecStore
	mgr/orchestrator: add test for default implementation for apply()
	python-common: validate ServiceSpec.service_type
	fixup mgr/cephadm: Fix ceph orch apply -i
	mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
	mgr/orchestrator: `orch daemon add` accepts a yaml
	mgr/cephadm: apply_drivegroups() returns a single Completion
	mgr/cephadm: remove `trivial_result()`
	mgr/cephadm: Fix `ceph orch apply -i`
	Merge pull request #33994 from dillaman/wip-librbd-poll-event-race
	doc: document `clone cancel` command
	test: add `clone cancel` tests
	mgr/volumes: introduce "clone cancel" volume command
	mgr/volumes: allow canceling a single asynchronous job for a volume
	mgr/volumes: helper for looking up a clone entry index
	mgr/volumes: periodically check if clone operations should be canceled
	mgr/volumes: periodically check if copy operations should be canceled
	mgr/volumes: introduce 'canceled' state in clone op state machine
	qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
	qa/suites/rados/verify/validater/valgrind: less bluestore logging
	qa/suites/rados/verify/validater: increase heartbeat grace
	Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
	Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
	ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
	ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
	qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits
	doc/install: edits
	doc/cephadm: more edits
	doc/cephadm/install: edits
	doc/cephadm/adoption: improvements
	doc/cephadm/install: a few edits
	doc/cephadm/install: do not install ceph-common on host (by default)
	doc/cephadm: drop os recs link
	doc/cephadm/upgrade: improvements
	doc/cephadm/upgrade: document upgrade
	doc/cephadm/install: revamp install docs
	doc: reorganize cephadm docs
	doc/cephadm/administration: update docs on customizing SSH config
	doc/cephadm/administration: add a note about the 'removed' dir
	mgr/balancer: tolerate pgs outside of target weight map
	qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack
	Merge PR #33974 into octopus
	Merge PR #33442 into octopus
	Merge PR #33997 into octopus
	Merge PR #34000 into octopus
	use quay octopus tip until 15.2 tag is available
	python-common: reduce output of ServiceSpec.to_json()
	python-common,mgr/cephadm: move assert_valid_host to service_spec
	mgr/cephadm: add HostAssignment.validate()
	mgr/dashboard: adapt create_osds interface change
	mon/MgrMonitor: make 'mgr fail' work with no arguments
	cephadm: add allow_ptrace option to enable SYS_PTRACE
	update default container images
	mgr/cephadm: limit number of times check host is performed in the serve loop
	Merge PR #33961 into octopus
	Merge PR #33952 into octopus
	Merge PR #33990 into octopus
	Merge PR #33955 into octopus
	Merge PR #33936 into octopus
	mgr/orch: add --all-available-devices to 'orch apply osd'
	qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1
	cephadm: add tests
	qa/tasks/cephadm: pass -v to bootstrap
	mgr/cephadm: only try to place mons on hosts matching public_network
	mgr/cephadm: keep track of host networks, ips
	cephadm: automatically infer mon public_network, if we can
	cephadm: add list-networks command
	cephadm: bootstrap: deploy monitoring stack by default
	librbd: defer event socket completion until after callback issued
	cephadm: add-repo: add --version
	mgr/cephadm: respect 'unmanaged' flag in spec
	mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate
	mgr/orch: orch ls: rename SPEC -> PLACEMENT
	mgr/orch: add 'unmanaged' property to ServiceSpec
	cephadm: rename distro args in repo methods
	mgr/orch: combine 'orch daemon add <type> ...' into one command
	mgr/orch: combine 'orch apply <type> [<placement>]' into one command

Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-03-19 11:07:16 -05:00
Venky Shankar
ba4b9358e1 test: add clone cancel tests
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-03-18 11:03:17 +05:30
Sage Weil
603383605f Merge PR #33885 into master
* refs/pull/33885/head:
	Merge pull request #33848 from mchangir/octopus-tests-remove-suprious-whitespace
	Merge PR #33746 into octopus
	Merge PR #33830 into octopus
	Merge PR #33732 into octopus
	Merge PR #33620 into octopus
	Merge pull request #33876 from tchaikov/octopus-cephadm-mypy
	cephadm: add "assert foo is not None" for mypy check
	Merge pull request #33067 from tspmelo/wip-rbd-delete-with-snapshot
	cephadm: add grafana adopt
	Merge PR #33771 into octopus
	Merge PR #33850 into octopus
	Merge PR #33853 into octopus
	Merge PR #33857 into octopus
	Merge PR #32990 into octopus
	Merge PR #33713 into octopus
	Merge PR #33838 into octopus
	qa/tasks/cephadm: no default mon|mgr|crash service specs
	qa/suites/rados/cephadm/upgrade: upgrade start point that supports the no-spec option
	Merge PR #33832 into octopus
	cephadm: bootstrap: wait for mgr to restart after enabling a module
	mgr: add 'mgr_status' tell command
	Merge pull request #33839 from rhcs-dashboard/44538-fix-rgw-grafana-get-put-latencies
	Merge pull request #33743 from votdev/issue_43869_fix_qa_test
	cephadm: create initial mon and mgr service specs too
	cephadm: no need to pregenerate a crash key for the bootstrap host
	mgr/cephadm: do not complain when we don't have enough hosts
	mgr/cephadm: remove orphan daemons
	mgr/cephadm: report size=0 for fabricated ServiceDescription
	mgr/cephadm: safety check to prevent removing all mon|mgr daemons
	mgr/cephadm: prevent scaling mon|mgr below count=1
	mgr/cephadm: do not remove daemons from remove_service
	Merge pull request #33805 from tchaikov/wip-44500
	spec: Podman (temporarily) requires apparmor-abstractions on suse
	mgr/cephadm: Make sure we don't co-locate the same daemon
	monitoring: fix RGW grafana chart 'Average GET/PUT Latencies'
	tests: remove spurious whitespace
	mgr/cephadm: fix service list filtering
	Merge PR #33825 into octopus
	Merge PR #33811 into octopus
	Revert "Merge pull request #33673 from cbodley/wip-denc-enum"
	mgr/cephadm: fix upgrade order
	Merge PR #33801 into octopus
	Merge PR #33822 into octopus
	cephadm: bootstrap: tolerate error return from -h
	Merge PR #33809 into octopus
	Merge PR #32678 into octopus
	cephadm: use `sh` instead of `bash` during enter
	ceph.in: only shut down rados on clean exit
	common/ceph_timer: Pass reference to waited time on stack
	common/ceph_timer: Add test
	common/ceph_timer: Use unique_function, allowing noncopyable events
	common/ceph_timer: Couple cleanups
	common/ceph_timer: Fix namespaces
	common/ceph_timer: Add missing includes
	common/ceph_timer.h: Don't indent contents of a namespace
	mgr/dashboard: Crush rule modal
	mgr/dashboard: Preserve rule selection on pool type change
	mgr/dashboard: Crush rule is only send during replicated pool creation
	mgr/dashboard: Explicit returns in pool form
	mgr/dashboard: Removes fork join in pool form
	mgr/dashboard: Hide ECP actions during ec pool edit
	mgr/dashboard: Pool form erasure/replicated boolean
	mgr/dashboard: Change pool info API endpoint
	mgr/dashboard: Moves ECP info endpoint to UI-API
	mgr/cephadm: add _remove_osds_bg back to main loop
	mgr/cephadm/osd: update removal report immediately
	qa/tasks/ceph_manager: use StringIO for capturing COT output
	qa/standalone/scrub/osd-scrub-repair: force osdmap prop to osds
	qa/standalone/scrub/osd-scrub-test: wait longer for update
	qa/tasks/ceph_manager: capture stderr for COT
	qa/suites/rados/ceph: drop opensuse for now
	mon/MonClient: send logs to mon on separate schedule than pings
	mgr/dashboard: Fix missing ImageSpec usage
	mgr/dashboard: Allow removing RBD with snapshots
	mgr/dashboard: Refactor and cleanup tasks.mgr.dashboard.test_user
	mgr/dashboard: support multiple DriveGroups when creating OSDs
	mon/MonClient: send logs to mon even if we have no keelalive2
	cephadm: flag dashboard user to change password

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2020-03-11 17:38:59 -05:00
Kefu Chai
8fd63d2384
Merge pull request #32262 from kshtsk/wip-ceph-py-py3
qa/tasks: py3 compat

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-03-11 23:22:04 +08:00
Milind Changire
f7d056a026 tests: remove spurious whitespace
remove spurious whitespace from test_snapshot.py

Fixes: https://tracker.ceph.com/issues/44528
Signed-off-by: Milind Changire <mchangir@redhat.com>
2020-03-10 16:11:07 +05:30
Ramana Raja
d2e327c41c test_volumes: fix _verify_clone_attrs call
... to verify the attributes of clone and source subvolume belonging
to different subvolume groups.

Introduced in e22d546beb

Fixes: https://tracker.ceph.com/issues/44438
Signed-off-by: Ramana Raja <rraja@redhat.com>
2020-03-06 21:17:26 +05:30
Kyr Shatskyy
875b45f3c9 qa/tasks/cephfs/kernel_mount: get rid of StringIO.StringIO for py3
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-03-06 10:19:55 +01:00
Kyr Shatskyy
9f6c764f10 qa/tasks/cephfs: get rid of StringIO for py3
Use io.BytesIO and six.ensure_str for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-03-06 10:19:55 +01:00
Thomas Bechtold
46e22c422b qa: Enable basic mypy support for qa/ directory
A first step to do more automatic code checks on the qa/
directory. This is useful while transitioning to python3.

Also use log_exc to top-level to not run into:

error: Argument 1 to "log_exc" has incompatible type
  "Callable[[OSDThrasher], Any]"; expected "OSDThrasher"

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
2020-03-05 06:54:56 +01:00
Thomas Bechtold
f5e77561e9 qa: Fix problems detected by mypy
This is a first step to enable mypy on the qa/ directory.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
2020-03-05 06:53:31 +01:00
Kyr Shatskyy
e46eb8348e qa/tasks: fix imports for py3 compatibility
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-03-04 13:09:16 +08:00
Patrick Donnelly
416a26e733
Merge PR #33636 into master
* refs/pull/33636/head:
	qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
2020-03-03 08:17:43 -08:00
Ramana Raja
8034044f94
Merge pull request #33120 from vshankar/wip-mgr-volumes-clone-sync-attrs
mgr/volumes: sync inode attributes for cloned subvolumes

Reviewed-by: Ramana Raja <rraja@redhat.com>
2020-03-03 16:13:34 +05:30
Patrick Donnelly
0c8899c985
qa: add upgrade test for volume upgrade from legacy
This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-03-02 20:27:15 -08:00
Patrick Donnelly
b58c14a2d8
Merge PR #33363 into master
* refs/pull/33363/head:
	mds: add perf counters for openfiletable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-03-02 17:01:48 -08:00
Patrick Donnelly
61681cffd2
Merge PR #33506 into master
* refs/pull/33506/head:
	client: add client_fs mount option support

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-03-02 16:58:02 -08:00
Venky Shankar
e22d546beb test: add test for verifying inode attrbiutes sync on clone
Also, for existing tests, additionally verify inode attributes
for clones.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-02-28 11:42:17 -05:00
Xiubo Li
1a58e64299 client: add client_fs mount option support
"client_fs" is one alias for "client_mds_namespace=" and it will be
cleaner and be more user-friendly to use. "client_mds_namespace="
will be kept and backwards compatibility used.

Update the documents at the same time.

Fixes: https://tracker.ceph.com/issues/44212
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-02-27 19:30:31 -05:00
Patrick Donnelly
a7d273cef7
Merge PR #31200 into master
* refs/pull/31200/head:
	qa/cephfs: test case for auto reconnect after blacklisted

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-26 06:26:13 -08:00
Milind Changire
10fafc0ca6 mds: add perf counters for openfiletable
Add perf counters for:
* total omap objects
* total omap key-value pairs
* total omap updates
* total omap removes

Fixes: https://tracker.ceph.com/issues/43750
Signed-off-by: Milind Changire <mchangir@redhat.com>
2020-02-26 12:09:58 +05:30
Patrick Donnelly
aebc88c31e
Merge PR #33427 into master
* refs/pull/33427/head:
	qa/cephfs: rewrite a bit of code xfstests_dev.py
	qa/cephfs: update xfstests-dev deps for RHEL 8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-25 11:05:57 -08:00
Patrick Donnelly
1d3ed17dcb
Merge PR #32657 into master
* refs/pull/32657/head:
	test: query using mds id, not rank
	mgr: re-enable mds `scrub status` info in ceph status
	mon: filter out ceph normal ceph entity types when dumping service metadata
	mgr: filter out normal ceph services when processing service map
	mgr: helper function to check if a service is a normal ceph service

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-25 11:04:06 -08:00
Patrick Donnelly
c0c0b5a265
Merge PR #33078 into master
* refs/pull/33078/head:
	qa: avoid subtree rep in test_version_splitting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-02-25 09:14:37 -08:00
Patrick Donnelly
a80921307e
qa: avoid subtree rep in test_version_splitting
Root subtree may be replicated which would open client sessions early.

Fixes: https://tracker.ceph.com/issues/43796
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-24 12:28:15 -08:00
Venky Shankar
92b2008936 test: verify purge queue w/ large number of subvolumes
Fixes: http://tracker.ceph.com/issues/44207
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-02-24 04:27:02 -05:00
Venky Shankar
5ec09a228c test: pass timeout argument to mount::wait_for_dir_empty()
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-02-24 04:27:02 -05:00
Rishabh Dave
0c1dd0a51d qa/cephfs: rewrite a bit of code xfstests_dev.py
The variable storing the major version number plays an important role
while updating deps, therefore use a better name that is more
descriptive and makes spotting thereby easier.

Also, add an explanation for why we we have list of deps for fedora and
remove a redundant line of code.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-02-20 20:13:06 +05:30
Rishabh Dave
0f88a21267 qa/cephfs: update xfstests-dev deps for RHEL 8
btrfs-progs-devel is not availble on RHEL 8 anymore.

Fixes: https://tracker.ceph.com/issues/43964
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-02-20 20:10:11 +05:30
Venky Shankar
fb4cd42cbc test: query using mds id, not rank
Fixes: http://tracker.ceph.com/issues/42835
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-02-18 02:23:13 -05:00
Patrick Donnelly
2c9642131f
qa: add tests for mds_join_fs cluster affinity
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-13 07:51:10 -08:00
Patrick Donnelly
e5dbe0b366
qa: accept operation on current fs status
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-13 07:51:10 -08:00
Patrick Donnelly
d21b651352
qa: add method to enable multifs
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-13 07:51:09 -08:00
Patrick Donnelly
92e5b7b15b
qa: fix nested generator use
Apparently Python doesn't allow nested generator calls. Hurray.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-13 07:51:09 -08:00
Patrick Donnelly
8729281121
qa: manage config changes through mons
This provides a generic framework for modifying Ceph configuration
changes in tests through the monitors rather than the asok interface or
local ceph.conf changes. Any changes are reverted during test teardown.

A future patch will convert existing tests manipulating the local
ceph.conf or admin socket.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-13 07:51:09 -08:00
Patrick Donnelly
29d850fb7e
Merge PR #32570 into master
* refs/pull/32570/head:
	cephfs-shell: Add tests for setxattr, getxattr and listxattr
	cephfs-shell: Add listxattr command
	cephfs-shell: Add getxattr command
	cephfs-shell: Add setxattr command
	doc: Update about extended attributes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-02 06:56:50 -08:00
Patrick Donnelly
3a6f107331
Merge PR #31633 into master
* refs/pull/31633/head:
	cephfs-shell: Instead of assert use stat for tests in rmdir
	cephfs-shell: Add function for common rmdir test code
	cephfs-shell: Add rmdir test for non empty directory
	cephfs-shell: Add rmdir -p test for non empty directory
	cephfs-shell: Add rmdir -p test for non existing dir
	cephfs-shell: Add rmdir -p test to delete all dirs in given path
	cephfs-shell: Add rmdir -p test for root directory with empty directories
	cephfs-shell: Add rmdir test for valid file
	cephfs-shell: Add rmdir test for invalid directory
	cephfs-shell: Add rmdir test for valid directory
	cephfs-shell: Fix rmdir '-p' issues

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-02-02 06:52:23 -08:00
Venky Shankar
b5970ff80d test: add subvolume clone tests
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2020-01-31 05:09:14 -05:00
Varsha Rao
e601242afe cephfs-shell: Add tests for setxattr, getxattr and listxattr
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-01-30 01:16:27 +05:30
Patrick Donnelly
1a0258ed3c
Merge PR #32644 into master
* refs/pull/32644/head:
	qa: ignore trimmed cache items for dead cache drop
	qa: use unit test comparisons

Reviewed-by: Zheng Yan <zyan@redhat.com>
2020-01-22 08:21:04 -08:00
Varsha Rao
d4b6a5c2d4 cephfs-shell: Instead of assert use stat for tests in rmdir
Using assert causes the tests to fail on teuthology due to this test being run
separately from the ceph cluster. Instead use stat for testing.

Signed-off-by: Varsha Rao <varao@redhat.com>
2020-01-22 16:30:11 +05:30
Varsha Rao
d21df7560e cephfs-shell: Add function for common rmdir test code
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-01-22 16:30:11 +05:30
Varsha Rao
c512b94127 cephfs-shell: Add rmdir test for non empty directory
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-01-22 16:30:11 +05:30