Commit Graph

2019 Commits

Author SHA1 Message Date
Sage Weil
4537b14b34 Merge PR #41574 into master
* refs/pull/41574/head:
	qa/tasks/vstart_runner: add LocalCluster.run
	qa/tasks/cephfs/test_nfs: fiddle with sudo
	mgr/nfs/export: some cleanup, minor refactoring
	mgr/nfs/cluster: remove unused @cluster_setter
	nfs/mgr: fix help message case
	doc/cephfs/fs-nfs-export: add note about export update behavior
	mgr/nfs: move user create/delete into helper
	mgr/nfs: refactor _delete_user helper
	mgr/nfs: refactor create_export_from_dict() helper
	mgr/nfs: keep 'nfs export get' around for backward-compat
	mgr/nfs: rename method
	qa/tasks/cephfs/test_nfs: test new export via apply
	doc/cephfs/fs-nfs-export: be consistent with cluster_id and _ vs -
	mgr/nfs: addr -> client_addr for 'nfs export create ...'
	mgr/nfs: fix tests
	mgr/nfs: 'nfs export get' -> 'nfs export info'
	mgr/nfs: binding -> pseudo_path
	mgr/nfs: more revisions based on review
	mgr/nfs: adjust NFSExceptoin errno arg
	doc/cephfs: update 'nfs export {get,apply}' docs
	mgr/nfs: merge FSExport back into ExportMgr
	doc/radosgw/nfs: document mgr/nfs way to add/remove rgw exports
	mgr/nfs: merge 'nfs export {update,import}' -> 'nfs export apply'
	mgr/nfs: test export creation and list
	mgr/nfs: test export_update (+ fixes)
	mgr/nfs: test Export.validate(); several fixes
	mgr/nfs: test that export <-> block+dict conversions go both ways
	mgr/nfs: clean up test a bit
	mgr/nfs/export: fix export validation
	mgr/nfs/export: fix tests
	mgr/nfs: handle option addr/client block in create_export()
	mgr/nfs: allow multiple addrs for new exports
	mgr/nfs: fix/finish rgw export
	mgr/nfs/module: clusterid -> cluster_id
	mgr/nfs/export: fix export_update_1 to type check
	mgr/nfs/cluster: fix type error
	mgr/nfs/export: wrap long lines
	mgr/nfs: ExportMgr._delete_export only works for cephfs for now
	mgr/nfs: Remove pool_ns from NFSCluster
	mgr/nfs: Remove ExportMgr.rados_namespace
	mgr/nfs: flake8
	mgr/nfs: Add type checking
	mgr/nfs: Add __eq__ method to Export
	mgr/nfs: Add some compatibility to mgr/dashboard
	mgr/nfs: Fix whitespace handling
	mgr/nfs: Copy unit tests from mgr/dashboard
	mgr/nfs: partially implement rgw export support
	mgr/nfs: abstract FSAL; add RGWFSAL
	mgr/nfs: refactor to merge 'update' and 'import' code
	mgr/nfs: add 'nfs export import' command
	mgr/nfs: refactor 'nfs export update' and export validation
	mgr/nfs: fix _fetch_export to distinguish between clusters
	mgr/nfs: move export ganesha conf translation into caller
	mgr/nfs: name nfs cephfs client key 'nfs.{cluster_id}.{export_id}'
	mgr/nfs: add --addr to 'nfs export create'
	mgr/nfs: add --squash to 'nfs export create'
	mgr/nfs/export_utils: include false but non-None items in config
	vstart.sh: enable nfs module
	mgr/cephadm: nfs: drop attr_expiration_time from top-level config
	mgr/cephadm: remove Dir_Chunk = 0

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-06-26 10:41:27 -04:00
Kefu Chai
29064f1bf8
Merge pull request #41937 from liewegas/mgr-crash
mgr: generate crash dumps for Python exceptions in mgr modules

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-26 22:18:14 +08:00
Sage Weil
3edc04a46b qa/suites/rados/mgr: whitelist module crash during selftest
One of the selftests triggers an exception from serve().

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-25 13:48:45 -04:00
Sage Weil
fe9963b03c qa/suites/rados/dashboard: fix e2e test
Move roles into task yaml.  Rename e2e.

Fixes: https://tracker.ceph.com/issues/51292
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-23 09:54:40 -05:00
Sage Weil
5a2382a5a8 mgr/nfs: binding -> pseudo_path
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-21 14:13:15 -04:00
Patrick Donnelly
b946d5a6b2
qa: use centos latest for fs:upgrade
Fixes: https://tracker.ceph.com/issues/51250
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-16 12:16:36 -07:00
Sage Weil
091a32e130 Merge PR #41844 into master
* refs/pull/41844/head:
	qa/suites/orch/cephadm/dashboard: remove remaining bits

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-06-15 15:42:26 -04:00
Patrick Donnelly
03674f5197
Merge PR #41821 into master
* refs/pull/41821/head:
	qa: specify distro for fs:bugs

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-06-15 10:33:38 -07:00
Sage Weil
ebb5a3f0bc qa/suites/orch/cephadm/dashboard: remove remaining bits
Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-14 13:00:45 -05:00
Sage Weil
9074e87611 Merge PR #41827 into master
* refs/pull/41827/head:
	qa: move dashboard e2e from cephadm -> rados suite

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2021-06-14 09:11:04 -04:00
Sage Weil
ac05b3568f qa: move dashboard e2e from cephadm -> rados suite
This test fails ~20% of the time.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-12 07:52:54 -05:00
Patrick Donnelly
d6c66f3fa6
qa,pybind/mgr: allow disabling .mgr pool
This is mostly for testing: a lot of tests assume that there are no
existing pools. These tests relied on a config to turn off creating the
"device_health_metrics" pool which generally exists for any new Ceph
cluster. It would be better to make these tests tolerant of the new .mgr
pool but clearly there's a lot of these. So just convert the config to
make it work.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-11 19:35:17 -07:00
Patrick Donnelly
71d2c81d41
qa: add upgrade test for devicehealth
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-11 19:35:17 -07:00
Patrick Donnelly
95f0e9c959
Merge PR #39505 into master
* refs/pull/39505/head:
	qa: test nowsync option in kernel client workflows
	qa: deep merge top level overrides for fuse/kclient

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
2021-06-11 17:10:41 -07:00
Patrick Donnelly
8eeb1455ee
qa: specify distro for fs:bugs
Fixes: https://tracker.ceph.com/issues/51184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-11 16:25:50 -07:00
Patrick Donnelly
26605723cf
qa: update cephfs-shell distro to ubuntu 20.04
18.04 is no longer built.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-09 16:51:49 -07:00
Sage Weil
64281bb394 Merge PR #41229 into master
* refs/pull/41229/head:
	qa/suites/upgrade/pacific-x/stress-split: add

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2021-06-09 15:55:25 -04:00
Patrick Donnelly
84ae38594d
qa: test nowsync option in kernel client workflows
Fixes: https://tracker.ceph.com/issues/49341
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 19:15:12 -07:00
Kefu Chai
104f054cee qa/suites/orch/rook/smoke: stop testing on ubuntu 18.04
even rook does not really install ceph packages in the host directly, it
uses the ceph container image. but teuthology insists on checking the
existence of debian packages by querying shaman server when it sees a
teuthology facet file which includes:

os_type: ubuntu
os_version: "18.04"

but since we've stopped building ubuntu/bionic packages, teuthology
just complains when we are scheduling test suites which are composed
from facets in qa/suites/orch/rook/smoke.

in this change, the ubuntu_18.04.yaml is dropped because ubuntu/bionic
does not really increase the test coverage of ceph. it helps to test
the rook and container runtime though.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-05 01:11:14 +08:00
Kefu Chai
dba26fc7a8
Merge pull request #41652 from tchaikov/wip-qa-asock-or
qa/tasks/admin_socket: support "foo || bar" as command

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-06-04 13:50:38 +08:00
Patrick Donnelly
a52712f955
Merge PR #41443 into master
* refs/pull/41443/head:
	test: update log-ignorelist for fs:mirror test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:23:17 -07:00
Patrick Donnelly
4e1f812461
Merge PR #39910 into master
* refs/pull/39910/head:
	test: Add test for mgr hang when osd is full
	mgr: Set client_check_pool_perm to false
	mds: Add full caps to avoid osd full check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:22:23 -07:00
Radoslaw Zarzynski
cec7c15f19 qa: use dump_metrics as alternative of get_heap_property
"get_heap_property *" asock commands are exposed to operators
to check the tcmalloc internals for understanding the performance
of the memory subsystem. but crimson uses the builtin seastar allocator
which is not backed by tcmalloc. but we can dump the metrics using
the "dump_metrics" asock command which is only available from
crimson-osd.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 14:24:23 +08:00
Yuval Lifshitz
679ddf5d11
Merge pull request #41026 from TRYTOBE8TME/wip-rgw-rabbitmq
qa/tasks: Adding RabbitMQ task for bucket notification tests
2021-06-02 07:47:39 +03:00
Kalpesh
2e0b8a2a1f qa/tasks: Adding RabbitMQ task for bucket notification tests
This commit majorly consists of the RabbitMQ task which is a required and supported endpoint in bucket notification tests.
And some related changes in the AMQP tests. Major changes are:
1. Addition of RabbitMQ task
2. Documentation update for the steps to execute AMQP tests
3. Addition of attributes to the tests
4. Tox dependency removal from kafka.py

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2021-06-01 23:34:31 +05:30
Sage Weil
9ab9cc26e2 Merge PR #41007 into master
* refs/pull/41007/head:
	qa/tasks/cephfs/test_nfs: fix info test
	doc/cephfs/fs-nfs-exports: document --ingress --virtual-ip
	mgr/nfs: move ingress vs virtual_ip check to cluster interface
	PendingReleaseNotes: clarify deprecated
	PendingReleaseNotes: note breaking CLI changes
	doc/cephadm/nfs: document nfs+ingress
	qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
	mgr/nfs: take --ingress argument to 'nfs cluster create'
	mgr/cephadm: adjust debug output for device refresh
	mgr/cephadm: ingress: fix log msg
	mgr/cephadm: fix logging of config/placement errors
	common/options: enable nfs module for new clusters
	cephadm: --stop-signal=SIGTERM
	mgr/orchestrator: default nfs pool, namespaces
	mgr/cephadm: nfs: create pool if it doesn't yet exist
	doc/cephadm/nfs: update
	mgr/nfs: change 'nfs cluster info'
	mgr/nfs: take optional virtual_ip for deploying ingress
	mgr/nfs: remove 'nfs cluster update'
	mgr/nfs: factor out ganesha pool creation
	mgr/nfs: delete -> rm for CLI
	mgr/nfs: add some type annotations
	python-common: fix IngressSpec yaml dump
	mgr/cephadm: ingress: remove eth0 default
	qa/tasks/cephadm: allow mounting volumes in shell
	cephadm: add -v arg to shell
	qa/tasks/vip: add 'vip.exec' task
	mgr/orchestrator: add --port arg to 'orch apply nfs'
	mgr/cephadm: nfs: add purge
	mgr/cephadm: ingress: support nfs
	mgr/cephadm: do not reconfigure daemons on deleted services
	mgr/cephadm: nfs: shell out to rados tool for conf creation
	mgr/cephadm: nfs: add rank to grace file from mgr module
	mgr/cephadm: nfs: bind ganesha to appropriate ip:port
	mgr/cephadm: enable ranked daemons for nfs
	mgr/cephadm: support creation of daemons with ranks
	mgr/cephadm: make _plan show removed daemon names
	mgr/cephadm/schedule: assign/map ranks
	mgr/cephadm: add rank[_generation] properties
	mgr/cephadm/inventory: store optional rank_map along with specs
	mgr/cephadm: include service_name is generated DaemonDescription
	mgr/orchestrator: include service_name in DaemonDescription dump
	mgr/cephadm/inventory: fix deleted check
	mgr/cephadm: simplify
	mgr/cephadm/schedule: make placement shuffle deterministic
	mgr/cephadm: document CephadmService flags

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2021-05-25 16:17:44 -04:00
Sage Weil
e2db373698 qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
Still missing a full client mount test, though!

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-25 10:15:45 -04:00
Venky Shankar
d01b8c8566 test: update log-ignorelist for fs:mirror test
Also, have a fs:mirror specific whitelist override since
the current one is a symlink to a common YAML probably used
by other tests.

Fixes: http://tracker.ceph.com/issues/50867
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-25 04:43:48 -04:00
Neha Ojha
9241144022
Merge pull request #41487 from neha-ojha/wip-toc
qa/suites/rados/thrash-old-clients: remove luminous and mimic and use centos_latest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-05-24 14:44:18 -07:00
Neha Ojha
ece5ed1ac9
Merge pull request #41486 from neha-ojha/wip-49139-new
qa: use ubuntu_latest for perf suites and remove cosbench workloads

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-05-24 12:53:46 -07:00
Yuri Weinstein
9305b8c7ea
Merge pull request #41504 from yuriw/wip-yuriw-master
qa/tests - removed ref to 18.04 distro as it's not supported on master+

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-05-24 12:21:00 -07:00
Neha Ojha
30eb7467aa qa/suites/rados/thrash-old-clients: use centos_latest.yaml
use centos_latest instead of bionic because this is only common
distro for which we build packages for nautilus and above.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-24 18:34:04 +00:00
Yuri Weinstein
d8794db5b0 qa/tests - removed ref to 18.04 distro as it's not supported on master+
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2021-05-24 10:40:23 -07:00
Sage Weil
04ce0496e8 Merge PR #41451 into master
* refs/pull/41451/head:
	qa/suites/rados: include rook test in rados

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2021-05-24 13:30:27 -04:00
Kotresh HR
2bd6ba8026 test: Add test for mgr hang when osd is full
Add fs suite for tests requiring one node as well.

Fixes: https://tracker.ceph.com/issues/50532
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-05-22 19:09:07 +05:30
Matt Benjamin
5276c2a3d0
Merge pull request #40910 from galsalomon66/update_s3select_submodule_and_s3tests_head_10apr
rgw/s3select: update s3select submodule to last commit, (new features), update for test coverage(s3test)
2021-05-21 19:17:53 -04:00
Neha Ojha
b7237c9e2d qa/suites/rados/thrash-old-clients: remove luminous and mimic
We support N-3 client versions.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 22:03:41 +00:00
Neha Ojha
fd350fd015 qa: remove cosbench workloads from perf suites
Due to https://tracker.ceph.com/issues/49139

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 20:17:11 +00:00
gal salomon
7c9ac4ed3d update to s3select/master (new features)
Signed-off-by: gal salomon <gal.salomon@gmail.com>

force-branch to s3test/master

Signed-off-by: gal salomon <gal.salomon@gmail.com>

update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

closing documentation gaps between previous and current functionalities(WIP)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

additional features

Update s3select.rst

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

adding cast to bool
Signed-off-by: gal salomon <gal.salomon@gmail.com>

editorial; more description for substring and trim

Signed-off-by: gal salomon <gal.salomon@gmail.com>
2021-05-21 21:58:20 +03:00
Neha Ojha
5957d1797a qa: use ubuntu_latest for perf suites
Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 17:39:44 +00:00
Sage Weil
5db5c8c292 qa/suites/rados: include rook test in rados
This just to make sure we don't break mgr/orchestrator.

Note that we already symlink ../orch/cephadm, so this makes rados
include all of orch/.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-20 12:41:52 -05:00
Sage Weil
b9d8dc483a Merge PR #41286 into master
* refs/pull/41286/head:
	qa/suites/orch/rook: disable centos for now
	qa/suites/orch/rook/smoke: initial smoke suite
	qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
	qa/tasks/rook: simplify shutdown
	qa/tasks/rook: archive logs
	qa/tasks/rook: more orderly cluster teardown
	qa/tasks/rook: deploy ceph via rook on top of kubernetes
	qa/tasks/kubeadm: install kubernetes with kubeadm
	qa/suites: move rados/cephadm -> orch/cephadm; symlink
	qa/tasks/cephadm: add whitespace between functions
	qa/tasks/cephadm: clean up ctx.manager setup

Reviewed-by: Sébastien Han <seb@redhat.com>
2021-05-19 07:55:30 -04:00
Sage Weil
ad8096e0a0 qa/suites/orch/rook: disable centos for now
We are hitting an selinux denial:

type=AVC msg=audit(1621353663.595:6016): avc: denied { write } for pid=28812 comm="cephcsi" name="pids.max" dev="cgroup" ino=2933 scontext=system_u:system_r:container_t:s0:c43,c177 tcontext=system_u:object_r:cgroup_t:s0 tclass=file permissive=1

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-18 15:19:16 -05:00
Sage Weil
ab33ca3655 qa/suites/orch/rook/smoke: initial smoke suite
- os: 18.04, 20.04, centos, centos+kubic
- nodes: 1 or 3
- rook: 1.6.2 or master
- k8s: 1.21 always (for now)
- net: calico always (flannel + centos is wonky)
- workload: none, rados bench

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-18 15:19:16 -05:00
Sage Weil
5b25f8a2e5 qa/suites: move rados/cephadm -> orch/cephadm; symlink
Move cephadm under orch/ top-level suite.  Symlink so that we
still include it in a rados run.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-18 11:14:14 -05:00
Casey Bodley
46c0292870 rgw: remove the civetweb frontend from src and qa
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2021-05-17 15:00:31 -04:00
Sage Weil
7ba8692a2e qa/suites/upgrade/pacific-x/stress-split: add
Copied from octopus and adjusted starting version

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-07 13:23:04 -05:00
Neha Ojha
d3692a3e92
Merge pull request #40016 from neha-ojha/wip-default-mclock
use mclock_scheduler as the default scheduler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
2021-05-07 08:08:39 -07:00
Neha Ojha
c8e48c5c25 qa/suites/rados/standalone: remove mon_election symlink
The standalone tests need parameters to be passed as ceph_args to
override defaults.

This was just doubling the number of standalone tests being run in each rados
run with no effect!

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-07 00:42:53 +00:00
Sridhar Seshasayee
cc1fc98ea4 qa/suites/rados/mgr/tasks/progress: use high_recovery_ops for faster recovery
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2021-05-06 17:54:38 +00:00