Commit Graph

8401 Commits

Author SHA1 Message Date
Sage Weil
b18427da4b Merge PR #41509 into master
* refs/pull/41509/head:
	common/cmdparse: fix CephBool validation for tell commands
	mgr/nfs: fix 'nfs export create' argument order
	common/cmdparse: emit proper json
	mon/MonCommands: add -- seperator to example
	qa/tasks/cephfs/test_nfs: fix export create test
	mgr: make mgr commands compat with pre-quincy mon
	doc/_ext/ceph_commands: handle non-positional args in docs
	mgr: fix reweight-by-utilization cephbool flag
	mon/MonCommands: convert some CephChoices to CephBool
	mgr/k8sevents: fix help strings
	pybind/mgr/mgr_module: fix help desc formatting
	mgr/orchestrator: clean up 'orch {daemon add,apply} rgw' args
	mgr/orchestrator: add end_positional to a few methods
	mgr/orchestrator: reformat a few methods
	pybind/ceph_argparse: stop parsing when we run out of positional args
	pybind/ceph_argparse: remove dead code
	pybind/mgr/mgr_module: infer non-positional args
	pybind/mgr/mgr_module: add separator for non-positional args
	command/cmdparse: use -- to separate positional from non-positional args
	pybind/ceph_argparse: adjust help text for non-positional args
	pybind/ceph_argparse: track a 'positional' property on cli args

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-07 10:02:52 -04:00
Sage Weil
8683cccd06 qa/tasks/cephfs/test_nfs: fix export create test
Everything after --readonly is non-positional.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-04 16:56:17 -04:00
Kefu Chai
104f054cee qa/suites/orch/rook/smoke: stop testing on ubuntu 18.04
even rook does not really install ceph packages in the host directly, it
uses the ceph container image. but teuthology insists on checking the
existence of debian packages by querying shaman server when it sees a
teuthology facet file which includes:

os_type: ubuntu
os_version: "18.04"

but since we've stopped building ubuntu/bionic packages, teuthology
just complains when we are scheduling test suites which are composed
from facets in qa/suites/orch/rook/smoke.

in this change, the ubuntu_18.04.yaml is dropped because ubuntu/bionic
does not really increase the test coverage of ceph. it helps to test
the rook and container runtime though.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-05 01:11:14 +08:00
Kefu Chai
dba26fc7a8
Merge pull request #41652 from tchaikov/wip-qa-asock-or
qa/tasks/admin_socket: support "foo || bar" as command

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-06-04 13:50:38 +08:00
Patrick Donnelly
a12db7941b
Merge PR #41499 into master
* refs/pull/41499/head:
	qa/tasks/mds_thrash: fix thrash iteration never skip

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:33:27 -07:00
Patrick Donnelly
a52712f955
Merge PR #41443 into master
* refs/pull/41443/head:
	test: update log-ignorelist for fs:mirror test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:23:17 -07:00
Patrick Donnelly
4e1f812461
Merge PR #39910 into master
* refs/pull/39910/head:
	test: Add test for mgr hang when osd is full
	mgr: Set client_check_pool_perm to false
	mds: Add full caps to avoid osd full check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:22:23 -07:00
Neha Ojha
11252f6117
Merge pull request #41308 from sseshasa/wip-osd-benchmark-for-mclock
osd: Run osd bench test to override default max osd capacity for mclock

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-06-03 08:39:22 -07:00
Radoslaw Zarzynski
cec7c15f19 qa: use dump_metrics as alternative of get_heap_property
"get_heap_property *" asock commands are exposed to operators
to check the tcmalloc internals for understanding the performance
of the memory subsystem. but crimson uses the builtin seastar allocator
which is not backed by tcmalloc. but we can dump the metrics using
the "dump_metrics" asock command which is only available from
crimson-osd.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 14:24:23 +08:00
Kefu Chai
83e4edcd80 qa/tasks/admin_socket: support "foo || bar" as command
so we can cater the needs of different implementation of osd, i.e.,
classic osd and crimson osd. they offer different set of asock commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 14:23:46 +08:00
Patrick Donnelly
5871240363
Merge PR #41635 into master
* refs/pull/41635/head:
	qa: increase fragmentation to improve uniform distribution

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-06-02 08:18:22 -07:00
Sridhar Seshasayee
328271d587 qa/tasks: Enhance wait_until_true() to check & retry recovery progress
With mclock scheduler enabled, the recovery throughput is throttled based
on factors like the type of mclock profile enabled, the OSD capacity among
others. Due to this the recovery times may vary and therefore the existing
timeout of 120 secs may not be sufficient.

To address the above, a new method called _is_inprogress_or_complete() is
introduced in the TestProgress Class that checks if the event with the
specified 'id' is in progress by checking the 'progress' key of the
progress command response. This method also handles the corner case where
the event completes just before it's called.

The existing wait_until_true() method in the CephTestCase Class is
modified to accept another function argument called "check_fn". This is
set to the _is_inprogress_or_complete() function described earlier in the
"test_turn_off_module" test that has been observed to fail due to the
reasons already described above. A retry mechanism of a maximum of 5
attempts is introduced after the first timeout is hit. This means that
the wait can extend up to a maximum of 600 secs (120 secs * 5) as long as
there is recovery progress reported by the 'ceph progress' command result.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2021-06-02 14:19:48 +05:30
Yuval Lifshitz
679ddf5d11
Merge pull request #41026 from TRYTOBE8TME/wip-rgw-rabbitmq
qa/tasks: Adding RabbitMQ task for bucket notification tests
2021-06-02 07:47:39 +03:00
Patrick Donnelly
1c40ee32f9
qa: increase fragmentation to improve uniform distribution
Fixes: https://tracker.ceph.com/issues/51060
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-01 20:25:17 -07:00
Ilya Dryomov
bb92c4ff83
Merge pull request #41588 from idryomov/wip-rbd-trash-purge
librbd: don't stop at the first unremovable image when purging 

Reviewed-by: Mykola Golub <mgolub@suse.com>
2021-06-01 21:56:57 +02:00
Kalpesh
2e0b8a2a1f qa/tasks: Adding RabbitMQ task for bucket notification tests
This commit majorly consists of the RabbitMQ task which is a required and supported endpoint in bucket notification tests.
And some related changes in the AMQP tests. Major changes are:
1. Addition of RabbitMQ task
2. Documentation update for the steps to execute AMQP tests
3. Addition of attributes to the tests
4. Tox dependency removal from kafka.py

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2021-06-01 23:34:31 +05:30
Ilya Dryomov
dcd193c35e qa/tasks/qemu: precise repos have been archived
Fixes: https://tracker.ceph.com/issues/51033
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-06-01 12:54:16 +02:00
Ilya Dryomov
16d9a68a3e librbd: don't stop at the first unremovable image when purging
As there is no inherent ordering, there may be multiple removable
images past the unremovable image.  On top of that, removing a clone
may make its parent removable so perform an additional pass if any
image gets removed.

Fixes: https://tracker.ceph.com/issues/51021
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-05-31 11:44:47 +02:00
Milind Changire
26fbbefa82
Merge pull request #40831 from vshankar/wip-cephfs-mirror-incremental-sync
cephfs-mirror: incremental sync

Reviewed-by: Milind Changire <mchangir@redhat.com>
2021-05-27 13:39:23 +05:30
Ilya Dryomov
dfa0164f71
Merge pull request #41279 from pkalever/promote-attach
rbd: promote rbd-nbd attach and detach at rbd integrated cli

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2021-05-27 09:58:32 +02:00
Prasanna Kumar Kalever
e2aa4128f1 qa/workunits/rbd: use rbd cli for device attach/detach commands
Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2021-05-26 10:27:09 +05:30
Sage Weil
9ab9cc26e2 Merge PR #41007 into master
* refs/pull/41007/head:
	qa/tasks/cephfs/test_nfs: fix info test
	doc/cephfs/fs-nfs-exports: document --ingress --virtual-ip
	mgr/nfs: move ingress vs virtual_ip check to cluster interface
	PendingReleaseNotes: clarify deprecated
	PendingReleaseNotes: note breaking CLI changes
	doc/cephadm/nfs: document nfs+ingress
	qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
	mgr/nfs: take --ingress argument to 'nfs cluster create'
	mgr/cephadm: adjust debug output for device refresh
	mgr/cephadm: ingress: fix log msg
	mgr/cephadm: fix logging of config/placement errors
	common/options: enable nfs module for new clusters
	cephadm: --stop-signal=SIGTERM
	mgr/orchestrator: default nfs pool, namespaces
	mgr/cephadm: nfs: create pool if it doesn't yet exist
	doc/cephadm/nfs: update
	mgr/nfs: change 'nfs cluster info'
	mgr/nfs: take optional virtual_ip for deploying ingress
	mgr/nfs: remove 'nfs cluster update'
	mgr/nfs: factor out ganesha pool creation
	mgr/nfs: delete -> rm for CLI
	mgr/nfs: add some type annotations
	python-common: fix IngressSpec yaml dump
	mgr/cephadm: ingress: remove eth0 default
	qa/tasks/cephadm: allow mounting volumes in shell
	cephadm: add -v arg to shell
	qa/tasks/vip: add 'vip.exec' task
	mgr/orchestrator: add --port arg to 'orch apply nfs'
	mgr/cephadm: nfs: add purge
	mgr/cephadm: ingress: support nfs
	mgr/cephadm: do not reconfigure daemons on deleted services
	mgr/cephadm: nfs: shell out to rados tool for conf creation
	mgr/cephadm: nfs: add rank to grace file from mgr module
	mgr/cephadm: nfs: bind ganesha to appropriate ip:port
	mgr/cephadm: enable ranked daemons for nfs
	mgr/cephadm: support creation of daemons with ranks
	mgr/cephadm: make _plan show removed daemon names
	mgr/cephadm/schedule: assign/map ranks
	mgr/cephadm: add rank[_generation] properties
	mgr/cephadm/inventory: store optional rank_map along with specs
	mgr/cephadm: include service_name is generated DaemonDescription
	mgr/orchestrator: include service_name in DaemonDescription dump
	mgr/cephadm/inventory: fix deleted check
	mgr/cephadm: simplify
	mgr/cephadm/schedule: make placement shuffle deterministic
	mgr/cephadm: document CephadmService flags

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2021-05-25 16:17:44 -04:00
Sage Weil
218eec938d qa/tasks/cephfs/test_nfs: fix info test
Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-25 10:15:45 -04:00
Sage Weil
e2db373698 qa/suites/rados/cephadm/smoke-roleless: test nfs, nfs + ingress
Still missing a full client mount test, though!

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-25 10:15:45 -04:00
Venky Shankar
da9788a9b5 test: add test to verify incremental snapshot updates
Fixes: http://tracker.ceph.com/issues/49939
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-25 08:45:44 -04:00
Venky Shankar
d01b8c8566 test: update log-ignorelist for fs:mirror test
Also, have a fs:mirror specific whitelist override since
the current one is a symlink to a common YAML probably used
by other tests.

Fixes: http://tracker.ceph.com/issues/50867
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-05-25 04:43:48 -04:00
Sunil Kumar Nagaraju
a65ca4e8ed
Merge pull request #41414 from sunilkumarn417/rh_downstream
qa/tasks/cephadm.py: Include bootstrap registry options for downstream
2021-05-25 09:00:07 +05:30
Neha Ojha
9241144022
Merge pull request #41487 from neha-ojha/wip-toc
qa/suites/rados/thrash-old-clients: remove luminous and mimic and use centos_latest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-05-24 14:44:18 -07:00
Neha Ojha
ece5ed1ac9
Merge pull request #41486 from neha-ojha/wip-49139-new
qa: use ubuntu_latest for perf suites and remove cosbench workloads

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-05-24 12:53:46 -07:00
Yuri Weinstein
9305b8c7ea
Merge pull request #41504 from yuriw/wip-yuriw-master
qa/tests - removed ref to 18.04 distro as it's not supported on master+

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-05-24 12:21:00 -07:00
Neha Ojha
30eb7467aa qa/suites/rados/thrash-old-clients: use centos_latest.yaml
use centos_latest instead of bionic because this is only common
distro for which we build packages for nautilus and above.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-24 18:34:04 +00:00
Yuri Weinstein
d8794db5b0 qa/tests - removed ref to 18.04 distro as it's not supported on master+
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2021-05-24 10:40:23 -07:00
Sage Weil
04ce0496e8 Merge PR #41451 into master
* refs/pull/41451/head:
	qa/suites/rados: include rook test in rados

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2021-05-24 13:30:27 -04:00
Lianne
2b50cefa89 qa/tasks/mds_thrash: fix thrash iteration never skip
Signed-off-by: Lianne <liyan.wang@xtaotech.com>
2021-05-24 17:17:44 +08:00
sunilkumarn417
364fb5899b qa/tasks/cephadm: Include bootstrap registry options for downstream
- registry-url, registry-username and registry-password bootstrap options are
supported now. This is needed to access monitoring service container images.
- usage of RHEL distribution based cephadm in download_cephadm task.

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>
2021-05-24 12:38:54 +05:30
Kotresh HR
2bd6ba8026 test: Add test for mgr hang when osd is full
Add fs suite for tests requiring one node as well.

Fixes: https://tracker.ceph.com/issues/50532
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2021-05-22 19:09:07 +05:30
Matt Benjamin
5276c2a3d0
Merge pull request #40910 from galsalomon66/update_s3select_submodule_and_s3tests_head_10apr
rgw/s3select: update s3select submodule to last commit, (new features), update for test coverage(s3test)
2021-05-21 19:17:53 -04:00
Neha Ojha
b7237c9e2d qa/suites/rados/thrash-old-clients: remove luminous and mimic
We support N-3 client versions.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 22:03:41 +00:00
Neha Ojha
fd350fd015 qa: remove cosbench workloads from perf suites
Due to https://tracker.ceph.com/issues/49139

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 20:17:11 +00:00
gal salomon
7c9ac4ed3d update to s3select/master (new features)
Signed-off-by: gal salomon <gal.salomon@gmail.com>

force-branch to s3test/master

Signed-off-by: gal salomon <gal.salomon@gmail.com>

update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

closing documentation gaps between previous and current functionalities(WIP)

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

additional features

Update s3select.rst

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>

Update s3select.rst

adding cast to bool
Signed-off-by: gal salomon <gal.salomon@gmail.com>

editorial; more description for substring and trim

Signed-off-by: gal salomon <gal.salomon@gmail.com>
2021-05-21 21:58:20 +03:00
Neha Ojha
5957d1797a qa: use ubuntu_latest for perf suites
Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 17:39:44 +00:00
Sage Weil
8b95c4b7c5 qa/tasks/cephadm.conf: log_to_journald=false
For teuthology runs, we set log_to_stderr=false, so that we only see
derr-level events in the container log (and teuthology.log).  Now that we
log directly to journald, set log_to_journald=false too, so that we don't
see level-20 logs in teuthology.log.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-21 09:54:46 -04:00
Sage Weil
ad0a62dc9d mgr/nfs: delete -> rm for CLI
The rest of the CLI uses 'rm' in place of 'remove' or 'delete', so let's
deprecate 'delete' and add 'rm'.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-20 18:38:18 -04:00
Patrick Donnelly
21b69fda3c
Merge PR #41084 into master
* refs/pull/41084/head:
	test: test to verify dir path removal when no mirror daemons are running
	pybind/mirroring: advance state machine from stalled state
	pybind/mirroring: start from correct state during policy init

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-20 12:42:19 -07:00
Sage Weil
5db5c8c292 qa/suites/rados: include rook test in rados
This just to make sure we don't break mgr/orchestrator.

Note that we already symlink ../orch/cephadm, so this makes rados
include all of orch/.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-20 12:41:52 -05:00
Ilya Dryomov
338920fcf6
Merge pull request #41070 from sunnyku/wip-rbd-50522
rbd: fix default pool handling for nbd map/unmap

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2021-05-20 16:41:22 +02:00
Kefu Chai
84850b135d
Merge pull request #41258 from ronen-fr/wip-ronenf-repair-stat
osd/scrub: separate between PG state flags and internal scrubber operation

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-05-20 14:29:05 +08:00
Sage Weil
b711a75277 qa/tasks/cephadm: allow mounting volumes in shell
Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-19 08:43:14 -04:00
Sage Weil
54542fdaab qa/tasks/vip: add 'vip.exec' task
Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-19 08:43:14 -04:00
Sage Weil
b9d8dc483a Merge PR #41286 into master
* refs/pull/41286/head:
	qa/suites/orch/rook: disable centos for now
	qa/suites/orch/rook/smoke: initial smoke suite
	qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
	qa/tasks/rook: simplify shutdown
	qa/tasks/rook: archive logs
	qa/tasks/rook: more orderly cluster teardown
	qa/tasks/rook: deploy ceph via rook on top of kubernetes
	qa/tasks/kubeadm: install kubernetes with kubeadm
	qa/suites: move rados/cephadm -> orch/cephadm; symlink
	qa/tasks/cephadm: add whitespace between functions
	qa/tasks/cephadm: clean up ctx.manager setup

Reviewed-by: Sébastien Han <seb@redhat.com>
2021-05-19 07:55:30 -04:00