Commit Graph

794 Commits

Author SHA1 Message Date
Sage Weil
ac05b3568f qa: move dashboard e2e from cephadm -> rados suite
This test fails ~20% of the time.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-12 07:52:54 -05:00
Radoslaw Zarzynski
cec7c15f19 qa: use dump_metrics as alternative of get_heap_property
"get_heap_property *" asock commands are exposed to operators
to check the tcmalloc internals for understanding the performance
of the memory subsystem. but crimson uses the builtin seastar allocator
which is not backed by tcmalloc. but we can dump the metrics using
the "dump_metrics" asock command which is only available from
crimson-osd.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 14:24:23 +08:00
Neha Ojha
9241144022
Merge pull request #41487 from neha-ojha/wip-toc
qa/suites/rados/thrash-old-clients: remove luminous and mimic and use centos_latest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-05-24 14:44:18 -07:00
Neha Ojha
ece5ed1ac9
Merge pull request #41486 from neha-ojha/wip-49139-new
qa: use ubuntu_latest for perf suites and remove cosbench workloads

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-05-24 12:53:46 -07:00
Neha Ojha
30eb7467aa qa/suites/rados/thrash-old-clients: use centos_latest.yaml
use centos_latest instead of bionic because this is only common
distro for which we build packages for nautilus and above.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-24 18:34:04 +00:00
Sage Weil
04ce0496e8 Merge PR #41451 into master
* refs/pull/41451/head:
	qa/suites/rados: include rook test in rados

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2021-05-24 13:30:27 -04:00
Neha Ojha
b7237c9e2d qa/suites/rados/thrash-old-clients: remove luminous and mimic
We support N-3 client versions.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 22:03:41 +00:00
Neha Ojha
fd350fd015 qa: remove cosbench workloads from perf suites
Due to https://tracker.ceph.com/issues/49139

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 20:17:11 +00:00
Neha Ojha
5957d1797a qa: use ubuntu_latest for perf suites
Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-21 17:39:44 +00:00
Sage Weil
5db5c8c292 qa/suites/rados: include rook test in rados
This just to make sure we don't break mgr/orchestrator.

Note that we already symlink ../orch/cephadm, so this makes rados
include all of orch/.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-20 12:41:52 -05:00
Sage Weil
5b25f8a2e5 qa/suites: move rados/cephadm -> orch/cephadm; symlink
Move cephadm under orch/ top-level suite.  Symlink so that we
still include it in a rados run.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-05-18 11:14:14 -05:00
Neha Ojha
d3692a3e92
Merge pull request #40016 from neha-ojha/wip-default-mclock
use mclock_scheduler as the default scheduler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
2021-05-07 08:08:39 -07:00
Neha Ojha
c8e48c5c25 qa/suites/rados/standalone: remove mon_election symlink
The standalone tests need parameters to be passed as ceph_args to
override defaults.

This was just doubling the number of standalone tests being run in each rados
run with no effect!

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-05-07 00:42:53 +00:00
Sridhar Seshasayee
cc1fc98ea4 qa/suites/rados/mgr/tasks/progress: use high_recovery_ops for faster recovery
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2021-05-06 17:54:38 +00:00
Kefu Chai
fee3028abc
Merge pull request #41014 from smithfarm/wip-mempool-cacheline-49781
qa: verify the benefits of mempool cacheline optimization

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-04-30 19:36:17 +08:00
Loïc Dachary
7fe0ac7c11 qa: verify the benefits of mempool cacheline optimization
There already is a test to verify the mempool sharding works, in the sense that
it uses at least half of the variables available to count the number of
allocated objects and their total size. This new test verifies that, with
sharding, object counting is at least twice faster than without sharding. It
also collects cacheline contention data with the perf c2c tool. The manual
analysis of this data shows the optimization gain is indeed related to cacheline
contention.

Fixes: https://tracker.ceph.com/issues/49896

Signed-off-by: Loïc Dachary <loic@dachary.org>
2021-04-30 12:11:13 +08:00
Josh Durgin
0e273e6760
Merge pull request #40593 from ideepika/wip-new-testing-params
qa/config/rados: add dispatch delay testing params

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sam Just <sjust@redhat.com>
2021-04-28 11:26:58 -07:00
Sage Weil
75480f52e4 Merge PR #40941 into master
* refs/pull/40941/head:
	qa/suites/rados/cephadm/smoke-roleless: test client-keyring
	qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts
	cephadm: distribute client.admin keyring+conf to label:_admin on bootstrap
	doc/cephadm: document the default 'admin' label
	mgr/cephadm: 'ceph orch client-keyring ...' commands to manage keyring files
	mgr/cephadm: reimplement ceph.conf pushing
	mgr/cephadm: use _write_remote_file for ceph.conf
	mgr/cephadm: _write_remote_file helper
	mgr/cephadm: add placementspec for which hosts get ceph.conf

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Adam King <adking@redhat.com>
2021-04-28 14:26:35 -04:00
Sage Weil
e41931d042 qa/suites/rados/cephadm/smoke-roleless: test client-keyring
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-27 18:29:50 -04:00
Sage Weil
b0dcaf2cfa qa/tasks/cephadm.py: adjust client.admin key mode; place on all hosts
Except during upgrades, since it is not supported there.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-27 18:29:50 -04:00
Yuri Weinstein
b6c84d5621 qa/tests: changed simlink to upgrade/parallel only
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2021-04-23 08:20:01 -07:00
Deepika Upadhyay
b2c2a4326c qa/config/rados: add dispatch delay testing params
these parameters have proven to catch some of the uncaught bugs such as:
https://tracker.ceph.com/issues/48417, adopting them will help in
preventing more such hard to debug bugs.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-04-19 12:28:18 +05:30
Sage Weil
9c1f128885 qa/suites/rados/cephadm/smoke-*: use cephadm.wait_for_service
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-16 16:00:31 -04:00
Sage Weil
c20323d114 qa/suites/rados/cephadm/smoke-singlehost: test --single-host-defaults
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-16 16:00:31 -04:00
Sage Weil
16b30f2858 qa/suites/rados/cephadm/smoke-*: use cephadm.wait_for_service
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-16 09:49:45 -05:00
Sage Weil
3ff3f697b4 qa/suites/rados/cephadm/smoke-roleless: test rgw-ingress
Test this properly by downing each rgw and haproxy in turn and ensuring
that things remain up.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-16 09:49:45 -05:00
Sage Weil
7e16bf3468 qa/suites/rados/cephadm/smoke-roleless: add rgw-ingress test case
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-16 07:36:47 -05:00
Kefu Chai
d54605c7e6
Merge pull request #40837 from liewegas/cephadm-qa-shorten-thrash
qa/suites/rados/thrash: shorten radosbench

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-04-16 12:42:05 +08:00
Sage Weil
babbab14f4 qa/suites/rados/thrash: shorten radosbench
This is the longest of the thrash workloads; reducing it will bring
this test in line with the others (<= 45 min).

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-13 17:42:21 -05:00
Ilya Dryomov
7eb9c5ddb2 Merge branch 'master' into wip-unauthorized-gids
Sync up with master up to commit 3d8e73b266 ("Merge pull request
#40731 from tchaikov/wip-yamlize-options").  Specifically, bring in
src/common/options.cc yamlization and move new auth-related options
into src/common/options/global.yaml.in.

Conflicts:
	src/common/options.cc
	src/common/options/global.yaml.in

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-04-13 15:42:06 +02:00
Sage Weil
dcd90a1c8d Merge PR #40626 into master
* refs/pull/40626/head:
	qa/suites/rados/objectstore: separate store_test tests
	qa/standalone: split osd/ into 2 directories

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-04-12 22:38:49 -04:00
Sage Weil
d4ac4f3b78 qa/suites/rados/objectstore: separate store_test tests
This takes 5 hours currently.

- Separate out filestore and memstore into separate task (~1 hr)
- Split bluestore into -a and -b (a tests exclude SynethicMatrixC,
  b tests include it)

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-12 16:27:29 -05:00
Sage Weil
0f65e5cffa qa/standalone: split osd/ into 2 directories
The whole osd/ directory takes 3 hours to run.  Of that, about half is
osd-backfill*:

2021-04-05T20:38:55.932 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh...
2021-04-05T20:47:27.184 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh...
2021-04-05T20:55:59.497 INFO:tasks.workunit:Running workunit osd/osd-backfill-space.sh...
2021-04-05T21:48:47.549 INFO:tasks.workunit:Running workunit osd/osd-backfill-stats.sh...
2021-04-05T22:17:09.197 INFO:tasks.workunit:Running workunit osd/osd-bench.sh...

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-12 09:59:17 -05:00
Kefu Chai
caf364db5e
Merge pull request #38120 from kiizawa/wip-cls-remote-read
osd: allow remote read by calling cls method from within cls context

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-04-12 16:42:52 +08:00
Samuel Just
055ebe381d
Merge pull request #39216 from myoungwon/wip-manifest-dedup-test
osd, test: reworks for manifest dedup test cases

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-04-09 12:42:18 -07:00
Kefu Chai
39b2b5edc0 qa/tasks/mgr: skip test_diskprediction_local on python>=3.8
query the python version before trying to test diskprediction_local

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-04-07 21:27:44 +08:00
Sage Weil
3e80f61efe qa/suites/upgrade/octopus-x: disable insecure global_id reclaim health warnings
These will trigger on upgrade; suppress them so that our health gates
will still work.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-06 17:29:23 -04:00
Sage Weil
7ca7418322 cephadm: set auth_allow_insecure_global_id_reclaim for mon on bootstrap
If this is a fresh pacific cluster, let's assume that there won't be
legacy clients connecting.  (And if there are, let's put the burden on
the user to enable them to do so insecurely.)

This is in contrast to upgrades, where our focus is on not breaking
anything.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-06 17:28:55 -04:00
Ken Iizawa
7d0ea1b861 qa: allow all the tasks that run rados/test.sh to load object classes for testing remote-reads functionality
Fixes: https://tracker.ceph.com/issues/48182
Signed-off-by: Ken Iizawa <iizawa.ken@fujitsu.com>
2021-04-06 19:57:14 +09:00
Sage Weil
05b3ce2585 qa/tasks/cephadm: add apply() method/task
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-05 11:44:22 -05:00
myoungwon oh
b5f5649ecd qa: fix typo to call rollback op
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-04-01 18:44:37 +09:00
Sage Weil
64668275b5 qa/suites/rados/cephadm: stop testing on broken focal kubic podman
See https://tracker.ceph.com/issues/49633

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-29 07:51:08 -05:00
myoungwon oh
dfce69e995 qa: add dedup-io-snaps test
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-03-29 17:10:25 +09:00
myoungwon oh
9e8601ddf5 qa: add dedup-io-mixed test
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-03-29 17:09:49 +09:00
Sage Weil
fe9454722a Merge PR #40314 into master
* refs/pull/40314/head:
	qa/tasks/cephadm: use 'orch apply mon' to deploy mons

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
2021-03-23 22:58:14 -04:00
Sage Weil
a17593a177 qa/tasks/cephadm: use 'orch apply mon' to deploy mons
The 'orch daemon add ...' command is not idempotent and can cause
duplicate (and failing) attempts to add the same mon.

Maintain the ability to add mons the old way for the benefit of testing
upgrades from early octopus versions of cephadm.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-22 16:28:40 -05:00
Sage Weil
ccbf40317f Merge PR #40290 into master
* refs/pull/40290/head:
	qa/suites/rados/cephadm/upgrade: wait for rgw servicemap entries to refresh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-03-22 13:26:06 -04:00
Patrick Donnelly
6ad5901422
Merge PR #39191 into master
* refs/pull/39191/head:
	pybind/mgr/snap_schedule: use ceph VFS
	pybind/mgr/snap_schedule: idempotentize table creation
	mgr: add ceph sqlite VFS
	doc: add libcephsqlite
	ceph.spec,debian: package libcephsqlite
	test/libcephsqlite,qa: add tests for libcephsqlite
	libcephsqlite: rework architecture and backend
	SimpleRADOSStriper: wait for finished aios after write
	SimpleRADOSStriper: add new minimal async striper
	mon: define simple-rados-client-with-blocklist profile
	librados: define must renew lock flag
	common: add timeval conversion for durations
	Revert "libradosstriper: add function to read into char*"
	test_libcephsqlite: test random inserts
	cephsqlite: fix compiler errors
	cmake: improve build inst for cephsqlite
	libcephsqlite: sqlite interface to RADOS
	libradosstriper: add function to read into char*

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-03-22 10:06:11 -07:00
Kefu Chai
abee0b31e3
Merge pull request #40143 from neha-ojha/wip-objectstore-debug
qa/suites/rados/objectstore/backends: use default level for filestore

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-03-22 15:21:20 +08:00
Sage Weil
3f3d955b19 qa/suites/rados/cephadm/upgrade: wait for rgw servicemap entries to refresh
rgw changed the way it registered in the service map.  Wait a bit for
the old entries to be flushed out.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-21 11:22:07 -05:00