Commit Graph

491 Commits

Author SHA1 Message Date
Sage Weil
5def1df5e8 Merge PR #31064 into master
* refs/pull/31064/head:
	test: Test balancer module commands
	mgr: Improve balancer module status
	mgr: Release GIL before calling OSDMap::calc_pg_upmaps()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-11-07 10:57:56 -06:00
Lenz Grimmer
fe8f786b6e
mgr/dashboard: add missing test_orchestrator suite (#31198)
mgr/dashboard: add missing test_orchestrator suite

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-11-06 14:35:43 +00:00
Sebastian Wagner
157fb06fac mgr/orchestrator: check for DEVICE_{IDENT|FAULT}_ON
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-11-05 13:02:29 +01:00
Tatjana Dehler
8244028ca2 mgr/dashboard: add missing test_orchestrator suite
Fixes: https://tracker.ceph.com/issues/42244
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-11-04 13:49:29 +01:00
Sage Weil
d927374bb4 Merge PR #31168 into master
* refs/pull/31168/head:
	ceph-daemon: try py2 import before py3
	qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed
	qa/standalone/test_ceph_damon.sh: test with python2 and python3
	mgr/ssh: python, not python3
	ceph-daemon: python, not python3
	ceph-daemon: os.makedirs
	ceph-daemon: configparser is ConfigParser on py2
	ceph-daemon: avoid py3-isms

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>
2019-10-28 14:59:43 -05:00
Sage Weil
9fe9653c8c qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed
Centos7 doesn't have it by default.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-28 12:15:47 -05:00
Kefu Chai
674bd8a9e6
Merge pull request #30434 from smithfarm/wip-41820
qa: enable dashboard tests to be run with "--suite rados/dashboard"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-10-27 09:18:16 +08:00
David Zafman
3a0e2c8ff1 test: Test balancer module commands
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-10-24 18:56:19 -07:00
Sage Weil
47777b9c0d qa/suites/rados/singleton-nomsgr/ceph-daemon: run test_ceph_daemon.sh
Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-23 15:08:55 -05:00
Nathan Cutler
493ee6d78f qa: enable dashboard tests to be run with "--suite rados/dashboard"
This moves dashboard.yaml from rados/mgr into a new, separate rados/dashboard
suite. The common elements it uses are moved from rados/mgr into qa/ and
replaced with symlinks.

Fixes: https://tracker.ceph.com/issues/41820
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-10-21 12:31:51 +02:00
Kefu Chai
df8bb8b8f6
Merge pull request #30646 from shyukri/wip-qa-mgr-balancer
qa/mgr/balancer: Add cram based test for altering target_max_misplaced_ratio setting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-10-18 15:43:33 +08:00
Sage Weil
71d74aa8c6 qa: more tries for mon tell when injecting msgr failures
With failure injection the default 2 tries isn't quite enough

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-11 14:16:42 -05:00
Sage Weil
cbaa6c079b qa/suites/rados/singleton*: symlink to same msgr-failures
Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-11 09:10:48 -05:00
Ricardo Dias
048f0543b2
Merge pull request #30864 from s0nea/wip-dashboard-comment-failing-suites-out
mgr/dashboard: comment failing QA suites out

Ricardo Dias <rdias@suse.com>
2019-10-11 14:27:50 +01:00
Tatjana Dehler
a747efad29 mgr/dashboard: comment questionable suites out
We're currently facing some issues with our integration
tests. Because of that we agreed on commenting questionable
suites out to be able to run all other suites on open pull
requests.

'test_health' and 'test_perf_counters' are commented out
because they led to issues in relation to
https://tracker.ceph.com/issues/41538
As soon as the issue has been fixed, we need to re-add
these two suites again.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-10-11 10:58:41 +02:00
Tatjana Dehler
9eb64d758f mgr/dashboard: sort QA suites alphabetically
Sort QA suites alphabetically to recognize a missing
suite faster.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-10-11 10:56:45 +02:00
Sage Weil
4e50520efe qa/suites/rados/thrash-old-clients: exclude ceph-daemon on nautilus installs
i did the original addition by grepping for ceph-mgr-ssh, but that's
included in nautilus so I missed this one!

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-09 13:12:26 -05:00
Sage Weil
6fd67f19e3 Merge PR #30603 into master
* refs/pull/30603/head:
	ceph-daemon: -n type.id instead of -i id
	ceph-daemon: drop unused VERSION
	ceph-daemon: clean up dir helpers, tighten up permissions
	ceph-daemon: fchmod before writing to keyring file
	test_ceph_daemon.sh: skip ssh until container image has remoto
	ceph-daemon: decode utf-8 in run() helper
	mgr/ssh: clean up debug cruft
	mgr/ssh: clean up bare except: block
	ceph-daemon: clean up bare except: blocks
	ceph-daemon: all imports to top
	ceph-volume: no_tmpfs -> tmpfs
	doc/bootstrap: add new bootstrap documentation
	ceph-daemon: add --output-pub-ssh-key for bootstrap
	ceph-daemon: make 'shell' easier to use
	ceph-daemon: support docker; prefer podman
	qa: add ceph-daemon
	debian: ceph-daemon package, required by ceph-mgr-ssh
	ceph.spec.in: ceph-daemon package, required by ceph-mgr
	common/options: cleanup whitespace
	mgr/ssh: simplify getting the cluster fsid
	mgr/ssh: pipe ceph-daemon script to stdin of python3
	ceph-daemon: add support for args and/or stdin from top of script
	ceph-daemon: make ceph-volume use get_config_and_keyring
	ceph-daemon: ls: behave if /var/log/ceph doesn't exist
	ceph-daemon: implement 'adopt' for legacy style daemons
	ceph-daemon: fix fsid detection for legacy osds
	ceph-daemon: make rm-cluster clean up system-ceph*.slice too
	ceph-daemon: configure ssh orchestrator
	ceph-daemon: be more restrictive with file permissions
	mgr/ssh: create osd with ceph-daemon
	mgr/ssh: pass daemon id separately to _create_daemon
	ceph-daemon: add --config-and-keyring to ceph-volume command
	ceph-daemon: create log path for shell (if needed)
	mgr/ssh: use _run_ceph_daemon for _create_daemon
	mgr/ssh: factor _run_ceph_daemon out of _get_device_inventory
	mon/ConfigMonitor: allow entity type only for 'config get'
	ceph-daemon: add ceph-volume subcommand
	ceph-daemon: remove unused CephContainer dname property
	ceph-daemon: drop useless uid/gid checks
	mgr/ssh: deploy new mgrs with ceph-daemon
	mgr/ssh: factor _create_daemon out of create_mon
	mon/MonCap: allow mgr to create new auth keys
	mgr/ssh: run c-v with podman when getting inventory
	mgr/ssh: simplify ssh connection management
	mgr/ssh: use ceph-daemon for deploying mon
	ceph-daemon: allow --mon-network for deploying new mon (vs specifying IP)
	ceph-daemon: --config-and-keyring (not key)
	common/options: add 'image' config option
	test_ceph_daemon: specify image name
	vstart.sh: add --ssh to enable+configure ssh orchestrator
	mgr/ssh: use ssh identity from config-key, if present
	mgr/ssh: hardcode default ssh_config
	ceph-daemon: store ssh identity in mon config-key store
	ceph-daemon: --privileged arg for 'exec'
	ceph-daemon: make deploy work for osd (do a c-v prepare)
	ceph-daemon: make shell privileged
	ceph-daemon: move get_container_mounts to a helper
	ceph-daemon: pass full path for entrypoint
	ceph-daemon: make id portion of 'shell' optional
	ceph-volume: accept --no-tmpfs argument for bluestore
	ceph-daemon: 'unit' command
	ceph-daemon: fix run command to use call(), not check_output()
	src/ceph-daemon: whitespace
	ceph-daemon: add 'enter', 'exec' commands
	ceph-daemon: bind config to default location
	test_ceph_daemon.sh: test deploy mds too
	ceph-daemon: generate ssh keys
	ceph-daemon: --config, not --conf
	ceph-daemon: long lines
	ceph-daemon: add --config to bootstrap
	ceph-daemon: add 'shell' command
	ceph-daemon: do not import subprocess symbols directly
	ceph-daemon: add mons with 'deploy mon.x ...'
	ceph-daemon: add 'ls'
	ceph-daemon: simplify uid/gid a bit
	ceph-daemon: fix libudev
	ceph-daemon: autodetect uid/gid from container image
	ceph-daemon: default to empty log files, log to stderr (systemd journal)
	ceph-daemon: rm-{daemon,cluster}
	ceph-daemon: fix bootstrap config
	ceph-daemon: fix args.fsid usage
	ceph-daemon: be careful overwriting live files
	ceph-daemon: slurp some options over from the standard systemd unit
	ceph-daemon: add ceph.target and ceph-$fsid.target units
	test_ceph_daemon.sh: stupid test script
	ceph-daemon: bootstrap and deploy (mgr) work
	ceph-daemon: initial checkin
	ceph-mon: fix debug print of public_addr
2019-10-07 15:31:14 -05:00
Sage Weil
bbc7bb5a22 Merge PR #30217 into master
* refs/pull/30217/head:
	crimson: common/admin_socket kludge so that it builds
	mon/MonClient: fix sending mon command to a specific rank
	src/.gitignore: ignore .tox
	mon/MonClient: interpret numeric mon target name as rank
	mgr,mgr/MgrClient: use fsid to signal mon-mgr vs cli MCommands
	qa/workunits/cephtool: fix errpr checks for 'ceph daemon' commands
	common/ceph_context: make 'config unset' idempotent
	qa/tasks/dump_stuck: mon.a, not mon.0
	qa/suites/rados/singleton/all/admin-socket: fix test
	common/config: EPERM setting config option after startup
	qa/workunits/cephtool/test.sh: fix tell output error check
	common/admin_socket: pass Formatter from generic infrastructure
	common/admin_socket: pass ostream to call() for error output
	os/bluestore: fix asok hook return value
	rgw: fix asok return value
	common/ceph_context: return error code from asok commands
	test/pybind/test_rados: fix accidental mon tell test
	mon: print entity_name along with caps to debug log
	PendingReleaseNotes: notes about asok changes
	mgr/MgrClient: empty target string for 'tell' means active mgr
	common/admin_socket: report error code as part of output string
	osd: change trigger_[deep_]scrub tommands to a pg tell command
	osd: remove old command workqueue, threadpool
	osd: drop MMonCommand handling
	osdc/Objecter: resend OSD tell commands on EAGAIN
	osd: route tell commands to asok; migrate commands
	osd: use unique_ptr<Formatter> for asok_command
	common/ceph_context: add generic asok 'injectargs'
	common/admin_socket: allow dup prefixes
	common/admin_socket: refactor with sync and async execute_command variants
	common/admin_socket: pass input bufferlist
	osd: transition to call_async() for asok
	common/admin_socket: support alternative call_async()
	mon/MonClient: send tell commands out of band via MCommand
	mon: accept tell commands via MCommand and send them to asok handler
	common/admin_socket: return int from hook call()
	mgr/DaemonServer: route MCommand (for octopus+) to asok commands
	do not use 'ceph tell mgr'
	pybind/ceph_argparse: disambiguate mgr tell and CLI commands
	ceph: make 'ceph tell mgr.*' send to the active mgr
	ceph: send 'ceph tell mgr.X' to the right mgr
	librados: add rados_mgr_command_target
	mgr/MgrClient: add start_command variant that takes a target
	common/admin_socket: drop unregister_command(); use per-hook variant
	common/admin_socket: drop explicit prefix arg to register_command
	common/admin_socket: simplify command routing
	common/admin_socket: add ability to process MCommand via asok queue
	common/admin_socket: pass cmdvec to execute_command
	common/admin_socket: use pipe for general wakeup
	include/compat: add flags arg to pipe_cloexec
	common/admin_socket: drop unused args

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-06 09:08:28 -05:00
Sage Weil
f2e2cb1541 qa: add ceph-daemon
Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-04 20:33:35 -05:00
David Zafman
fdf93add0b
Merge pull request #30714 from dzafman/wip-41743
test: Ignore OSD_SLOW_PING_TIME* if injecting socket failures

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-04 18:28:48 -07:00
Sage Weil
7b644f599b qa/suites/rados/singleton/all/admin-socket: fix test
We can't set the filestore setting because filestore isn't active and so
the option isn't observed, so it isn't changeable.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-04 09:07:03 -05:00
Neha Ojha
dc473ec733
Merge pull request #30699 from neha-ojha/wip-more-recovery-tests
qa/suites/rados/thrash: increase async and partial recovery test coverage

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-10-03 16:11:55 -07:00
Neha Ojha
375217a3d7 qa/suites/rados: add more-async-partial-recovery.yaml in the thrash suite
This will increase both async and partial recovery.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
Neha Ojha
e953719f2c qa/suites/rados: add more-partial-recovery.yaml in the thrash suite
Increasing osd_object_clean_region_max_num_intervals to track more
clean regions, resulting in more partial recovery.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
Neha Ojha
a2eb45507a qa/suites/rados: add more-async-recovery.yaml in the thrash suite
Reducing osd_async_recovery_min_cost to 1 should allow more
asynchronous recovery to happen.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
David Zafman
ded58ef91d test: Ignore OSD_SLOW_PING_TIME* if injecting socket failures
Fixes: https://tracker.ceph.com/issues/41743

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-10-03 09:09:10 -07:00
Sage Weil
4bfd1c5114 Merge PR #30627 into master
* refs/pull/30627/head:
	qa/suites/rados/verify: whitelist MON_DOWN when using valgrind
	qa/suits/rados/basic/tasks/rados_api_tests: pgs can go degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-02 15:46:15 -05:00
Sage Weil
644d49ec98 Merge PR #30596 into master
* refs/pull/30596/head:
	qa/suites/rados/singleton/all/ec-lost-unfound: no rbd pool

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-02 09:34:18 -05:00
Kefu Chai
a119b442f8 os: remove KineticStore
kinetic-cpp-client and kinetic project as a whole are not acitvely
supported or maintained anymore. see

- https://github.com/Kinetic/kinetic-cpp-client
- http://www.openkinetic.org

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-10-01 12:56:01 +08:00
Shyukri Shyukriev
37a45deb5b qa/mgr/balancer: Add cram based test for altering target_max_misplaced_ratio setting
Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>
2019-09-30 14:17:16 +03:00
Sage Weil
52d706c75f qa/suites/rados/verify: whitelist MON_DOWN when using valgrind
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-29 10:27:01 -05:00
Kefu Chai
e659e86096
Merge pull request #29236 from liewegas/wip-read-hole-bypg
osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-09-29 14:39:32 +08:00
Sage Weil
379bf4b423 qa/suites/rados/singleton-nomsg/osd_stale_reads.yaml
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-28 11:51:18 -05:00
Sage Weil
7871075eab qa/suits/rados/basic/tasks/rados_api_tests: pgs can go degraded
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-27 16:10:30 -05:00
Sage Weil
764dc0d2cd qa/suites/rados/singleton/all/ec-lost-unfound: no rbd pool
This can interfere with the test.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-26 09:46:10 -05:00
Sage Weil
a4eb581419 qa/suites/rados/thrash-old-clients: tolerate MON_DOWN
The first thing this test does is upgrade all mons.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-25 08:20:43 -05:00
Neha Ojha
fe93108a83 qa/suites/rados/perf: test min recommended osd_memory_target
Add bluestore-basic-min-osd-mem-target.yaml to objectstore.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-09-11 10:32:03 -07:00
Neha Ojha
da19f55de3 qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore
Removing the symlink since we do not want to run performance tests
on filestore-xfs.yaml.

Fixes: https://tracker.ceph.com/issues/41661
Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-09-10 13:05:25 -07:00
Kefu Chai
037daf5982 qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-20 21:05:21 +08:00
Josh Durgin
3f18ed55aa
Merge pull request #28227 from sseshasa/monCachePriority
mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-09 14:23:39 -07:00
Sage Weil
0d9410036a Merge PR #27542 into master
* refs/pull/27542/head:
	qa/tasks: Fix ambiguous store_thrash, thrash_store

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-08 15:52:58 -05:00
Sridhar Seshasayee
1034782d4c mon/OSDMonitor: Add standalone test for mon_memory_target
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Kefu Chai
0a8c11d172 qa/suites/rados/rest/mgr-restful: whitelist OSDMAP_FLAGS
`rest/test-restful.sh` calls `test_mgr_rest_api.py`, which in turn
calls

```
    ('patch',  '/config/osd', {'pause': True}),
```

and rest module translates it to `ceph osd set key=pause`

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-07-27 01:29:05 +08:00
Jos Collin
cf23b86fce
qa/tasks: Fix ambiguous store_thrash, thrash_store
Fixes: http://tracker.ceph.com/issues/39159
Signed-off-by: Jos Collin <jcollin@redhat.com>
2019-07-25 09:43:18 +05:30
Sage Weil
0c0093e450 Merge PR #29034 into master
* refs/pull/29034/head:
	doc/mgr/crash: document missing commands, options
	qa/suites/rados/singleton/all/test-crash: whitelist RECENT_CRASH
	qa/suites/rados/mgr/tasks/insights: whitelist RECENT_CRASH
	qa/tasks/mgr/test_insights: crash module now rejects bad crash reports
	mgr/telemetry: fix remote into crash do_ls()
	mgr/crash: don't make these methods static
	mgr/BaseMgrModule: handle unicode health detail strings
	mgr/crash: verify timestamp is valid
	qa/suites/mgr: whitelist RECENT_CRASH
	mgr/crash: remove unused var
	mgr/crash: remove unused import 'six'
	qa/workunits/rados/test_crash: health check
	mgr/crash: improve validation on post
	mgr/crash: automatically prune old crashes after a year
	mgr/crash: raise RECENT_CRASH warning for recent (new) crashes
	mgr/crash: add 'crash ls-new'
	mgr/crash: add option and serve infra
	mgr/crash: keep copy of crashes in memory
	mgr/pg_autoscaler: adjust style to match built-in tables
	mgr/crash: make 'crash ls' a nice table with a NEW column
	mgr/crash: nicely format 'crash info' output
	mgr/crash: add 'crash archive <id>', 'crash archive-all' commands

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-07-24 17:20:21 -05:00
Kefu Chai
10af76ab1b
Merge pull request #28778 from myoungwon/wip-chunk-flush-op
src/osd: add tier-flush op

Reviewed-by: Sage Weil <sage@redhat.com>
2019-07-24 09:56:47 +08:00
Kefu Chai
361b7febcc
Merge pull request #28983 from myoungwon/wip-dedup-io-test
src/test, qa/suites/rados/thrash: add dedup test

Reviewed-by: Sage Weil <sage@redhat.com>
2019-07-24 09:54:54 +08:00
Sage Weil
31b5c43806 Merge PR #29114 into master
* refs/pull/29114/head:
	mgr/dashboard: qa: whitelist client eviction warning

Reviewed-by: Laura Paduano <lpaduano@suse.com>
2019-07-23 09:09:15 -05:00
Sage Weil
468cbcd3be Merge PR #29169 into master
* refs/pull/29169/head:
	qa/suites/rados/mgr/tasks/module_selftest: whitelist mgr client getting backlisted

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2019-07-23 09:07:31 -05:00