Commit Graph

1235 Commits

Author SHA1 Message Date
Ricardo Dias
048f0543b2
Merge pull request #30864 from s0nea/wip-dashboard-comment-failing-suites-out
mgr/dashboard: comment failing QA suites out

Ricardo Dias <rdias@suse.com>
2019-10-11 14:27:50 +01:00
Tatjana Dehler
a747efad29 mgr/dashboard: comment questionable suites out
We're currently facing some issues with our integration
tests. Because of that we agreed on commenting questionable
suites out to be able to run all other suites on open pull
requests.

'test_health' and 'test_perf_counters' are commented out
because they led to issues in relation to
https://tracker.ceph.com/issues/41538
As soon as the issue has been fixed, we need to re-add
these two suites again.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-10-11 10:58:41 +02:00
Tatjana Dehler
9eb64d758f mgr/dashboard: sort QA suites alphabetically
Sort QA suites alphabetically to recognize a missing
suite faster.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-10-11 10:56:45 +02:00
Sage Weil
4e50520efe qa/suites/rados/thrash-old-clients: exclude ceph-daemon on nautilus installs
i did the original addition by grepping for ceph-mgr-ssh, but that's
included in nautilus so I missed this one!

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-09 13:12:26 -05:00
Sage Weil
6fd67f19e3 Merge PR #30603 into master
* refs/pull/30603/head:
	ceph-daemon: -n type.id instead of -i id
	ceph-daemon: drop unused VERSION
	ceph-daemon: clean up dir helpers, tighten up permissions
	ceph-daemon: fchmod before writing to keyring file
	test_ceph_daemon.sh: skip ssh until container image has remoto
	ceph-daemon: decode utf-8 in run() helper
	mgr/ssh: clean up debug cruft
	mgr/ssh: clean up bare except: block
	ceph-daemon: clean up bare except: blocks
	ceph-daemon: all imports to top
	ceph-volume: no_tmpfs -> tmpfs
	doc/bootstrap: add new bootstrap documentation
	ceph-daemon: add --output-pub-ssh-key for bootstrap
	ceph-daemon: make 'shell' easier to use
	ceph-daemon: support docker; prefer podman
	qa: add ceph-daemon
	debian: ceph-daemon package, required by ceph-mgr-ssh
	ceph.spec.in: ceph-daemon package, required by ceph-mgr
	common/options: cleanup whitespace
	mgr/ssh: simplify getting the cluster fsid
	mgr/ssh: pipe ceph-daemon script to stdin of python3
	ceph-daemon: add support for args and/or stdin from top of script
	ceph-daemon: make ceph-volume use get_config_and_keyring
	ceph-daemon: ls: behave if /var/log/ceph doesn't exist
	ceph-daemon: implement 'adopt' for legacy style daemons
	ceph-daemon: fix fsid detection for legacy osds
	ceph-daemon: make rm-cluster clean up system-ceph*.slice too
	ceph-daemon: configure ssh orchestrator
	ceph-daemon: be more restrictive with file permissions
	mgr/ssh: create osd with ceph-daemon
	mgr/ssh: pass daemon id separately to _create_daemon
	ceph-daemon: add --config-and-keyring to ceph-volume command
	ceph-daemon: create log path for shell (if needed)
	mgr/ssh: use _run_ceph_daemon for _create_daemon
	mgr/ssh: factor _run_ceph_daemon out of _get_device_inventory
	mon/ConfigMonitor: allow entity type only for 'config get'
	ceph-daemon: add ceph-volume subcommand
	ceph-daemon: remove unused CephContainer dname property
	ceph-daemon: drop useless uid/gid checks
	mgr/ssh: deploy new mgrs with ceph-daemon
	mgr/ssh: factor _create_daemon out of create_mon
	mon/MonCap: allow mgr to create new auth keys
	mgr/ssh: run c-v with podman when getting inventory
	mgr/ssh: simplify ssh connection management
	mgr/ssh: use ceph-daemon for deploying mon
	ceph-daemon: allow --mon-network for deploying new mon (vs specifying IP)
	ceph-daemon: --config-and-keyring (not key)
	common/options: add 'image' config option
	test_ceph_daemon: specify image name
	vstart.sh: add --ssh to enable+configure ssh orchestrator
	mgr/ssh: use ssh identity from config-key, if present
	mgr/ssh: hardcode default ssh_config
	ceph-daemon: store ssh identity in mon config-key store
	ceph-daemon: --privileged arg for 'exec'
	ceph-daemon: make deploy work for osd (do a c-v prepare)
	ceph-daemon: make shell privileged
	ceph-daemon: move get_container_mounts to a helper
	ceph-daemon: pass full path for entrypoint
	ceph-daemon: make id portion of 'shell' optional
	ceph-volume: accept --no-tmpfs argument for bluestore
	ceph-daemon: 'unit' command
	ceph-daemon: fix run command to use call(), not check_output()
	src/ceph-daemon: whitespace
	ceph-daemon: add 'enter', 'exec' commands
	ceph-daemon: bind config to default location
	test_ceph_daemon.sh: test deploy mds too
	ceph-daemon: generate ssh keys
	ceph-daemon: --config, not --conf
	ceph-daemon: long lines
	ceph-daemon: add --config to bootstrap
	ceph-daemon: add 'shell' command
	ceph-daemon: do not import subprocess symbols directly
	ceph-daemon: add mons with 'deploy mon.x ...'
	ceph-daemon: add 'ls'
	ceph-daemon: simplify uid/gid a bit
	ceph-daemon: fix libudev
	ceph-daemon: autodetect uid/gid from container image
	ceph-daemon: default to empty log files, log to stderr (systemd journal)
	ceph-daemon: rm-{daemon,cluster}
	ceph-daemon: fix bootstrap config
	ceph-daemon: fix args.fsid usage
	ceph-daemon: be careful overwriting live files
	ceph-daemon: slurp some options over from the standard systemd unit
	ceph-daemon: add ceph.target and ceph-$fsid.target units
	test_ceph_daemon.sh: stupid test script
	ceph-daemon: bootstrap and deploy (mgr) work
	ceph-daemon: initial checkin
	ceph-mon: fix debug print of public_addr
2019-10-07 15:31:14 -05:00
Ali Maredia
d4872ce97a
Merge pull request #29783 from scarvalhojr/ssevault
rgw: add SSE-KMS with Vault using token auth
2019-10-06 22:33:36 -04:00
Sage Weil
bbc7bb5a22 Merge PR #30217 into master
* refs/pull/30217/head:
	crimson: common/admin_socket kludge so that it builds
	mon/MonClient: fix sending mon command to a specific rank
	src/.gitignore: ignore .tox
	mon/MonClient: interpret numeric mon target name as rank
	mgr,mgr/MgrClient: use fsid to signal mon-mgr vs cli MCommands
	qa/workunits/cephtool: fix errpr checks for 'ceph daemon' commands
	common/ceph_context: make 'config unset' idempotent
	qa/tasks/dump_stuck: mon.a, not mon.0
	qa/suites/rados/singleton/all/admin-socket: fix test
	common/config: EPERM setting config option after startup
	qa/workunits/cephtool/test.sh: fix tell output error check
	common/admin_socket: pass Formatter from generic infrastructure
	common/admin_socket: pass ostream to call() for error output
	os/bluestore: fix asok hook return value
	rgw: fix asok return value
	common/ceph_context: return error code from asok commands
	test/pybind/test_rados: fix accidental mon tell test
	mon: print entity_name along with caps to debug log
	PendingReleaseNotes: notes about asok changes
	mgr/MgrClient: empty target string for 'tell' means active mgr
	common/admin_socket: report error code as part of output string
	osd: change trigger_[deep_]scrub tommands to a pg tell command
	osd: remove old command workqueue, threadpool
	osd: drop MMonCommand handling
	osdc/Objecter: resend OSD tell commands on EAGAIN
	osd: route tell commands to asok; migrate commands
	osd: use unique_ptr<Formatter> for asok_command
	common/ceph_context: add generic asok 'injectargs'
	common/admin_socket: allow dup prefixes
	common/admin_socket: refactor with sync and async execute_command variants
	common/admin_socket: pass input bufferlist
	osd: transition to call_async() for asok
	common/admin_socket: support alternative call_async()
	mon/MonClient: send tell commands out of band via MCommand
	mon: accept tell commands via MCommand and send them to asok handler
	common/admin_socket: return int from hook call()
	mgr/DaemonServer: route MCommand (for octopus+) to asok commands
	do not use 'ceph tell mgr'
	pybind/ceph_argparse: disambiguate mgr tell and CLI commands
	ceph: make 'ceph tell mgr.*' send to the active mgr
	ceph: send 'ceph tell mgr.X' to the right mgr
	librados: add rados_mgr_command_target
	mgr/MgrClient: add start_command variant that takes a target
	common/admin_socket: drop unregister_command(); use per-hook variant
	common/admin_socket: drop explicit prefix arg to register_command
	common/admin_socket: simplify command routing
	common/admin_socket: add ability to process MCommand via asok queue
	common/admin_socket: pass cmdvec to execute_command
	common/admin_socket: use pipe for general wakeup
	include/compat: add flags arg to pipe_cloexec
	common/admin_socket: drop unused args

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-06 09:08:28 -05:00
Sage Weil
f2e2cb1541 qa: add ceph-daemon
Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-04 20:33:35 -05:00
David Zafman
fdf93add0b
Merge pull request #30714 from dzafman/wip-41743
test: Ignore OSD_SLOW_PING_TIME* if injecting socket failures

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-04 18:28:48 -07:00
Sage Weil
7b644f599b qa/suites/rados/singleton/all/admin-socket: fix test
We can't set the filestore setting because filestore isn't active and so
the option isn't observed, so it isn't changeable.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-04 09:07:03 -05:00
Neha Ojha
dc473ec733
Merge pull request #30699 from neha-ojha/wip-more-recovery-tests
qa/suites/rados/thrash: increase async and partial recovery test coverage

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-10-03 16:11:55 -07:00
Neha Ojha
375217a3d7 qa/suites/rados: add more-async-partial-recovery.yaml in the thrash suite
This will increase both async and partial recovery.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
Neha Ojha
e953719f2c qa/suites/rados: add more-partial-recovery.yaml in the thrash suite
Increasing osd_object_clean_region_max_num_intervals to track more
clean regions, resulting in more partial recovery.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
Neha Ojha
a2eb45507a qa/suites/rados: add more-async-recovery.yaml in the thrash suite
Reducing osd_async_recovery_min_cost to 1 should allow more
asynchronous recovery to happen.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-10-03 11:15:51 -07:00
David Zafman
ded58ef91d test: Ignore OSD_SLOW_PING_TIME* if injecting socket failures
Fixes: https://tracker.ceph.com/issues/41743

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-10-03 09:09:10 -07:00
Sage Weil
4bfd1c5114 Merge PR #30627 into master
* refs/pull/30627/head:
	qa/suites/rados/verify: whitelist MON_DOWN when using valgrind
	qa/suits/rados/basic/tasks/rados_api_tests: pgs can go degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-02 15:46:15 -05:00
Andrea Baglioni
955d3b517a rgw: add SSE-KMS with Vault using token auth
Restructure SSE-KMS tests which now has 3 scenarios for each KMS
backend: Barbican, Vault, and testing (keys stored in ceph.conf).

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
2019-10-02 16:38:01 +01:00
Sage Weil
644d49ec98 Merge PR #30596 into master
* refs/pull/30596/head:
	qa/suites/rados/singleton/all/ec-lost-unfound: no rbd pool

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-02 09:34:18 -05:00
Kefu Chai
a119b442f8 os: remove KineticStore
kinetic-cpp-client and kinetic project as a whole are not acitvely
supported or maintained anymore. see

- https://github.com/Kinetic/kinetic-cpp-client
- http://www.openkinetic.org

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-10-01 12:56:01 +08:00
Sage Weil
52d706c75f qa/suites/rados/verify: whitelist MON_DOWN when using valgrind
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-29 10:27:01 -05:00
Kefu Chai
e659e86096
Merge pull request #29236 from liewegas/wip-read-hole-bypg
osd: implement per-pg leases to avoid stale reads

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-09-29 14:39:32 +08:00
Sage Weil
379bf4b423 qa/suites/rados/singleton-nomsg/osd_stale_reads.yaml
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-28 11:51:18 -05:00
Sage Weil
7871075eab qa/suits/rados/basic/tasks/rados_api_tests: pgs can go degraded
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-27 16:10:30 -05:00
Sage Weil
764dc0d2cd qa/suites/rados/singleton/all/ec-lost-unfound: no rbd pool
This can interfere with the test.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-26 09:46:10 -05:00
Sage Weil
a4eb581419 qa/suites/rados/thrash-old-clients: tolerate MON_DOWN
The first thing this test does is upgrade all mons.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-25 08:20:43 -05:00
Patrick Donnelly
30909f5a6a
Merge PR #29824 into master
* refs/pull/29824/head:
	qa: whitelist new FS_INLINE_DATA_DEPRECATED health warning
	mds: add a HEALTH_WARN message when inline_data is enabled
	mds: log a warning message when mds is started on an fs with inline_data
	mon: deprecate CephFS inline_data support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>
2019-09-24 04:32:28 -07:00
Jason Dillaman
85f98c01c0 qa/suites/rbd: test case for rbd-mirror bootstrap
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-20 08:09:21 -04:00
Jeff Layton
9c406d0ab3 mon: deprecate CephFS inline_data support
The plan is to start deprecating this feature now so that we can remove
it in a future release. Change it to require the
--yes-i-really-really-mean-it flag, and to emit a custom
warning when that isn't specified.

For now, we leave the testing in place since we do want to be notified
if something breaks before we're ready to rip it out completely.

Fixes: https://tracker.ceph.com/issues/41311
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-09-19 09:15:13 -04:00
Casey Bodley
97f1be5d3a qa/rgw: rgw/website suite uses subdomain calling format
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-17 08:24:52 -04:00
Casey Bodley
bb56ac01c2 qa/rgw: add new rgw/website suite for static website tests
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-17 08:24:51 -04:00
Casey Bodley
8c801f073d
Merge pull request #30218 from cbodley/wip-qa-rgw-barbican
qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2019-09-17 08:17:30 -04:00
J. Eric Ivancich
e487604ad2
Merge pull request #29549 from smanjara/wip-multisite-metadata
rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-09-15 12:54:24 -04:00
Ilya Dryomov
d63570c5a5
Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap
qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-09-13 22:35:57 +02:00
Casey Bodley
71a87afb5d qa/barbican: pin barbican to 5.0.1 release
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Casey Bodley
53cd9b8dc6 qa/rgw: crypt suite uses clusters/fixed-1.yaml
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Casey Bodley
1c5a12fd9e qa/rgw: abstract out barbican under 2-kms/
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Adam Kupczyk
a5ec971402 tasks/barbican: Added task to deploy and configure barbican
Fixed keystone, rgw and s3tests tasks to work with it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-09-13 15:04:43 -04:00
Ilya Dryomov
81becbdc68 qa: add script to test how libceph handles huge osdmaps
That code will also handle moderately-sized osdmaps when the memory is
fragmented.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-09-13 19:21:54 +02:00
Neha Ojha
fe93108a83 qa/suites/rados/perf: test min recommended osd_memory_target
Add bluestore-basic-min-osd-mem-target.yaml to objectstore.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-09-11 10:32:03 -07:00
Neha Ojha
da19f55de3 qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore
Removing the symlink since we do not want to run performance tests
on filestore-xfs.yaml.

Fixes: https://tracker.ceph.com/issues/41661
Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-09-10 13:05:25 -07:00
Brad Hubbard
bedd022fe2
Merge pull request #29916 from badone/wip-tracker-41518-grafana-server
qa/ceph-ansible: Disable dashboard

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-08-29 09:52:33 +10:00
Brad Hubbard
131f5a145a qa/ceph-ansible: Disable dashboard
Recently a dashboard playbook was added which will fail without a
grafana server so disabling the dashboard

Fixes: https://tracker.ceph.com/issues/41518

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2019-08-27 13:51:37 +10:00
Patrick Donnelly
e4ad5973fe
qa: ignore RECENT_CRASH for multimds snapshot testing
These tests purposefully crash the MDS.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-26 15:23:51 -07:00
Patrick Donnelly
aba296aab8
qa: add debugging failed osd-release setting
See-also: https://tracker.ceph.com/issues/40773
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-22 07:19:51 -07:00
Shilpa Jagannath
280986f8e6 Added radosgw_admin_rest task to multisite yaml.
svc_zone related modifications

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
2019-08-22 13:44:48 +05:30
Kefu Chai
037daf5982 qa/suites/rados: whitelist POOL_APP_NOT_ENABLED warning
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-20 21:05:21 +08:00
Casey Bodley
f0575a7144
Merge pull request #26787 from soumyakoduri/bucket_name_validation
[rgw]:Validate bucket names as per revised s3 spec

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-08-16 10:53:54 -04:00
Patrick Donnelly
d1ce58257e
Merge PR #29431 into master
* refs/pull/29431/head:
	qa: fix malformed suite config

Reviewed-by: Zheng Yan <zyan@redhat.com>
2019-08-14 15:21:51 -07:00
Josh Durgin
3f18ed55aa
Merge pull request #28227 from sseshasa/monCachePriority
mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-09 14:23:39 -07:00
Sage Weil
f011c13547 Merge PR #29292 into master
* refs/pull/29292/head:
	os/bluestore: warn on no per-pool omap
	os/bluestore: fsck: warning (not error) by default on no per-pool omap
	os/bluestore: fsck: int64_t for error count
	os/bluestore: default size of 1 TB for testing
	os/bluestore: behave if we *do* set PGMETA and PERPOOL flags
	os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP
	os/bluestore: fsck: only generate 1 error per omap_head
	os/bluestore: make fsck repair convert to per-pool omap
	os/bluestore: teach fsck to tolerate per-pool omap
	os/bluestore: ondisk format change to 3 for per-pool omap
	mon/PGMap: add data/omap breakouts for 'df detail' view
	osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
	mon/PGMap: fix stored_raw calculation
	mon/PGMap: add in actual omap usage into per-pool stats
	osd: report per-pool omap support via store_statfs_t
	os/bluestore: set per_pool_omap key on mkfs
	osd/osd_types: count per-pool omap capable OSDs
	os/bluestore: report omap_allocated per-pool
	os/bluestore: add pool prefix to omap keys
	kv/KeyValueDB: take key_prefix for estimate_prefix_size()
	os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
	os/bluestore: make omap key helpers Onode methods
	os/bluestore: add Onode::get_omap_prefix() helper
	os/bluestore: change _do_omap_clear() args

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-09 10:40:45 -05:00