Commit Graph

865 Commits

Author SHA1 Message Date
Rick Chen
4abb79f159 mgr/diskprediction: add prototype diskprediction module
This module is written by Rick Chen <rick.chen@prophetstor.com> and
provides both a built-in local predictor and a cloud mode that queries
a cloud service (provided by ProphetStor) to predict device failures.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-17 08:20:57 -05:00
Patrick Donnelly
ea43ce7aae
Merge PR #21992 into master
* refs/pull/21992/head:
	qa: extend timeout for SessionMap flush

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-16 14:30:28 -07:00
Kefu Chai
59e6fbbe70
Merge pull request #24024 from p-na/fix-flaky-qa-tests
mgr/dashboard: Fix flaky QA tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
2018-09-13 21:52:35 +08:00
Patrick Donnelly
644a535ee6
qa: extend timeout for SessionMap flush
Fixes: https://tracker.ceph.com/issues/24129

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-12 12:52:00 -07:00
Patrick Nawracay
079cfeae4d mgr/dashboard: Cleanup
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-12 10:16:54 +02:00
Patrick Nawracay
49ab1315a3 mgr/dashboard: Fix flaky QA tests
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-12 10:16:54 +02:00
John Spray
0551d0e14b
Merge pull request #23497 from noahdesu/insights
mgr/insights: insights reporting module

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-11 15:56:10 +01:00
John Spray
b18bee1b19
Merge pull request #23146 from jcsp/wip-progress
mgr/progress: improve+test OSD out handling

Reviewed-by: Noah Watkins <nwatkins@redhat.com>
2018-09-11 15:37:51 +01:00
Kefu Chai
510d9e1345
Merge pull request #23723 from xiexingguo/wip-list-missing
osd/PrimaryLogPG: rename list_missing -> list_unfound command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-09-11 20:25:21 +08:00
John Spray
5ecd69099d qa: add tests for progress module
Signed-off-by: John Spray <john.spray@redhat.com>
2018-09-11 11:21:35 +01:00
vasukulkarni
10f1c4c9de
Merge pull request #23602 from smanjara/wip-test-netem
qa: Task to emulate network delay and packet drop between two given h…
2018-09-10 09:57:10 -07:00
Sage Weil
09ee3f3538 Merge PR #20469 into master
* refs/pull/20469/head:
	osd/PG: remove warn on delete+merge race
	osd: base project_pg_history on is_new_interval
	osd: make project_pg_history handle concurrent osdmap publish
	osd: handle pg delete vs merge race
	osd/PG: do not purge strays in premerge state
	doc/rados/operations/placement-groups: a few minor corrections
	doc/man/8/ceph: drop enumeration of pg states
	doc/dev/placement-groups: drop old 'splitting' reference
	osd: wait for laggy pgs without osd_lock in handle_osd_map
	osd: drain peering wq in start_boot, not _committed_maps
	osd: kick split children
	osd: no osd_lock for finish_splits
	osd/osd_types: remove is_split assert
	ceph-objectstore-tool: prevent import of pg that has since merged
	qa/suites: test pg merging
	qa/tasks/thrashosds: support merging pgs too
	mon/OSDMonitor: mon_inject_pg_merge_bounce_probability
	doc/rados/operations/placement-groups: update to describe pg_num reductions too
	doc/rados/operations: remove reference to lpgs
	osd: implement pg merge
	osd/PG: implement merge_from
	osdc/Objecter: resend ops on pg merge
	osd: collect and record pg_num changes by pool
	osd: make load_pgs remove message more accurate
	osd/osd_types: pg_t: add is_merge_target()
	osd/osd_types: pg_t::is_merge -> is_merge_source
	osd/osd_types: adding or substracting invalid stats -> invalid stats
	osd/PG: clear_ready_to_merge on_shutdown (or final merge source prep)
	osd: debug pending_creates_from_osd cleanup, don't use cbegin
	ceph-objectstore-tool: debug intervals update
	mgr/ClusterState: discard pg updates for pgs >= pg_num
	mon/OSDMonitor: fix long line
	mon/OSDMonitor: move pool created check into caller
	mon/OSDMonitor: adjust pgp_num_target down along with pg_num_target as needed
	mon/OSDMonitor: add mon_osd_max_initial_pgs to cap initial pool pgs
	osd/OSDMap: set pg[p]_num_target in build_simple*() methods
	mon/PGMap: adjust SMALLER_PGP_NUM warning to use *_target values
	mon/OSDMonitor: set CREATING flag for force-create-pg
	mon/OSDMonitor: start sending new-style pg_create2 messages
	mon/OSDMonitor: set last_force_resend_prenautilus for pg_num_pending changes
	osd: ignore pg creates when pool FLAG_CREATING is not set
	mgr: do not adjust pg_num until FLAG_CREATING removed from pool
	mon/OSDMonitor: add FLAG_CREATING on upgrade if pools still creating
	mon/OSDMonitor: prevent FLAG_CREATING from getting set pre-nautilus
	mon/OSDMonitor: disallow pg_num changes while CREATING flag is set
	mon/OSDMonitor: set POOL_CREATING flag until initial pool pgs are created
	osd/osd_types: add pg_pool_t FLAG_POOL_CREATING
	osd/osd_types: introduce last_force_resend_prenautilus
	osd/PGLog: merge_from helper
	osd: no cache agent or snap trimming during premerge
	osd: notify mon when pending PGs are ready to merge
	mgr: add simple controller to adjust pg[p]_num_actual
	mon/OSDMonitor: MOSDPGReadyToMerge to complete a pg_num change
	mon/OSDMonitor: allow pg_num to adjusted up or down via pg[p]_num_target
	osd/osd_types: make pg merge an interval boundary
	osd/osd_types: add pg_t::is_merge() method
	osd/osd_types: add pg_num_pending to pg_pool_t
	osd: allow multiple threads to block on wait_min_pg_epoch
	osd: restructure advance_pg() call mechanism
	mon/PGMap: prune merged pgs
	mon/PGMap: track pgs by state for each pool
	osd/SnapMapper: allow split_bits to decrease (merge)
	os/bluestore: fix osr_drain before merge
	os/bluestore: allow reuse of osr from existing collection
	os/filestore: (re)implement merge
	os/filestore: add _merge_collections post-check
	os: implement merge_collection
	os/ObjectStore: add merge_collection operation to Transaction
2018-09-07 15:55:21 -05:00
Ilya Dryomov
478aca82eb
Merge pull request #23976 from idryomov/wip-cram-git-clone
qa/tasks/cram: tasks now must live in the repository

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2018-09-07 19:57:42 +02:00
Sage Weil
6bd682f53d ceph-objectstore-tool: prevent import of pg that has since merged
We currently import a portion of the PG if it has split.  Merge is more
complicated, though, mainly because COT is operating in a mode where it
fast-forwards the PG to the latest OSDMap epoch, which means it has to
implement any transformations to the PG (split/merge) independently.
Avoid doing this for merge.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 12:09:05 -05:00
Sage Weil
0b59b7a688 qa/tasks/thrashosds: support merging pgs too
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 12:09:05 -05:00
Ilya Dryomov
592f566b4e qa/tasks/cram: tasks now must live in the repository
Commit 0d8887652d ("qa/tasks/cram: use suite_repo repository for all
cram jobs") removed hardcoded git.ceph.com links, but as it turned out
it is still used for nightlies.  There is no good way to accommodate
the different URL schemes, so let's get rid of URLs altogether.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2018-09-06 22:32:39 +02:00
Ilya Dryomov
e1c89b51c8 qa/tasks/workunit: factor out overrides and refspec logic
Allow for reuse in the cram task.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2018-09-06 22:31:55 +02:00
xie xingguo
85ba2f0a82 osd/PrimaryLogPG: s/list_missing/list_unfound/
Also:
- Do not print **offset** until specified
- Count missing objects correctly (used to be primary's local missing)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-06 09:52:20 +08:00
Vasu Kulkarni
13e100259e qa/tasks: update mirror link for maven, the original mirror no longer exists
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
2018-09-05 17:08:24 -07:00
Shilpa Jagannath
14da3f07f3 Adds an identifier to the greenlet spawned by task 'link_toggle', which can then looked up by task 'link_recover' to call end() on it. Also fixes task cleanup during unwind.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
2018-09-05 16:02:23 +05:30
Shilpa Jagannath
6da212f7cf Replaced hardcoded net interface names with yaml configurables.
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
2018-09-05 16:02:13 +05:30
Shilpa Jagannath
e7d0cdacc1 qa: Task to emulate network delay and packet drop between two given hosts
The task uses netem to emulate wide area network delay.
Provides three different configurable options.
1. standard delay: Constant delay with +/- 5ms jitter with normal distribution as default.
2. variable delay: To provide a delay between two given min-max range in milliseconds.
3. packet drop: Toggles packet drop and recovery in regular interval.

Useful in simulating network delays between two clusters while testing
rgw multisite and rbd mirroring configurations.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
2018-09-05 16:01:11 +05:30
Ilya Dryomov
a0df578139
Merge pull request #23905 from idryomov/wip-cram-suite-repo
qa/tasks/cram: use suite_repo repository for all cram jobs

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-09-04 14:27:28 +02:00
Lenz Grimmer
82412896ff
Merge pull request #23491 from p-na/per-osd-settings
mgr/dashboard: Add support for managing individual OSD settings in the backend

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-09-04 12:01:19 +02:00
Ilya Dryomov
0d8887652d qa/tasks/cram: use suite_repo repository for all cram jobs
Currently git.ceph.com is hardcoded for all cram jobs.  Testing
modifications is a pain: one needs to push to either ceph/ceph.git or
ceph/ceph-ci.git (depending on where the ceph branch is at, triggering
unnecessary builds in the latter case) and wait for the mirror to sync.
Runs scheduled against branches in developer's forks fail.

Move away from git.ceph.com to allow mixing branches and repositories,
similar to workunits.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2018-09-03 22:07:20 +02:00
Lenz Grimmer
65781a4fa2
Merge pull request #23322 from ricardoasmarques/wip-role-management-api
mgr/dashboard: Add REST API for role management

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2018-09-03 15:43:02 +02:00
Ricardo Marques
592e4f7c10 mgr/dashboard: Add REST API for role management
Fixes: https://tracker.ceph.com/issues/25138

Signed-off-by: Ricardo Marques <rimarques@suse.com>
2018-09-03 13:15:42 +01:00
Patrick Nawracay
e71466cc49 mgr/dashboard: Add support for managing individual OSD settings (backend)
Add options to mark OSDs in/out/down/reweight/lost/remove/destroy/create

Fixes: http://tracker.ceph.com/issues/24270

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-03 12:51:04 +02:00
Sage Weil
2c26fb0fe1 rados: drop mkpool, rmpool commands
- mkpool and rmpool users should use the normal cli/mon commands

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-31 09:27:36 -05:00
Sage Weil
d6def8ba11 ceph-authtool: remove auid options
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-31 09:26:19 -05:00
Noah Watkins
ea15b625f3 qa/mgr/selftest: handle always-on module fall out
need a non-always-on module. hello doesn't work because it isn't
installed. so switch to selftest.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-08-28 13:45:58 -07:00
Patrick Donnelly
3aa392ca73
Merge PR #23439 into master
* refs/pull/23439/head:
	qa: whitelist cap revoke warning
	doc: document cap revoke non-responders client eviction
	test: validate client eviction for cap revoke non-responders
	mds: add counter for tracking cap non-responding clients
	mds: evict clients that do not respond to cap revoke by MDS
	mds: pass timeout argument for fetching late clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-08-25 13:04:58 -07:00
David Zafman
687f63e599 test: Update tests for error message changes
Signed-off-by: David Zafman <dzafman@redhat.com>
2018-08-23 11:09:22 -07:00
David Zafman
b40784290f qa: Add new message to whitelist for scrub/repair tests
Signed-off-by: David Zafman <dzafman@redhat.com>
2018-08-23 11:09:22 -07:00
Noah Watkins
cdfc2e70df qa/tasks/mgr: do not enable always-on module
Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-08-21 16:25:05 -07:00
Kefu Chai
f7d452e8d1
Merge pull request #23558 from liewegas/wip-mgr-always-on
mgr: keep status, balancer always on
2018-08-22 00:18:30 +08:00
Venky Shankar
c0b1dacc9f test: validate client eviction for cap revoke non-responders
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2018-08-21 01:58:55 -04:00
Sage Weil
00223d2364 qa/tasks/mgr/test_module_selftest: use hello instead of status for disbled command test
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-18 09:29:00 -05:00
Noah Watkins
3d96b2faaa qa/tasks/mgr: add tests for insights module
Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-08-16 15:29:46 -07:00
Patrick Donnelly
0b5df6e00a
qa: add cephfs-shell skeleton test case
Right now just tests that "help" works.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-16 09:13:36 -07:00
Casey Bodley
b2fe116d97
Merge pull request #23402 from cbodley/wip-23659
qa/rgw: patch keystone requirements.txt

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2018-08-14 16:53:30 -04:00
Patrick Nawracay
7340e70d7f mgr/dashboard/backend: API to manage dashboard settings
Enables to change (set/unset) values of settings of the dashboard using
the REST API.

Fixes: https://tracker.ceph.com/issues/24273

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-08-10 11:32:25 +02:00
Casey Bodley
338b97a8b9 qa/rgw: patch keystone requirements.txt
an ugly workaround for a python dependency conflict that's broken the
rgw/tempest suite. allows us to preserve the pinned versions of
keystone/tempest without having to maintain a fork of the keystone
repository

Fixes: http://tracker.ceph.com/issues/23659

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-08-02 16:43:42 -04:00
Patrick Donnelly
2ea86fa4d3
Merge PR #21948 into master
* refs/pull/21948/head:
	ceph_volume_client: use integer division for pg_num
	packages.yaml: install py3 compatible cephfs and rados bindings...
	qa: test py2 and py3 for volume client
	qa: make test_volume_client.py py3 compatible
	qa/ceph-volume: allow executing python pyloads using python3
	cephfs.pyx: add py3 compatibility
	ceph-volume-client: add py3 compatibility

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-01 21:42:27 -07:00
Casey Bodley
ce45875d80
Merge pull request #23145 from cbodley/wip-qa-rgw-admin-rest
qa/rgw: add radosgw-admin-rest task to singleton suite

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2018-08-01 13:52:40 -04:00
Ricardo Dias
47a50eeba5
Merge pull request #21881 from sebastian-philipp/dashboard-pool-patch
mgr/dashboard: Add Pool update endpoint 

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2018-08-01 10:00:10 +01:00
Patrick Donnelly
a5e3b0512c
qa: test py2 and py3 for volume client
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-07-30 16:12:48 -07:00
Rishabh Dave
f28274dc70
qa: make test_volume_client.py py3 compatible
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2018-07-30 16:12:48 -07:00
Rishabh Dave
2f35ade05f
qa/ceph-volume: allow executing python pyloads using python3
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2018-07-30 16:12:48 -07:00
Casey Bodley
3a55e13c44
Merge pull request #23143 from joke-lee/qa-admin-api-user-info-through-access-key
qa: add test for https://github.com/ceph/ceph/pull/22790

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-07-27 15:27:32 -04:00