Commit Graph

4065 Commits

Author SHA1 Message Date
Sage Weil
5cd067b16b Merge PR #24104 into master
* refs/pull/24104/head:
	mgr/diskprediction: add prototype diskprediction module

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-18 07:08:05 -05:00
Kefu Chai
338612ad88
Merge pull request #24088 from dzafman/wip-35982
qa/standalone: Standalone test corrections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-17 22:35:43 +08:00
Rick Chen
4abb79f159 mgr/diskprediction: add prototype diskprediction module
This module is written by Rick Chen <rick.chen@prophetstor.com> and
provides both a built-in local predictor and a cloud mode that queries
a cloud service (provided by ProphetStor) to predict device failures.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-17 08:20:57 -05:00
Kefu Chai
f46523e464
Merge pull request #23955 from wjwithagen/wjw-fix-ceph-helpers.sh
test: Start using GNU awk and fix archiving directory

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-17 15:44:06 +08:00
Patrick Donnelly
ea43ce7aae
Merge PR #21992 into master
* refs/pull/21992/head:
	qa: extend timeout for SessionMap flush

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-16 14:30:28 -07:00
Mykola Golub
a68be2b452
Merge pull request #24036 from dillaman/wip-24688
rbd-mirror: optionally extract peer secrets from config-key

Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-09-16 11:53:04 +03:00
Cleber Rosa
f0f6e46084 qa/btrfs/test_rmdir_async_snap: remove binary file
On a quick look at the source code, I noticed this binary file, which
looks like was committed by mistake.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
2018-09-14 13:26:32 -04:00
David Zafman
ef6940fbb6 test: osd-backfill-stats.sh: Fix subtests to get primary which can change
Fixes: http://tracker.ceph.com/issues/35982

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-13 13:19:23 -07:00
David Zafman
6d53e2c380 test: Fix for error message changed in ceph-objectstore-tool
Fixes: http://tracker.ceph.com/issues/35982

Caused by: 6bd682f53d

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-13 13:19:11 -07:00
Mykola Golub
ad9fc49776
Merge pull request #23896 from Songweibin/wip-group-snap-rollback
rbd: add group snap rollback method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-09-13 19:35:10 +03:00
Jason Dillaman
d3a1a831ac qa/workunits/rbd: new test for rbd-mirror peer config-key secrets
Fixes: http://tracker.ceph.com/issues/24688
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-13 09:58:34 -04:00
Kefu Chai
59e6fbbe70
Merge pull request #24024 from p-na/fix-flaky-qa-tests
mgr/dashboard: Fix flaky QA tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
2018-09-13 21:52:35 +08:00
David Zafman
7f83a24553
Merge pull request #24018 from dzafman/wip-35912
qa/standalone: Minor test improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-12 13:15:44 -07:00
Patrick Donnelly
644a535ee6
qa: extend timeout for SessionMap flush
Fixes: https://tracker.ceph.com/issues/24129

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-12 12:52:00 -07:00
Sage Weil
4591f134c8 Merge PR #24025 into master
* refs/pull/24025/head:
	mon/OSDMonitor: reject pg_num -> 0

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-09-12 11:51:58 -05:00
songweibin
dec2ea83cf tools/rbd: support group snap rollback CLI
Signed-off-by: songweibin <song.weibin@zte.com.cn>
2018-09-12 23:42:15 +08:00
Kefu Chai
1578875194
Merge pull request #24013 from dzafman/wip-35845
test: Use a grep pattern that works across releases

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-12 23:00:39 +08:00
Patrick Nawracay
079cfeae4d mgr/dashboard: Cleanup
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-12 10:16:54 +02:00
Patrick Nawracay
49ab1315a3 mgr/dashboard: Fix flaky QA tests
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-12 10:16:54 +02:00
xie xingguo
6d8d20a55b mon/OSDMonitor: reject pg_num -> 0
Otherwise it will finally crash the mgr process:
```
-2> 2018-09-11 16:23:31.272 7ff521d59700 -1 /clove/xxG/ceph/src/osd/osd_types.cc: In function 'pg_t pg_t::get_parent() const'
 thread 7ff521d59700 time 2018-09-11 16:23:31.272096
/clove/xxG/ceph/src/osd/osd_types.cc: 587: FAILED ceph_assert(bits)
```

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-12 08:58:10 +08:00
John Spray
0551d0e14b
Merge pull request #23497 from noahdesu/insights
mgr/insights: insights reporting module

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-11 15:56:10 +01:00
John Spray
b18bee1b19
Merge pull request #23146 from jcsp/wip-progress
mgr/progress: improve+test OSD out handling

Reviewed-by: Noah Watkins <nwatkins@redhat.com>
2018-09-11 15:37:51 +01:00
Kefu Chai
510d9e1345
Merge pull request #23723 from xiexingguo/wip-list-missing
osd/PrimaryLogPG: rename list_missing -> list_unfound command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-09-11 20:25:21 +08:00
John Spray
46a74a07d9 qa: add task for progress module
Signed-off-by: John Spray <john.spray@redhat.com>
2018-09-11 11:21:35 +01:00
John Spray
09f5f9e1eb qa: add 4th OSD to mgr test cluster
This is useful for testing progress module.

Signed-off-by: John Spray <john.spray@redhat.com>
2018-09-11 11:21:35 +01:00
John Spray
5ecd69099d qa: add tests for progress module
Signed-off-by: John Spray <john.spray@redhat.com>
2018-09-11 11:21:35 +01:00
Noah Watkins
5c2615e3d7 qa/tasks/mgr: whitelist insights test health checks
these drive health history tracking tests.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>
2018-09-10 15:02:49 -07:00
David Zafman
050a4bb20e test: Allow directory specified to run-standalone.sh
Fixes: http://tracker.ceph.com/issues/35912

Example:
cd build
../qa/run-standalone.sh scrub

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-10 12:23:11 -07:00
David Zafman
6e3f04365f test: Trap termination so we can capture logs on teuthology timeout
Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-10 12:23:07 -07:00
vasukulkarni
10f1c4c9de
Merge pull request #23602 from smanjara/wip-test-netem
qa: Task to emulate network delay and packet drop between two given h…
2018-09-10 09:57:10 -07:00
David Zafman
dc80f8585a test: Use a grep pattern that works across releases
Fixes: http://tracker.ceph.com/issues/35845

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-10 08:21:36 -07:00
Sage Weil
d71258495e
Merge pull request #23997 from batrick/multimds-qa-broken-symlink
qa: fix symlink
2018-09-10 09:26:12 -05:00
Sage Weil
4d2a73c7f1 Merge PR #23845 into master
* refs/pull/23845/head:
	osd/OSDMap: include age in up and in counts for ceph status
	mon/OSDMonitor: set new_last_{up,in}_change
	osd/OSDMap: store last_up_change and last_in_change
	mgr/MgrMap: include mgr age in map printer
	mon/MgrMap: track active_changed timestamp
	mon: include mon quorum age in status
	include/utime: add utimespan_str helper

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-10 07:45:58 -05:00
Patrick Donnelly
a45852f8fd
qa: fix symlink
Introduced-by: 6ac1882dc4

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-08 19:21:57 -07:00
Sage Weil
f47921f293 qa/standalone/osd/osd-backfill-stats: fixes
Grep from the primary's log, not every osd's log.

For the backfill_remapped task in particular, after the pg_temp change it
just so happens that the primary changes across the pool size change and
thus two different primaries do (some) backfill.  Fix that test to pass
the correct primary.

Other tests are unaffected as they do not (happen to) trigger a primary
change and already satisfied the (removed) check that only one OSD does
backfill.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 17:11:18 -05:00
Sage Weil
09ee3f3538 Merge PR #20469 into master
* refs/pull/20469/head:
	osd/PG: remove warn on delete+merge race
	osd: base project_pg_history on is_new_interval
	osd: make project_pg_history handle concurrent osdmap publish
	osd: handle pg delete vs merge race
	osd/PG: do not purge strays in premerge state
	doc/rados/operations/placement-groups: a few minor corrections
	doc/man/8/ceph: drop enumeration of pg states
	doc/dev/placement-groups: drop old 'splitting' reference
	osd: wait for laggy pgs without osd_lock in handle_osd_map
	osd: drain peering wq in start_boot, not _committed_maps
	osd: kick split children
	osd: no osd_lock for finish_splits
	osd/osd_types: remove is_split assert
	ceph-objectstore-tool: prevent import of pg that has since merged
	qa/suites: test pg merging
	qa/tasks/thrashosds: support merging pgs too
	mon/OSDMonitor: mon_inject_pg_merge_bounce_probability
	doc/rados/operations/placement-groups: update to describe pg_num reductions too
	doc/rados/operations: remove reference to lpgs
	osd: implement pg merge
	osd/PG: implement merge_from
	osdc/Objecter: resend ops on pg merge
	osd: collect and record pg_num changes by pool
	osd: make load_pgs remove message more accurate
	osd/osd_types: pg_t: add is_merge_target()
	osd/osd_types: pg_t::is_merge -> is_merge_source
	osd/osd_types: adding or substracting invalid stats -> invalid stats
	osd/PG: clear_ready_to_merge on_shutdown (or final merge source prep)
	osd: debug pending_creates_from_osd cleanup, don't use cbegin
	ceph-objectstore-tool: debug intervals update
	mgr/ClusterState: discard pg updates for pgs >= pg_num
	mon/OSDMonitor: fix long line
	mon/OSDMonitor: move pool created check into caller
	mon/OSDMonitor: adjust pgp_num_target down along with pg_num_target as needed
	mon/OSDMonitor: add mon_osd_max_initial_pgs to cap initial pool pgs
	osd/OSDMap: set pg[p]_num_target in build_simple*() methods
	mon/PGMap: adjust SMALLER_PGP_NUM warning to use *_target values
	mon/OSDMonitor: set CREATING flag for force-create-pg
	mon/OSDMonitor: start sending new-style pg_create2 messages
	mon/OSDMonitor: set last_force_resend_prenautilus for pg_num_pending changes
	osd: ignore pg creates when pool FLAG_CREATING is not set
	mgr: do not adjust pg_num until FLAG_CREATING removed from pool
	mon/OSDMonitor: add FLAG_CREATING on upgrade if pools still creating
	mon/OSDMonitor: prevent FLAG_CREATING from getting set pre-nautilus
	mon/OSDMonitor: disallow pg_num changes while CREATING flag is set
	mon/OSDMonitor: set POOL_CREATING flag until initial pool pgs are created
	osd/osd_types: add pg_pool_t FLAG_POOL_CREATING
	osd/osd_types: introduce last_force_resend_prenautilus
	osd/PGLog: merge_from helper
	osd: no cache agent or snap trimming during premerge
	osd: notify mon when pending PGs are ready to merge
	mgr: add simple controller to adjust pg[p]_num_actual
	mon/OSDMonitor: MOSDPGReadyToMerge to complete a pg_num change
	mon/OSDMonitor: allow pg_num to adjusted up or down via pg[p]_num_target
	osd/osd_types: make pg merge an interval boundary
	osd/osd_types: add pg_t::is_merge() method
	osd/osd_types: add pg_num_pending to pg_pool_t
	osd: allow multiple threads to block on wait_min_pg_epoch
	osd: restructure advance_pg() call mechanism
	mon/PGMap: prune merged pgs
	mon/PGMap: track pgs by state for each pool
	osd/SnapMapper: allow split_bits to decrease (merge)
	os/bluestore: fix osr_drain before merge
	os/bluestore: allow reuse of osr from existing collection
	os/filestore: (re)implement merge
	os/filestore: add _merge_collections post-check
	os: implement merge_collection
	os/ObjectStore: add merge_collection operation to Transaction
2018-09-07 15:55:21 -05:00
Ilya Dryomov
478aca82eb
Merge pull request #23976 from idryomov/wip-cram-git-clone
qa/tasks/cram: tasks now must live in the repository

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2018-09-07 19:57:42 +02:00
Sage Weil
6bd682f53d ceph-objectstore-tool: prevent import of pg that has since merged
We currently import a portion of the PG if it has split.  Merge is more
complicated, though, mainly because COT is operating in a mode where it
fast-forwards the PG to the latest OSDMap epoch, which means it has to
implement any transformations to the PG (split/merge) independently.
Avoid doing this for merge.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 12:09:05 -05:00
Sage Weil
44de03d5e6 qa/suites: test pg merging
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 12:09:05 -05:00
Sage Weil
0b59b7a688 qa/tasks/thrashosds: support merging pgs too
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 12:09:05 -05:00
Sage Weil
4fc02a7f48 osd/OSDMap: include age in up and in counts for ceph status
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-07 09:07:50 -05:00
vasukulkarni
93748a325c
Merge pull request #23944 from ceph/wip-s3a-update-mirror
qa/tasks: update mirror link for maven
2018-09-06 14:44:29 -07:00
Ilya Dryomov
592f566b4e qa/tasks/cram: tasks now must live in the repository
Commit 0d8887652d ("qa/tasks/cram: use suite_repo repository for all
cram jobs") removed hardcoded git.ceph.com links, but as it turned out
it is still used for nightlies.  There is no good way to accommodate
the different URL schemes, so let's get rid of URLs altogether.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2018-09-06 22:32:39 +02:00
Ilya Dryomov
e1c89b51c8 qa/tasks/workunit: factor out overrides and refspec logic
Allow for reuse in the cram task.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2018-09-06 22:31:55 +02:00
Patrick Donnelly
532f2880bd
Merge PR #23673 into master
* refs/pull/23673/head:
	qa: automate distro/kernel matrix for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-06 10:57:32 -07:00
Patrick Donnelly
6ac1882dc4
qa: automate distro/kernel matrix for kclient
It's no longer necessary to pass `-k testing` to teuthology-suite. We're also
now regularly testing RHEL 7.5 kernel in upstream testing.

This work is prep for eventually integrating kclient into fs.

Fixes: http://tracker.ceph.com/issues/26995

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-06 10:23:59 -07:00
Willem Jan Withagen
bfe7a2afaa test: Start using GNU awk and fix archiving directory
awk uses some tests that the native FreeBSD awk does not support:
    like: BEGIN{print 0 < 90}

And TESTDIR is not set when calling ceph-helpers from smoke.sh
    So fix with keeping the archive in /tmp

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2018-09-06 15:50:20 +02:00
xie xingguo
85ba2f0a82 osd/PrimaryLogPG: s/list_missing/list_unfound/
Also:
- Do not print **offset** until specified
- Count missing objects correctly (used to be primary's local missing)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-06 09:52:20 +08:00
Vasu Kulkarni
13e100259e qa/tasks: update mirror link for maven, the original mirror no longer exists
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
2018-09-05 17:08:24 -07:00
Mykola Golub
cec53e9bd2 qa/workunits/rbd: replace usage of 'rados rmpool'
This command was dropped.

Signed-off-by: Mykola Golub <mgolub@suse.com>
2018-09-05 22:52:20 +03:00