Commit Graph

331 Commits

Author SHA1 Message Date
David Zafman
444aa9f9fe osd, mon: New pool recovery priority range -10 to 10
Use OSD_POOL_PRIORITY_MAX and OSD_POOL_PRIORITY_MIN constants
Scale legacy priorities if exceeds maximum

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-04-25 13:53:27 -07:00
Sage Weil
dc97651cbd Merge PR #27499 into master
* refs/pull/27499/head:
	qa/standalone/osd/osd-markdown: fix dup command disabling

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-04-12 06:54:58 -05:00
Sage Weil
f7216d0b2c qa/standalone/osd/osd-markdown: fix dup command disabling
The ceph cli tool checks for the presence of the variable, not its value.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-10 16:44:38 -05:00
Kefu Chai
3805935ae0
Merge pull request #26806 from xiexingguo/wip-repair-eio-rep
osd: automatically repair replicated replica on pulling error

Reviewed-by: David Zafman <dzafman@redhat.com>
2019-04-08 19:46:36 +08:00
xie xingguo
6a8aedc107 qa: add new test case for pulling error
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-04-04 11:04:43 +08:00
David Zafman
11f072fee1 Add checking of num_shards_repaired in osd stats
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-04-04 11:04:42 +08:00
Sage Weil
3c9db396ae Merge PR #27141 into master
* refs/pull/27141/head:
	mon/OSDMonitor: fix osd boot feature vs require_osd_release check
	include/ceph_features: retire 7 other old features
	include/ceph_features: retire ERASURE_CODE_PLUGINS_V2
	include/ceph_features: retire OSD_ERASURE_CODES
	include/ceph_features: update comment to align with N+2 upgrades
	include/ceph_features: adjust whitespace for retired and now usable features
	mon: remove check for jewel mons
	mds/FSMap: remove support for encoding jewel FSMap
	include/ceph_features: enable SERVER_OCTOPUS
	test/cli/osdmaptool/feature-set-unset-list: add octopus to output
	test/cli/osdmaptool/feature-set-unset-list: change unknown feature bit
	qa/releases/octopus.yaml: add octopus upgrade final step
	osd/OSDMap: octopus encoding features
	mon/OSDMonitor: add mon_debug_no_require_octopus
	mon/OSDMonitor: allow 'osd require-osd-release octopus'
	mon: add ondisk incompat octopus feature
	mon/mon_types: add mon feature for octopus
	include/ceph_features: SERVER_O -> SERVER_OCTOPUS

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-04-03 14:59:03 -05:00
Sage Weil
d667228c2e Merge PR #27146 into master
* refs/pull/27146/head:
	mon/MonMap: add min_quorum_size() helper
	mon/MDSMonitor: add 'mds ok-to-stop' command
	mon: add 'mon ok-to-{stop,add-offline,rm}' commands

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 13:49:19 -05:00
Sage Weil
3760e8f918 mon/OSDMonitor: add mon_debug_no_require_octopus
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-02 16:19:43 -05:00
Sage Weil
aa33a26e32 mon/MDSMonitor: add 'mds ok-to-stop' command
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-01 14:58:50 -05:00
Sage Weil
fbfa772047 mon/mon_types: add mon feature for octopus
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-01 11:26:33 -05:00
Sage Weil
cfba0acc01 mon: add 'mon ok-to-{stop,add-offline,rm}' commands
Helpers to decide when it is safe to stop a mon, add a mon that is
not started, or remove a mon.  (Adding and start a mon would always
be safe, but it takes time to sync, so it's not really possible to do
quickly.)

Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-01 11:05:52 -05:00
Sage Weil
420edba243 Merge PR #27169 into master
* refs/pull/27169/head:
	common/config: parse --default-$option as a default value

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-27 09:48:33 -05:00
Sage Weil
fdd2000631 common/config: parse --default-$option as a default value
Sometimes it is useful to specify an alternative default value for an
option via the command line such that it has a lower priority than the
mon config database, config file, the rest of the command line, or the
environment.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-26 11:00:27 -05:00
David Zafman
57abdb11fa osd, test: Add num_shards_repaired to osd_stat_t for pushes with repair set 3(3)
Fixes: http://tracker.ceph.com/issues/38616

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-25 16:03:36 -07:00
David Zafman
d2ca3d2feb osd: Track num_objects_repaired in pg stats 2(3)
Leave repair pg state on until recovery finishes or a new scrub starts

Fixes: http://tracker.ceph.com/issues/38616

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-25 16:03:36 -07:00
David Zafman
2202e5d0b1 test, osd: Improvements to auto_repair 1(3)
Allow auto_repair for replicated bluestore pools
Regular scrub within auto repair parameters will trigger deep scrub
New state failed_repair if PG repair attempt could not fix everything
Set failed_repair if not possible to repair anything

Fixes: http://tracker.ceph.com/issues/38616

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-23 09:52:40 -07:00
David Zafman
315d324889 test: osd-scrub-repair.sh: use corrupt_and_repair_lrc for lrc tests
Fix for argument handling of create_ec_pool()
Always pass a value for allow_overwrites for consistency

Caused by: 3ca750d41d

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-23 09:52:40 -07:00
Sage Weil
be1187575b Merge PR #27021 into master
* refs/pull/27021/head:
	msg: remove XioMessenger
	qa/suites/rados/thrash-old-clients: add nautilus
	qa/suites/rados/thrash-old-clients: add mimic v1 variant
	qa/suites/rados/thrash-old-clients: add mimic
	qa/suites/rados/thrash-old-clients: collapse msgr and client choice
	qa: remove simplemessenger tests
	ceph_test_msgr: remove simple
	msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-03-22 04:42:30 -05:00
Kefu Chai
f2b3bfa3aa
Merge pull request #26955 from liewegas/wip-slow-add
crush: various fixes for weight-sets, the osd_crush_update_weight_set option, and tests

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-22 15:42:13 +08:00
Sage Weil
28b4392a71 qa: remove simplemessenger tests
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-20 06:10:25 -05:00
Sage Weil
4c741c109d qa/standalone/crush/crush-choose-args: add weight-set tests
Verify we have the expected behavior for creates and moves that
maintain bucket summation, both with and without the
osd_crush_update_weight_set option enabled.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-20 04:57:51 -05:00
Sage Weil
f20c736e99 qa/standalone/crush/crush-choose-args: fix test
- Make the initial weight-set actually consistent (summing)
- Fix the intermediate state so that it reflects a correctly
  maintained summation.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-20 04:57:51 -05:00
Sage Weil
13d7c4f4ec Merge PR #26898 into nautilus
* refs/pull/26898/head:
	osd/PG: invalidate PG if merging with unexpected version
	osd,mon: include more pg merge metadata in pg_pool_t
	qa/standalone/osd/pg-split-merge.sh: reproduce pg merge problem with empty pgs
	osd: add osd_debug_no_{acting_change,purge_strays}

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-14 22:37:18 -05:00
Sage Weil
4bb4f7a891 Merge PR #26894 into nautilus
* refs/pull/26894/head:
	qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0
	erasure-code: ensure m >= 1
	mon/OSDMonitor: set ec min_size to k + min(1, m - 1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-13 22:07:45 -05:00
Sage Weil
52d5797c3d qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0
_DD is k=2 m=0, which we don't allow.  Switch it to cDD.

I confess I don't fully understand why this was _DD to begin with, but
I'm pretty sure mapping is there to control the order of results so that
it can be mapped to the CRUSH rule output sanely, and the coding portion
is not relevant to the test.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-13 12:46:50 -05:00
Sage Weil
fb915c4805 osd/PG: invalidate PG if merging with unexpected version
If the source or target PG version is 0'0, we may silently take the max
of the source and target and still leave the PG complete.  This
specifically can happen with an empty PG, as seen with bug 38655.  In
theory we could encounter one of the PGs with some other last_update
that doesn't match what we expect.  If that ever happens, make sure the
result is incomplete so that backfill can clean up.

Additionally check that the pool metadata for the last merge matches the
PGs at all.  This could mismatch if we have an osdmap gap and are forced
to do some merge without merge info at all... in which case we should
definitely invalidate: there should be newer copies of the PG(s), and we
have no idea whether the PGs we are merging are what we want.  If this is
some disaster recovery situation, an operator is always free to use
ceph-objectstore-tool to re-mark a PG complete (at their own peril!).

Fixes: http://tracker.ceph.com/issues/38655
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-12 10:08:46 -05:00
David Zafman
51a45e796e qa/test-erasure-code.sh: Don't grep entire bluestore directory
Bluestore caused grep crash with "grep: memory exhausted" due to
size of "block" storage.

Fixes: http://tracker.ceph.com/issues/38678

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-11 18:47:29 -07:00
David Zafman
d4915ee503 qa: Don't create rbd pool because it creates an object
This also reverts commit 10b9626ea7.

Fixes: http://tracker.ceph.com/issues/38631

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-11 16:57:51 -07:00
David Zafman
8114a2619b qa: Can't wait for clean when there aren't any pools/PGs.
Fixes: http://tracker.ceph.com/issues/38678

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-11 16:02:48 -07:00
Sage Weil
f978b27d2b qa/standalone/osd/pg-split-merge.sh: reproduce pg merge problem with empty pgs
This reproduces http://tracker.ceph.com/issues/38655

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-11 17:10:28 -05:00
Sage Weil
2ad02fbfe3 qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-09 09:34:49 -06:00
Sage Weil
10b9626ea7 qa/standalone/scrub/osd-scrub-repair: fix unfound grep
It's now "1/2 unfound":

             1/2 objects unfound (50.000%)

..presumably due to the rbd pool init creating the rbd_directory.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-08 18:23:48 -06:00
Sage Weil
30fc7f5e97 qa/standalone/ceph-helpers: fix test_wait_for_clean
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-08 18:07:10 -06:00
Sage Weil
1e2b0c7252 qa/standalone/ceph-helpers.sh: fix test_run_mon
- Only create each osd once
- forget the first osdmap dump test; it's pointless

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-08 17:43:00 -06:00
Sage Weil
bf74c1adc4 qa/standalone/osd/osd-rep-recov-eio: fix better
- no need for the default pool size
- no initial osds or it will collide with setup_osds later
- no need for rbd pool at all

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-08 17:41:11 -06:00
Sage Weil
62136d381a Merge PR #26794 into master
* refs/pull/26794/head:
	mon/MgrMonitor: only try to update always_on_modules if >= NAUTILUS
	qa/standalone/mon/msgr-v2-transition: add some tests for enabling msgr v2
	mon/MonmapMonitor: add 'ceph mon set-addrs <name> <addrvec>' command
	Revert "mon/MonClient: disable ms_bind_msgr2 if NAUTILUS feature not set"
	mon/OSDMonitor: use legacy_equals to compare osd addrs
	msg/msg_types: make legacy_equals() symmetrical
	mon/MDSMonitor: stop using get_orig_source_inst()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-07 22:12:52 -06:00
Sage Weil
c939eefa16 qa/standalone/mon/msgr-v2-transition: add some tests for enabling msgr v2
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-07 16:35:35 -06:00
Sage Weil
b59ff3860f qa/standalone/osd/osd-force-create-pg: create more pgs
Avoid warnings about too few pgs.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-06 16:27:56 -06:00
Sage Weil
cba0483b09 qa/standalone: make sure an osd is running before create_rbd_pool
'rbd pool init' now does IO.  Drop the pool, or change the pool size to 1.

Fixes: http://tracker.ceph.com/issues/38585
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-06 16:27:56 -06:00
Sage Weil
01316aa7bd qa/standalone/osd/pg-split-merge: fix import_after_merge_and_gap
This test introduces a map gap.  What *should* happen is that when there is
such a gap, we cannot import.  Previously, the test didn't reliably produce
a map gap at all, and didn't check that import failed--it verified that it
passed.

Fix the test so that it reliably produces a gap *and* reports
min_last_epoch_clean to the mon so we can trim.  Then verify we fail to
import, but can with --force.  But remove the pg again, because if we
force an import with a map gap the osd will refuse to start.

Fixes: http://tracker.ceph.com/issues/38525
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-03 10:23:27 -06:00
Sage Weil
c6a7b2cbd1 qa/standalone/osd/osd-markdown: disable CLI command dups
The markdown test is based on marking down a specific number of times, but
the duplicate commands from the CLI may not get absorbed/batched by the
mon, breaking the test.  Override the default qa/tasks/workunit.py
behavior of sending dups.

Fixes: http://tracker.ceph.com/issues/38359
Signed-off-by: Sage Weil <sage@redhat.com>
2019-02-18 15:02:25 -06:00
David Zafman
64beabc4c6 test: Limit loops waiting for force-backfill/force-recovery to happen
Fixes: http://tracker.ceph.com/issues/38309

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-13 17:44:53 -08:00
David Zafman
910a95b9c8 test: osd-backfill-stats.sh Fix check of multi backfill OSDs, skip remapped test
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-07 20:05:58 -08:00
David Zafman
690ff9a21f
Merge pull request #26213 from dzafman/wip-38041
osd: Fix recovery and backfill priority handling

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-02-07 17:26:34 -08:00
David Zafman
ca5cf14fa8 test: Add scripts to test backfill/recovery priority handling
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-07 15:46:23 -08:00
Sage Weil
dcdca44aa4 qa/standalone/ceph-helpers: fix health_ok test
Stopping the osd daemon won't reliably get you HEALTH_WARN or ERR; you have
to make sure it is also marked down.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-02-07 12:10:34 -06:00
David Zafman
36e305c4b6 test: Ignore kill_daemons() error
Workaround for: http://tracker.ceph.com/issues/38195

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-05 11:31:32 -08:00
David Zafman
bca4fe98b1 test: Fix kill_daemon() to check after last large sleep
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-05 11:30:04 -08:00
David Zafman
cc6339c0cd test: Increase timeouts in osd-backfill-space.sh because of failure seen
Fixes: http://tracker.ceph.com/issues/38027

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-02-05 11:29:32 -08:00