Commit Graph

2049 Commits

Author SHA1 Message Date
David Zafman
39cc14bdc1
Merge pull request #27503 from dzafman/wip-39099
osd: Give recovery for inactive PGs a higher priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-04-25 15:06:56 -07:00
David Zafman
444aa9f9fe osd, mon: New pool recovery priority range -10 to 10
Use OSD_POOL_PRIORITY_MAX and OSD_POOL_PRIORITY_MIN constants
Scale legacy priorities if exceeds maximum

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-04-25 13:53:27 -07:00
Sage Weil
d64d14a577 Merge PR #26730 into master
* refs/pull/26730/head:
	src/test: remove unused objects to calculate accurate dedup ratio.
	src/tools: fix counting total objects
	src/tools: fix printing wrong offset.
	src/tools: fix dedup tool's name
	qa/workunit: add rabin chunk test
	src/tool: add rabin fingerprint option to cephdeduptool
	src/common: add rabin fingerprint class
	Initail work for rabin fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>
2019-04-24 20:38:02 -05:00
Mykola Golub
22973d9d46
Merge pull request #27683 from dillaman/wip-24668
qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>
2019-04-24 15:40:43 +03:00
myoungwon oh
8a8d5e89bb src/test: remove unused objects to calculate accurate dedup ratio.
Signed-off-by: Myoungwon Oh <ommw@sk.com>
2019-04-24 18:22:37 +09:00
Xie Xingguo
78bcae1d6b
Merge pull request #27719 from xiexingguo/wip-crush-node-flags
mon/OSDMonitor: crush node flags - two fixes; add tests

Reviewed-by: Sage Weil <sage@redhat.com>
2019-04-24 08:44:08 +08:00
Jason Dillaman
b7c7a4f39e
Merge pull request #27521 from trociny/wip-rbd-remove-clone_v2-parent
librbd: optionally move parent image to trash on remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-04-23 08:11:38 -04:00
xie xingguo
bae2231cc5 qa: add crush-node-flags test
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-04-23 14:20:17 +08:00
Jason Dillaman
b184acd218 qa/workunits/rbd: use more recent qemu-iotests that support Bionic
Fixes: http://tracker.ceph.com/issues/24668
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-04-22 12:46:47 -04:00
Jason Dillaman
5b0a611a48 test/librbd: drop 'ceph_test_librbd_api' target
This was used for API backwards compatibility testing, but now that
the C++ API will not remain stable, it serves no purpose.

Fixes: http://tracker.ceph.com/issues/39072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-04-19 14:18:00 -04:00
Mykola Golub
6e1434eefc librbd: optionally move parent image to trash on remove
and auto-delete when the last clone is detached

Signed-off-by: Mykola Golub <mgolub@suse.com>
2019-04-19 08:53:38 +01:00
Ilya Dryomov
b56e933373
Merge pull request #27271 from idryomov/wip-krbd-wac-on-dm
qa: add device mapper and lvm test cases for stable pages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-04-14 21:31:00 +02:00
myoungwon oh
d72cc5a0ee src/tools: fix dedup tool's name
Signed-off-by: Myoungwon Oh <ommw@sk.com>
2019-04-08 19:19:09 +09:00
myoungwon oh
a095765f2f qa/workunit: add rabin chunk test
Signed-off-by: Myoungwon Oh <ommw@sk.com>
2019-04-04 23:58:57 +09:00
Sage Weil
3760e8f918 mon/OSDMonitor: add mon_debug_no_require_octopus
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-02 16:19:43 -05:00
Sage Weil
2806de1147 mon/OSDMonitor: allow 'osd require-osd-release octopus'
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-02 16:19:43 -05:00
Ilya Dryomov
9c736f57ee qa: krbd_wac.sh: add lvm test case
The script isn't generic anymore, move it to the rbd directory.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-03-29 11:20:30 +01:00
Ilya Dryomov
316ae0cfa4 qa: krbd_stable_pages_required.sh: add device mapper test cases
device mapper core wasn't propagating BDI_CAP_STABLE_WRITES.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-03-28 19:34:15 +01:00
Sage Weil
1dad9d2fe1 Merge PR #27155 into master
* refs/pull/27155/head:
	mon/ConfigMonitor: make 'config reset' idempotent

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-26 11:35:45 -05:00
Ilya Dryomov
0ba913ba2c
Merge pull request #27192 from idryomov/wip-krbd-discard-fixups
qa/workunits/rbd: fixups for the new krbd discard behavior

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-03-26 14:09:41 +01:00
Jason Dillaman
6e32108204
Merge pull request #27137 from trociny/wip-qa-rbd-qos
qa: add RBD QOS functional test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-03-26 08:21:18 -04:00
Ilya Dryomov
f3f757aaa3 qa/workunits/rbd: fixups for the new krbd discard behavior
With discard_granularity set to alloc_size, we no longer get object
size alignment from blk_bio_discard_split().

This assumption is pretty deeply ingrained in krbd_data_pool.sh, so
make it explicit.  For krbd_fallocate.sh, just fix the expectation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-03-25 22:06:14 +01:00
Ilya Dryomov
673673f53d qa: krbd_fallocate.sh: zero can be munged to truncate
The test case is issuing discards that span two objects: the tail of
the first object is truncated, the head of the second object is zeroed.
These discards aren't serial, so there is a race:

  discard i ~ i + 1: truncate i, zero i + 1
  discard i + 1 ~ i + 2: truncate i + 1, zero i + 2

can be executed as

  truncate i + 1, zero i + 2, truncate i, zero i + 1

For object i + 1, the sequence ends up being truncate tail, then zero
head.  This zero op is munged to truncate on the OSD, resulting in size
0 instead of OBJECT_SIZE / 2.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2019-03-25 22:06:14 +01:00
xie xingguo
794e3dfc80 mon/ConfigMonitor: make 'config reset' idempotent
This partially revert 1bc9c86d08.
It's generally not a good idea if a command is not idempotent.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-25 20:35:42 +08:00
Xie Xingguo
c9dbaf279e
Merge pull request #27090 from xiexingguo/wip-config-mon-core
mon/ConfigMonitor: make 'num' of 'config reset' optional; allow target version 0

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-25 13:28:41 +08:00
xie xingguo
1bc9c86d08 mon/ConfigMonitor: make 'num' of 'config reset' command optional
This way it can be used to fast cancel/undo the last command.
Also make the tip message a litter bit nicer..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-25 09:41:05 +08:00
Kefu Chai
b8082ebc56 qa/workunits/mon/config.sh: s|bin/ceph|ceph|
so we can verify that ceph is able to reject wrong debug levels.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-03-23 22:43:08 +08:00
Mykola Golub
63f3d7e377 qa/workunits/rbd: add QOS functional test
Signed-off-by: Mykola Golub <mgolub@suse.com>
2019-03-22 14:58:22 +00:00
Sage Weil
be1187575b Merge PR #27021 into master
* refs/pull/27021/head:
	msg: remove XioMessenger
	qa/suites/rados/thrash-old-clients: add nautilus
	qa/suites/rados/thrash-old-clients: add mimic v1 variant
	qa/suites/rados/thrash-old-clients: add mimic
	qa/suites/rados/thrash-old-clients: collapse msgr and client choice
	qa: remove simplemessenger tests
	ceph_test_msgr: remove simple
	msg: remove SimpleMessenger

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-03-22 04:42:30 -05:00
Kefu Chai
32df73f9f2
Merge pull request #26940 from xiexingguo/wip-monc-add-con
mon/MonClient: weight-based mon selection

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-03-22 15:43:04 +08:00
Sage Weil
cc9a9142fd msg: remove XioMessenger
There is a lot of good stuff going on here, but nobody is investing in xio
and it is not expected to be the path forward for RDMA.  If that ever
changes, we can resurrect the code.  Until then, let's clean up the tree
and reduce friction for changes going forward.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-21 00:57:33 -05:00
Sage Weil
1245809516 Merge PR #26604 into master
* refs/pull/26604/head:
	qa/workunits/rest: Better detection of rest url

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-19 22:45:43 -05:00
xie xingguo
2f195b7072 mon: add set-weight command
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-20 11:23:32 +08:00
Sage Weil
4bb4f7a891 Merge PR #26894 into nautilus
* refs/pull/26894/head:
	qa/standalone/erasure-code/test-erasure-code: adjust test to avoid m=0
	erasure-code: ensure m >= 1
	mon/OSDMonitor: set ec min_size to k + min(1, m - 1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-13 22:07:45 -05:00
Sage Weil
ab0a652826 erasure-code: ensure m >= 1
Fixes: http://tracker.ceph.com/issues/38682
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-12 13:12:58 -05:00
Sage Weil
ccda488815 crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets
The crush/builder.c crush_add_bucket method resizes the max_buckets array
but a power of 2 when it has to expand, but the code in CrushWrapper was
assuming that if the array grew the pos for the new bucket would be the
last position in the new array.  This led to a situation where the
crush_choose_arg_map args array size didn't match max_buckets, and
eventually caused a crash.

Fixes: http://tracker.ceph.com/issues/38664
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-12 11:26:43 -05:00
Sage Weil
4c5ed29925 Merge PR #26764 into master
* refs/pull/26764/head:
	mgr: 'osd df' by specified class or (crush) name
	mon/OSDMonitor: add 'osd crush get-device-class' command
2019-03-07 08:52:56 -06:00
xie xingguo
af02d1031d mgr: 'osd df' by specified class or (crush) name
For large clusters, we use device classes to isolate storage pools.
The existing 'osd df' output turns out to be too nosiy, say, if
you care about only single storage pool with osds possibly spanning over
all hosts.

With this change you are now being able to do 'osd df' by class (or by pool,
if you simply use classes to separate different pools), or by a specified
crush bucket name you are currently interested in, which is much more
convenient.

Some examples:
```
$ bin/ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE   RAW USE DATA    OMAP META  AVAIL   %USE  VAR  PGS STATUS TYPE NAME
-1       0.05878        - 60 GiB 6.4 GiB  23 MiB  0 B 6 GiB  54 GiB 10.60 1.00   -        root default
-3       0.02939        - 30 GiB 3.2 GiB  12 MiB  0 B 3 GiB  27 GiB 10.60 1.00   -            host ceph11
 3   aaa 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  56     up         osd.3
 4   bbb 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  58     up         osd.4
 5   ccc 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  60     up         osd.5
-5       0.02939        - 30 GiB 3.2 GiB  12 MiB  0 B 3 GiB  27 GiB 10.60 1.00   -            host ceph12
 0   aaa 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  50     up         osd.0
 1   bbb 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  61     up         osd.1
 2   ccc 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  51     up         osd.2
                    TOTAL 60 GiB 6.4 GiB  23 MiB  0 B 6 GiB  54 GiB 10.60
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

$ bin/ceph osd df tree class aaa
ID CLASS WEIGHT  REWEIGHT SIZE   RAW USE DATA    OMAP META  AVAIL   %USE  VAR  PGS STATUS TYPE NAME
-1       0.05878        - 20 GiB 2.1 GiB 7.8 MiB  0 B 2 GiB  18 GiB 10.60 1.00   -        root default
-3       0.02939        - 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00   -            host ceph11
 3   aaa 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  56     up         osd.3
-5       0.02939        - 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00   -            host ceph12
 0   aaa 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  50     up         osd.0
                    TOTAL 20 GiB 2.1 GiB 7.8 MiB  0 B 2 GiB  18 GiB 10.60
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

$ bin/ceph osd df tree name ceph11
ID CLASS WEIGHT  REWEIGHT SIZE   RAW USE DATA    OMAP META  AVAIL   %USE  VAR  PGS STATUS TYPE NAME
-3       0.02939        - 30 GiB 3.2 GiB  12 MiB  0 B 3 GiB  27 GiB 10.60 1.00   -            host ceph11
 3   aaa 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  56     up         osd.3
 4   bbb 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  58     up         osd.4
 5   ccc 0.00980  1.00000 10 GiB 1.1 GiB 3.9 MiB  0 B 1 GiB 9.0 GiB 10.60 1.00  60     up         osd.5
                    TOTAL 30 GiB 3.2 GiB  12 MiB  0 B 3 GiB  27 GiB 10.60
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

```

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-06 11:10:56 +08:00
Xie Xingguo
ad8e7d33b1
Merge pull request #26729 from xiexingguo/wip-recovery-priority-restrictions
mon/OSDMonitor: add boundary check for pool recovery_priority

Reviewed-by: David Zafman <dzafman@redhat.com>
2019-03-05 20:16:18 +08:00
xie xingguo
733fff22bb mon/OSDMonitor: add boundary check for pool recovery_priority
See https://github.com/ceph/ceph/pull/26705

Fixes: http://tracker.ceph.com/issues/38578
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-05 08:15:39 +08:00
Sage Weil
bd52f35fd0 Merge PR #26444 into master
* refs/pull/26444/head:
	ceph_features: deprecate MON_SINGLE_PAXOS
	osdmap: map versions to releases
	mon: retire some `osd set/unset` flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-04 17:26:12 -06:00
Joao Eduardo Luis
dfafdf3dae mon: retire some osd set/unset flags
Signed-off-by: Joao Eduardo Luis <joao@suse.de>
2019-03-02 11:36:23 +00:00
Patrick Donnelly
eac565f7b0
Merge PR #26659 into master
* refs/pull/26659/head:
	qa: remove libcephfs-java tests

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2019-03-01 06:53:20 -08:00
Patrick Donnelly
df62f2d91d
qa: remove libcephfs-java tests
These have bit-rotted and no longer work. No cycles from interested parties
available to fix.

Fixes: https://tracker.ceph.com/issues/38487
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-02-28 13:32:15 -08:00
Brad Hubbard
d1892aaf6b qa/workunits/rest: Better detection of rest url
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2019-02-28 07:29:25 +10:00
Jason Dillaman
9d694ba351 qa/workunits/rbd: delete pools before stopping rbd-mirror
This better mimics the behavior of teuthology and tests rbd-mirror
daemon's ability to handle a pool deletion.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-02-25 13:31:43 -05:00
Xie Xingguo
be26cb08e5
Merge pull request #26560 from xiexingguo/wip-pool-force-recovery
mgr: add per pool force-recovery/backfill commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
2019-02-23 15:30:14 +08:00
xie xingguo
fa98f09674 mgr: add per pool force-recovery/backfill commands
For those with multiple storage pools sharing the same devices,
I think it would make much more sense to offer per-pool
commands to bring pools with high priority, e.g., because they
are hosting data of more importance than others, back to normal
quickly.

Fixes: http://tracker.ceph.com/issues/38456
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-02-23 12:18:24 +08:00
Sage Weil
0a7077a855 Merge PR #26237 into master
* refs/pull/26237/head:
	Revert "qa: update test_envlibrados_for_rocksdb.sh for libradospp split"
	doc/librados: explicitly mention that the C++ API is not stable
	ceph.spec: force use of upgrade devtoolset-gcc under RHEL 7
	librados: add symbol versioning to the C++ API
	librados: add symbol versioning to the C API
	librados: revert librados3/libradoscc back to librados2

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-02-21 11:18:46 -06:00
Mykola Golub
a51d9188bb
Merge pull request #26521 from dillaman/wip-38385
qa/workunits: fixed mon address parsing for rbd-mirror
2019-02-21 16:46:05 +02:00