Commit Graph

80191 Commits

Author SHA1 Message Date
Yan, Zheng
1f27ebbef9 mds: optimize MDCache::try_subtree_merge
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-11-22 16:25:02 +08:00
Yan, Zheng
fb9277c143 mds: optimize import/export state access
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-11-03 14:18:14 +08:00
Sage Weil
aa828be008
Merge pull request #17983 from liewegas/wip-balancer
mgr/balancer: make crush-compat mode work

Reviewed-by: John Spray <john.spray@redhat.com>
2017-11-01 14:46:07 -05:00
Matt Benjamin
e857d71acf
Merge pull request #18657 from cbodley/wip-21990
rgw: remove placement_rule from rgw_link_bucket()
2017-11-01 11:47:24 -04:00
Yuri Weinstein
1465a86ad9
Merge pull request #18534 from ZVampirEM77/wip-emlc-contentmd5
rgw: lc support Content-MD5 request header and fix a rgw crash bug

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2017-11-01 08:27:29 -07:00
Yuri Weinstein
223f45e4e1
Merge pull request #18610 from cbodley/wip-osd-subop-num
osd: add processed_subop_count for cls_cxx_subop_version()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2017-11-01 08:24:13 -07:00
Yuri Weinstein
b51f285676
Merge pull request #18641 from joke-lee/rgw_modify_subuser_access_error
rgw: modify s3 type subuser access permissions fail through admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 08:22:12 -07:00
Kefu Chai
2a3d3b87fe
Merge pull request #18659 from tchaikov/wip-21993
qa: stop testing deprecated "ceph osd create"

Reviewed-by: Sage Weil <sage@redhat.com>
2017-11-01 23:08:55 +08:00
Kefu Chai
01a65f0a0d
Merge pull request #15193 from mynaramana/arraycode
osd: introduce sub-chunks to erasure code plugin interface

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-11-01 22:50:24 +08:00
John Spray
e18a529c9b
Merge pull request #18637 from yaozongyou/fix-dashboard-bug
osd: make stat_bytes and stat_bytes_used counters PRIO_USEFUL

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
2017-11-01 15:07:36 +01:00
Sage Weil
5ba89bf7ce
Merge pull request #18347 from Synesis-LLC/fix-21773
mgr: skip first non-zero incremental in PGMap::apply_incremental()

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-11-01 08:39:11 -05:00
Sage Weil
1df0acca0e
Merge pull request #18453 from liewegas/wip-osd-devs
osd: enumerate device names in a simple way

Reviewed-by: John Spray <john.spray@redhat.com>
2017-11-01 08:36:59 -05:00
Sage Weil
4efbb32f6f
Merge pull request #18509 from tchaikov/wip-osd-cleanups
osd,mgrclient: pass daemon_status by rvalue ref and other cleanups

Reviewed-by: John Spray <john.spray@redhat.com>
2017-11-01 08:36:37 -05:00
Casey Bodley
8e62e35266 rgw: remove placement_rule from cls_user_bucket_entry
placement_rule is no longer needed in cls_user_bucket_entry, because the
only time that it's needed, we can read it from the bucket instance in
RGWRados::update_containers_stats()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 09:08:34 -04:00
Jason Dillaman
5220c7369f
Merge pull request #18504 from badone/wip-GroupSnapshotNamespace-group_pool-uninitialised
cls/rbd: Silence gcc7 maybe-uninitialized warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-11-01 09:08:33 -04:00
Casey Bodley
dd5d278a4f rgw: remove placement_rule from rgw_link_bucket()
with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 09:08:33 -04:00
Casey Bodley
531195d80d rgw: take placement_rule from bucket info in update_containers_stats
in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 09:08:27 -04:00
Sage Weil
a59a1df853 mgr/balancer: simplify pool_info tracking
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2c300bc8d7 mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'
The verbose output is helpful for debugging and understanding what is
being measured, but most of the time all that you care about is the score.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
6513e8b0fc mgr/balancer: fix pg vs object terminology
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
3f0b50b50a mgr/balancer: restrict to time of day
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
297e2d65ab mgr/module: adjust osd_weight min step to .005
That should be ~1 PG or less on average; no real sense it taking a
step smaller than that!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
70b503495a mgr/balancer: if score regresses, take a few more steps
To avoid getting stuck in a local minima, take a few additional
steps if our score drops and see if it gets better.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
1ad3101818 mgr/balancer: allow 5% misplaced
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
0dbcf737ea mgr/balancer: more aggressive steps
We are smart enough to back off if we misplace too much.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2c9c18d1ec qa/suites/rados/thrash/d-balancer: enable balancer in various modes
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
861c37fd99 mgr/balancer: crush-compat: phase out osd_weights
Phase out the osd_weight values as we optimize the crush compat
weight set.  Allow a small regression in overall score if it means
we are taking a step to remove the osd_weight value.

Fix the 'osd reweightn' invocation.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
c2bba5820a mgr/balancer: crush_compat: cope with 'out' osds
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
f1f3014ffc mgr/balancer: stop if we get a perfect score
..and fix the 'worse' threshold (we're just working around floating
point imprecision; 1.01 was too coarse).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
37bd64405b mgr/balancer: more dead code
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
df2e008d33 mgr/balancer: crush-compat: throttle changes based on max_misplaced
Take smaller steps if we overshoot max_misplaced.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
020be37fd0 mgr/balancer: remove dead code
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
f6f2d253c3 mgr/balancer: include pg up mapping in MappingState
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
99fcd161bd mgr/balancer: normalize weight-set weights to sum to target weight
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
26a7ec0c8a mgr/balancer: note root id in Eval
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
d9a31595ba mgr/balancer: make crush-compat mode work!
- it does multiple iterations, like the upmap optimizer.
- it decreases the step size if it isn't improving, in the hope
that it is overshooting the minimum
- debug output is cleaned up a bit (the info level should be
genuinely useful)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2912364ff8
Merge pull request #16651 from jcsp/wip-mgr-module-interface
mgr: python interface rework + enable modules to run in standby mode

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-11-01 07:27:20 -05:00
John Spray
5861c4c022 mon: include disabled modules in mgr module ls
Otherwise, when someone wants to see what's possible
to do with `mgr module enable` they have to trawl
through the whole mgr map dump.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
d220e1adc3 doc: describe using mgr module ... commands
...including the new "mgr services" command.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
ceb2a91190 doc: describe how to implement standby modules
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
9988ebed95 qa: fix mgr caps
This was still using Kraken era settings

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
05e648be6a qa: expand mgr testing
Some extra coverage of the dashboard, including its standby
redirect mode and the publishing of URIs.

Also invoking the command_spam mode of the selftest module.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
a382c3f1ca mgr/selftest: extend test and add background spam mode
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
27ee148e04 mgr: drop GIL around set_uri, set_health_checks
These didn't need to keep the GIL to go and do their
pure C++ parts, and by keeping it they could deadlock
while trying to take ActiveMgrModules::lock.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
58dfa97ba8 mgr: fix ~MonCommandCompletion
This was doing a Py_DECREF outside of the Gil.

Fixes: http://tracker.ceph.com/issues/21593
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
29193a47e6 mgr: update for SafeThreadState
A bunch of the previous commits were done
before this class existed, so updating in
one go instead of trying to edit history
in fine detail.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
7e61f79f5d mgr: refactor PyOSDMap etc implementation
Implement real python classes from the C side,
rather than exposing only module methods.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
Sage Weil
eacc902145 mgr/PyOSDMap: add CRUSH get_item_weight
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 08:20:22 -04:00
John Spray
0d5b1d70e6 mgr: fix py_module_registry shutdown
Was calling way too early, which did a
Py_Finalize before the modules had been
joined.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:22 -04:00
John Spray
bb4e71ed2e mgr: fix thread naming
Was passing a reference to a local stringstream into
Thread::create, not realising that it was taking a char*
reference instead of a copy.  Result was garbage (or usually,
all threads having the name of the last one created)

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:22 -04:00