Commit Graph

80419 Commits

Author SHA1 Message Date
Jason Dillaman
5220c7369f
Merge pull request #18504 from badone/wip-GroupSnapshotNamespace-group_pool-uninitialised
cls/rbd: Silence gcc7 maybe-uninitialized warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-11-01 09:08:33 -04:00
Casey Bodley
dd5d278a4f rgw: remove placement_rule from rgw_link_bucket()
with the fallback in RGWRados::update_containers_stats(), we no longer
need to pass placement_rule into the cls_user_bucket_entry

this removes the dependency between bucket entrypoint metadata and its
bucket instance metadata during multisite sync

Fixes: http://tracker.ceph.com/issues/21990

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 09:08:33 -04:00
Casey Bodley
531195d80d rgw: take placement_rule from bucket info in update_containers_stats
in case the cls_user_bucket_entry doesn't contain a placement_rule, take
it from the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-11-01 09:08:27 -04:00
Sage Weil
a59a1df853 mgr/balancer: simplify pool_info tracking
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2c300bc8d7 mgr/balancer: less verbose on 'eval' by default; add 'eval-verbose'
The verbose output is helpful for debugging and understanding what is
being measured, but most of the time all that you care about is the score.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
6513e8b0fc mgr/balancer: fix pg vs object terminology
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
3f0b50b50a mgr/balancer: restrict to time of day
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
297e2d65ab mgr/module: adjust osd_weight min step to .005
That should be ~1 PG or less on average; no real sense it taking a
step smaller than that!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
70b503495a mgr/balancer: if score regresses, take a few more steps
To avoid getting stuck in a local minima, take a few additional
steps if our score drops and see if it gets better.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
1ad3101818 mgr/balancer: allow 5% misplaced
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
0dbcf737ea mgr/balancer: more aggressive steps
We are smart enough to back off if we misplace too much.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2c9c18d1ec qa/suites/rados/thrash/d-balancer: enable balancer in various modes
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
861c37fd99 mgr/balancer: crush-compat: phase out osd_weights
Phase out the osd_weight values as we optimize the crush compat
weight set.  Allow a small regression in overall score if it means
we are taking a step to remove the osd_weight value.

Fix the 'osd reweightn' invocation.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
c2bba5820a mgr/balancer: crush_compat: cope with 'out' osds
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
f1f3014ffc mgr/balancer: stop if we get a perfect score
..and fix the 'worse' threshold (we're just working around floating
point imprecision; 1.01 was too coarse).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
37bd64405b mgr/balancer: more dead code
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
df2e008d33 mgr/balancer: crush-compat: throttle changes based on max_misplaced
Take smaller steps if we overshoot max_misplaced.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
020be37fd0 mgr/balancer: remove dead code
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
f6f2d253c3 mgr/balancer: include pg up mapping in MappingState
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
99fcd161bd mgr/balancer: normalize weight-set weights to sum to target weight
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
26a7ec0c8a mgr/balancer: note root id in Eval
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
d9a31595ba mgr/balancer: make crush-compat mode work!
- it does multiple iterations, like the upmap optimizer.
- it decreases the step size if it isn't improving, in the hope
that it is overshooting the minimum
- debug output is cleaned up a bit (the info level should be
genuinely useful)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 07:28:49 -05:00
Sage Weil
2912364ff8
Merge pull request #16651 from jcsp/wip-mgr-module-interface
mgr: python interface rework + enable modules to run in standby mode

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-11-01 07:27:20 -05:00
John Spray
5861c4c022 mon: include disabled modules in mgr module ls
Otherwise, when someone wants to see what's possible
to do with `mgr module enable` they have to trawl
through the whole mgr map dump.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
d220e1adc3 doc: describe using mgr module ... commands
...including the new "mgr services" command.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
ceb2a91190 doc: describe how to implement standby modules
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
9988ebed95 qa: fix mgr caps
This was still using Kraken era settings

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:43 -04:00
John Spray
05e648be6a qa: expand mgr testing
Some extra coverage of the dashboard, including its standby
redirect mode and the publishing of URIs.

Also invoking the command_spam mode of the selftest module.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
a382c3f1ca mgr/selftest: extend test and add background spam mode
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
27ee148e04 mgr: drop GIL around set_uri, set_health_checks
These didn't need to keep the GIL to go and do their
pure C++ parts, and by keeping it they could deadlock
while trying to take ActiveMgrModules::lock.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
58dfa97ba8 mgr: fix ~MonCommandCompletion
This was doing a Py_DECREF outside of the Gil.

Fixes: http://tracker.ceph.com/issues/21593
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
29193a47e6 mgr: update for SafeThreadState
A bunch of the previous commits were done
before this class existed, so updating in
one go instead of trying to edit history
in fine detail.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
John Spray
7e61f79f5d mgr: refactor PyOSDMap etc implementation
Implement real python classes from the C side,
rather than exposing only module methods.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:21:42 -04:00
Sage Weil
eacc902145 mgr/PyOSDMap: add CRUSH get_item_weight
Signed-off-by: Sage Weil <sage@redhat.com>
2017-11-01 08:20:22 -04:00
John Spray
0d5b1d70e6 mgr: fix py_module_registry shutdown
Was calling way too early, which did a
Py_Finalize before the modules had been
joined.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:22 -04:00
John Spray
bb4e71ed2e mgr: fix thread naming
Was passing a reference to a local stringstream into
Thread::create, not realising that it was taking a char*
reference instead of a copy.  Result was garbage (or usually,
all threads having the name of the last one created)

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:22 -04:00
John Spray
df8797320b mgr: cut down duplication between active+standby
...by using PyModuleRunner class from ActivePyModule too.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
e2442c1e20 mgr: fix os._exit overrides
These would throw an exception when passed
a status code.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
7b629ae465 mon/MgrMonitor: reset services map on drop_active
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
4f7007d1b0 mgr/dashboard: implement standby mode
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
3048e85cd7 pybind/mgr: add MgrStandbyModule
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
c1471c7501 mgr: standby modules come up and run now
...they still don't have access to any config though.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
25566d1edc mgr: enable running modules in standby mode
Modules can implement a second, separate class
that has access to very little state about the
system and can't implement commands.

They have just enough information to redirect
or forward incoming requests/traffic to the
active instance of the module on the active mgr.

This enables module authors to create modules
that end users can access via any (running) mgr node
at any time, rather than having to first work out
which mgr node is active.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:21 -04:00
John Spray
70d45a6b93 mgr: clean up python source file naming
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:20 -04:00
John Spray
9718896c8b mgr: refactor python module management
Separate out the *loading* of modules from
the *running* of modules.

This is a precursor to enabling modules to run
in standby mode.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:20 -04:00
John Spray
089e105dd7 pybind/mgr: use set_uri hook from dashboard+restful modules
No more guessing the URL!

Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:20 -04:00
John Spray
a0183a63fa mgr: enable python modules to advertise their service URI
Fixes: http://tracker.ceph.com/issues/17460
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:20 -04:00
John Spray
c3c3e4e90b mon/MgrMonitor: store services in map and expose with command
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:20 -04:00
John Spray
236841b3b6 messages: services in MMgrBeacon
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:19 -04:00
John Spray
3f703bd91f mon/MgrMap: store list of services
Signed-off-by: John Spray <john.spray@redhat.com>
2017-11-01 08:20:19 -04:00