This is to avoid starting the python modules
before the mon_status and health information
is available.
Fixes: http://tracker.ceph.com/issues/20383
Signed-off-by: John Spray <john.spray@redhat.com>
Kill old mgr_modules option.
Add new mgr_initial_modules option, on the mon, for the initial cluster
mgrmap.
Add ls, enable, disable commands.
Respawn mgr if the module list changes. In the future we could enable
new modules without a full restart, but disabling probably requires (and
is best handled by) a respawn.
Signed-off-by: Sage Weil <sage@redhat.com>
To make it a bit more obvious what's going on, otherwise
user's first sign is when they try and use a CLI bit
and get a command not found.
Signed-off-by: John Spray <john.spray@redhat.com>
In the front of this file, '__func__' has been added to dout_prefix as below:
" #define dout_prefix *_dout << "mgr.server " << __func__ << " " "
Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
Add some brief documentation on updating their caps and link to it.
Fixes: http://tracker.ceph.com/issues/20296
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
It was failing to pass up the Mgr::ms_dispatch
return value, so messages were being consumed
even if Mgr was returning false. Mgr returns
false to enable Client to see FSMap, so the
symptom was the Client not working.
While we're here, refactor switch into if, because
there were only two paths in the switch and we
don't expect to be adding more.
Signed-off-by: John Spray <john.spray@redhat.com>
This deadlocked if the module's shutdown call was trying
to take a pythonland lock that was held by a thread
that was in turn trying to call into C++ land and take
PyModules::lock.
Signed-off-by: John Spray <john.spray@redhat.com>
Anyone instantiating librados instances ended up
appearing as a weird entry in DaemonStateIndex.
Come back to this at the point we want e.g. stats
from RGWs or NFS gateways.
Signed-off-by: John Spray <john.spray@redhat.com>
* reverts c93daaf92b
* add "#include "include/assert.h" to where <cassert> or <assert.h> are
included, or avoid including them if possible.
Signed-off-by: Kefu Chai <kchai@redhat.com>
- It is ugly to unwind all of the Mgr state so that we can reactivate
later.
- It is perhaps impossible to do shut down the python state reliably.
- Respawning provides a clean state and is reliable.
This mostly just copies MDSServer::respawn().
Fixes: http://tracker.ceph.com/issues/19595
Fixes: http://tracker.ceph.com/issues/19549
Signed-off-by: Sage Weil <sage@redhat.com>
This clutters up the log. We should instead log to health, or other
important events.
Note that we still have ceph health logged at regular intervals, so some
of this will surface there.
Signed-off-by: Sage Weil <sage@redhat.com>
we cannot apply pending_inc twice and expect the result is the same. in
other words, pg_map.apply_incremental(pending_inc) is not an idempotent
operation.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This is a goofy workaround that we're also doing in Mgr::init(). Someday
we should come up with a more elegant solution. In the meantime, this
works just fine!
Signed-off-by: Sage Weil <sage@redhat.com>
We want to drop updates for pgs for pools that don't exist. Keep an
updated set of those pools instead of relying on the previous PGMap
having them instantiated. (The previous map may drift due to bugs.)
Signed-off-by: Sage Weil <sage@redhat.com>
We were doing an incremental per osd stat report; this screws up the
delta stats updates when there are more than a handful of OSDs. Instead,
do it with the same period as the mgr->mon reports.
Signed-off-by: Sage Weil <sage@redhat.com>
Instantiate barebones pg records (creating+stale) in our PGMap when pgs
are created. These will switch to 'creating' when the pgs is in the
process of creating, and peering etc. The 'stale' is an indicator that
the mon may not have even asked the pg to create them yet.
All of the old meticulous tracking in PGMap for mappings for creating
pgs is useless to us; OSDMonitor has new code to handle it. This is
fast and simple.
Signed-off-by: Sage Weil <sage@redhat.com>
* extract send_report() out of tick() so it can be reused.
* add a commmand "mgr report-mon" for mgr, so we are able to flush the
the mgr stats to mon actively without waiting for the tick. this
could help with the tests.
Signed-off-by: Kefu Chai <kchai@redhat.com>
we don't send pgstat to mon from osd once we set
CEPH_OSDMAP_REQUIRE_LUMINOUS. and cleanup the pgstat queue when we
do the switch.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Everything summary-ish that we need to send to the mon is moved into a
parent class. The child PGMap retains the detail.
The parent gets its own encode(), and PGMap::encode_digest() will call it
to encode just the summary info.
Squashed in here is a new num_pg_by_osd that could have been done
in a preceding patch but I did things in the wrong order. :(
Signed-off-by: Sage Weil <sage@redhat.com>
We are about to want similar time-based stuff in our other
classes and it's simplest to coalesce them into a single wakeup.
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
Also assert in OSD::ms_get_authorizer since it is dangerous to proceed
if we are shutting down.
Fixes: http://tracker.ceph.com/issues/19900
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
This provides a reasonable amount of isolation between mgr
modules. Notably, with this change, it's possible to have more
than one mgr module use cherrypy without conflicts.
Each MgrPyModule gets its own python sub-interpreter. The logger,
the ceph_state module and sys.path need to be set up separately
for each sub-interpreter, so all that happens in MgrPyModule's
constructor (previously this was done on the main interpreter
in PyModules::init()).
Each sub-interpreter has its own python thread state. The main
interpreter also has a python thread state, but that is almost
unused except for during setup and teardown of the whole beast.
On top of that, an additional python thread state is necessary
for each OS thread that calls into python code (e.g. the serve()
method).
Some care needs to be taken to ensure that the right thread state
is active at the right time; note how the call to handle_pyerror()
in PyModules::init() had to be moved inside MgrPyModle::load().
PyModules should be using the main thread state, and MgrPyModule
should usually be using its sub-interpreter thread state, except
for very early in its constructor (before the sub-interpreter has
been created), and in the serve() method where another python
thread state is created to map to the separate OS thread that is
responsible for invoking this method.
The ceph_state module (PyState.cc) naturally has no idea what
context it's being run in, so uses PyThreadState_Get() when it
needs to know the python thread state.
Signed-off-by: Tim Serong <tserong@suse.com>
Prep work for loading modules in separate python sub-interpreters.
According to the Python C API docs, mixing the PyGILState_*() API
with sub-interpreters is unsupported, so I've replaced these calls
with a Gil class, which will acquire and release the GIL against
a specific python thread state.
Note the manual python thread state creation in MgrPyModule::serve().
The PyGILState_*() APIs would have done that for us, but we're not
using them anymore, so have to do it by hand (see
https://docs.python.org/2/c-api/init.html#non-python-created-threads
for some description of this).
Signed-off-by: Tim Serong <tserong@suse.com>
Creating an MgrStandby in order to call its usage() method has
two problems:
1) MgrStandby wants to connect to a running cluster; if the
cluster isn't up, you'll be left staring at a blinking cursor.
2) MgrStandby::usage() doesn't print anything anyway.
This commit takes pretty much the same approach to printing
usage information as ceph_mon.cc, ceph_osd.cc and ceph_mds.cc.
Signed-off-by: Tim Serong <tserong@suse.com>
if the size of the cluster is quite large, we don't need to walk through
all daemons to do the cull.
* smaller scope protected by the lock
* check smaller range in "all"
* use vector<string> for holding victims: all of them are of
"daemon_type".
Signed-off-by: Kefu Chai <kchai@redhat.com>
For modules that would like a CephContext* inside
a capsule, suitable for passing to librados constructor.
Signed-off-by: John Spray <john.spray@redhat.com>
Stop the MgrClient callbacks to report PG stats at the
start of shutdown() so that we don't get a callback
during/after we are done shutting down.
Protect the cb update with the MgrClient's lock so that
we don't race with MgrClient::send_report() itself.
Fixes: http://tracker.ceph.com/issues/19638
Signed-off-by: Sage Weil <sage@redhat.com>
Let's be consistent and push the m->put down into
the handle_(M<class>* m) functions the way
we do in the rest of the codebase.
Signed-off-by: John Spray <john.spray@redhat.com>
I think we got away with this when Client::init was
re-initialising the MonClient, but now that (bogus)
stuff isn't happening any more as Client doesn't
init/teardown monc/objecter itself.
Signed-off-by: John Spray <john.spray@redhat.com>
Run the python module calls in a finisher so that
they don't block the daemonserver lock and so that
they can call back into mgr stuff if they need to.
Fix passing through commands to python modules, this
was giving EINVAL because only things with a MgrCommand
were getting let in.
Also fix get_command_descriptions, which was not
including the output of the formatter in the response.
Signed-off-by: John Spray <john.spray@redhat.com>
otherwise we can shutdown a PyModules while it still being init'ed
Fixes: http://tracker.ceph.com/issues/19743
Signed-off-by: Kefu Chai <kchai@redhat.com>
* also do prevent python from registering its own signal handler, it does
not make sense in our embedded use case.
* pass python interpreter's path to embedded python before initializing
it. python uses this path to look up the "site" modules, etc.
so it can use other interpreter if it is not intalled into $PATH,
otherwise the "python" in $PATH will always be used even the
PYTHON_EXECUTABLE in CMake's cache is pointing to another python
interpreter.
if one want to debug ceph-mgr with a customize build python, this would
be helpful.
Signed-off-by: Kefu Chai <kchai@redhat.com>
If config-key put fails, we now log the failure rather than asserting that
it must always succeed.
Fixes: http://tracker.ceph.com/issues/19629
Signed-off-by: Tim Serong <tserong@suse.com>
ds is intended for command output (like stdout), to be shown/saved
when there's no error returned; ss is for error status (like stderr)
to further explain the error. mgr Python tracebacks should go to
the latter.
Signed-off-by: Dan Mick <dan.mick@redhat.com>
If an exception is raised in any of the python module functions
invoked by mgr (for example, ImportError if rest_framework isn't
available when serve() in called in the rest module), print the
module name along with the failed method name.
This commit also fixes what looks to me like a missed call to
PyGILState_Release() in the failure case in MgrPyModule::serve().
Signed-off-by: Tim Serong <tserong@suse.com>
In abde684 we started always fetching metadata on fsmap changes
for legacy daemons (i.e. those without addr fields) -- this works
but is very inefficient.
By injecting an 'addr' field based on the address in
the FSMap, if the field is missing, we can reload
metadata only on daemon restarts.
Signed-off-by: John Spray <john.spray@redhat.com>
1. this belongs on the mon, operating against the latest
osdmap.
2. it shouldn't go in PGMonitor because that is about to
be turned off.
Signed-off-by: Sage Weil <sage@redhat.com>
use unique_ptr to manage the lifecycle of MgrPyModule and ServeThread,
it's easier and safer. without this chance, we don't free allocated
MgrPyModule if it fails to load().
Fixes: http://tracker.ceph.com/issues/19590
Signed-off-by: Kefu Chai <kchai@redhat.com>
Wire up the sighup_handler to reopen logs, like the other daemons.
Fixes: http://tracker.ceph.com/issues/19568
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Saw a run that showed the notify_all pg_summary as the last
message for this thread; hopefully we can see if it is
blocked in notify_all() or not.
Signed-off-by: Sage Weil <sage@redhat.com>
This avoids a deadlock if some other message arrives before
the command reply and needs to take the mgr lock in dispatch.
Signed-off-by: Sage Weil <sage@redhat.com>
If we go active multiple times we may already have an fsmap
subscription registered via MonClient, which means that we
won't receive a net new FSMap during the init phase. This is
trivial to reproduce; just 'ceph mgr fail x' and when it
activates the second time init() will get stuck waiting.
The core issue is that monc subs are stateful, but we forget
the various maps with the previous Mgr incarnation. Sipmle
workaround is to reset the mon connection as we init.
Signed-off-by: Sage Weil <sage@redhat.com>
