In read cases track stats in PG::unstable_stats
Include unstable_stats in write_info() and publish_stats_to_osd()
For now this information may not get persisted
fixes: #2209
Signed-off-by: David Zafman <david.zafman@inktank.com>
pg_stats_lock to pg_stats_publish_lock
pg_stats_valid to pg_stats_publish_valid
pg_stats_stable to pg_stats_publish
update_stats() to publish_stats_to_osd()
clear_stats() to clear_publish_stats()
Signed-off-by: David Zafman <david.zafman@inktank.com>
This resolves the leveldb growth-without-bound problem observed by
mikedawson, and all the badness that stems from it. Enable this by
default until we figure out why leveldb is not behaving better.
While we are at it, trim more states at a time. This will make
compaction less frequent, which should help given that there is some
overhead unrelated to the amount of deleted data.
Fixes: #4815
Signed-off-by: Sage Weil <sage@inktank.com>
Each time we trim a PaxosService, have leveldb compact so that the
space from removed states is reclaimed.
This is probably not optimal if leveldb's heuristics are doing the right
thing, but it currently appears as if they are not.
Signed-off-by: Sage Weil <sage@inktank.com>
Add a prefix compaction opteration to the transaction that will be
performed after the transaction applies.
Signed-off-by: Sage Weil <sage@inktank.com>
This is an opportunistic time to optimize our local data since we are
out of quorum. It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.
Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB. And a 9GB store compact down to 900MB in only 1 minute.
Signed-off-by: Sage Weil <sage@inktank.com>
This is an opportunistic time to optimize our local data since we are
out of quorum. It serves as a safety net for cases where leveldb's
automatic compaction doesn't work quite right and lets things get out
of hand.
Anecdotally we have seen stores in excess of 30GB compact down to a few
hundred KB. And a 9GB store compact down to 900MB in only 1 minute.
Signed-off-by: Sage Weil <sage@inktank.com>
This is a workaround that makes the warning go away. Not certain there
isn't something we should be changing...
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joe Buck <joe.buck@inktank.com>
After Monitor::init_paxos() has loaded all of the PaxosService state,
we should then map creating pgs to osds. This ensures we do so after the
osdmap has been loaded and the pgs actually map somewhere meaningful.
Fixes: #4675
Signed-off-by: Sage Weil <sage@inktank.com>
This avoids calculating new pg creation mappings if the osdmap isn't
loaded yet, which currently happens when during Monitor::paxos_init()
on startup. Assuming osdmap epoch is nonzero, it should always be
safe to do this (although possibly unnecessary).
More cleanup here is certainly possible, but this is one step toward fixing
the bad behavior for #4675.
Signed-off-by: Sage Weil <sage@inktank.com>
Factor out the portion of the function that remaps creating pgs to osds
from the part that sends those pending creates out.
Signed-off-by: Sage Weil <sage@inktank.com>
If we get a dup reply something is probably wrong! We should make sure
it appears more loudly in the log. In particular, it can lead to out
of sync cap state; see #4853.
Signed-off-by: Sage Weil <sage@inktank.com>
A sequence like:
- ceph-fuse starts, make_request on getattr
- waits for mds to be active
- tries to open a session
- mds restarts, recovers
- eventually gets session open reply
- sends first getattr (even tho mds is in reconnect state)
- gets mdsmap update that mds is now active
- kicks request, resends getattr
- get first reply
- ignore second reply, caps get out of sync
The bug is that we send the first request when the MDS is still in
the reconnect state. The fix is to loop in make_request so that we
ensure all conditions are satisfied before sending the request. Any
time we wait, we loop, so that we know all conditions (still) pass if
we make it to the end.
Fixes: #4853
Signed-off-by: Sage Weil <sage@inktank.com>
