We had a lot of duplicate code between the WorkloadGenerator and the
TestFileStoreState classes, and the last one is far more versatile than
what we initially had in the WorkloadGenerator. Therefore, delegate
everything we can to the TestFileStoreState class.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Fix wait_for_ready() and make the C_OnFinished class' member variables
protected instead of private (to allow proper inheritance).
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
This fixes#2342. We shouldn't call notify on the dispatcher
context. We should also make sure that we don't hold
the client lock while waiting for the responses.
Also, pushed the client_lock locking into the
ctx->notify().
Signed-off-by: Yehuda Sadeh <yehuda@hq.newdream.net>
Thrash the osdmap for N iterations. Randomly mark OSDs up, down, in, out,
and up_thru in order to generate a difficult osdmap history for peering
to chew through.
Signed-off-by: Sage Weil <sage@newdream.net>
Peef_info_requested should be a strict subset of the probe set. Filter
osds that are dropped from probe from peer_info_requested. We could also
restart peering from scratch here, but this is less expensive, because we
don't have to re-probe everyone.
Once we adjust the probe and peer_info_requested sets, (re)check if we're
done: we may have been blocedk on a previous peer_info_requested entry.
The situation I saw was:
"recovery_state": [
{ "name": "Started\/Primary\/Peering\/GetInfo",
"enter_time": "2012-04-25 14:39:56.905748",
"requested_info_from": [
{ "osd": 193}]},
{ "name": "Started\/Primary\/Peering",
"enter_time": "2012-04-25 14:39:56.905748",
"probing_osds": [
79,
191,
195],
"down_osds_we_would_probe": [],
"peering_blocked_by": []},
{ "name": "Started",
"enter_time": "2012-04-25 14:39:56.905742"}]}
Once in this state, cycling osd.193 doesn't help, because the prior_set
is not affected.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Reviewed-by: Samuel Just <samuel.just@dreamhost.com>
Prevent the monitor from marking osds down or out when too many are already
in that state. At this point the cluster is already broken and there is
little point in continuing to mark things down/out.
Setting these to 0 obviously disables the feature (by setting a minimum
of 0).
Signed-off-by: Sage Weil <sage@newdream.net>
So we can generalize beyond NO* flags. We'll soon be adding other reasons
to not mark things up/down/in/out. This lets us keep all though checks in
one place.
The helper methods will tell us why we can't do the thing (e.g., "NODOWN
flag is set"). The callers will generally tell us exactly what didn't
happen (e.g., "failure report of X ignored").
Signed-off-by: Sage Weil <sage@newdream.net>
get_coll_at(int pos) should return the collection at the map's position
'pos', but 'pos' was being used as a map key. Therefore, we add a new
function 'get_coll(int key)' to mimic this behavior, and we make
'get_coll_at()' follow its intended behavior.
This patch may affect the test_filestore_idempotent_sequence tester, since
it uses the 'get_coll_at()' function a lot, and we changed this function's
behavior.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
So far, it hasn't triggered any segfault, but I'm not yet convinced there
is no problem whatsoever.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
We need to distinguish between the old 0 (meaning undefined) and
the new 0 (meaning switch to 0 and disable the flags). So rev the
encoding version on PGMap::Incremental, and if you decode an old
version with [near]full_ratio == 0, set the ratio to -1 instead. Then
when applying the Incremental interpret -1 as no change.
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
Reviewed-by: Sage Weil <sage@newdream.net>
Do not mark down osds out when NOOUT flag is set. This is more or less
equivalent to setting a very long 'mon osd down out interval', but
reversible and less annoying.
Signed-off-by: Sage Weil <sage@newdream.net>
If the NOIN osdmap flag is set, do not mark booting osds in. Normally
we would for a range of reasons (always, new, auto-marked-out), but block
them all.
Do not limit manual 'ceph osd in N' commands.
Signed-off-by: Sage Weil <sage@newdream.net>
The down_pending_out tracks OSDs that are down that we may want to
auto-mark out. If an osd boots, it should be removed from this list
because it is no longer down; it doesn't matter whether it is marked in
or not.
Signed-off-by: Sage Weil <sage@newdream.net>
If the NODOWN osdmap flag is set,
- ignore osd failure reports
- do not mark osds down due to lack of osd/pg stats
We *do* still allow explicit admin 'ceph osd down N' commands, and a
booting OSD to mark the previous instance of itself down.
Signed-off-by: Sage Weil <sage@newdream.net>
If NOUP is set, do not send the boot message.
We already send onetime subscriptions to the osdmap, so we will find out
about osdmap flag changes. If it is cleared later, we'll pass into
start_boot() and _got_boot_version() again and send it then.
Signed-off-by: Sage Weil <sage@newdream.net>
Do not add an osd attempting to boot to the map if NOUP is sent. Instead,
send it the latest osdmap so it knows that it's not allowed to boot.
Signed-off-by: Sage Weil <sage@newdream.net>
In particular, the OSD may return EBUSY if there are still watchers.
Ignore ENOENT, as that may indicate we are cleaning up a previously
aborted removal.
Fixes: #2311
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
We clear out the osd entry when an osd goes up or down. Thus, if we find
it missing from an up osd, we should start the timer. Otherwise we get
behavior like this
2012-04-24 13:22:47.888291 7fa5bc587700 mon.peon5752@0(leader).osd e21633 OSDMonitor::handle_osd_timeouts: never got MOSDPGStat info from osd 521. Marking down!
2012-04-24 13:22:50.076394 7fa5bcd88700 log [INF] : osd.521 [2607:f298:4:2243::7088]:6806/53217 boot
2012-04-24 13:22:52.903558 7fa5bc587700 mon.peon5752@0(leader).osd e21638 OSDMonitor::handle_osd_timeouts: never got MOSDPGStat info from osd 521. Marking down!
2012-04-24 13:23:15.144532 7fa5bcd88700 log [INF] : osd.521 [2607:f298:4:2243::7088]:6806/53217 boot
2012-04-24 13:23:17.967118 7fa5bc587700 mon.peon5752@0(leader).osd e21663 OSDMonitor::handle_osd_timeouts: never got MOSDPGStat info from osd 521. Marking down!
2012-04-24 13:23:22.173778 7fa5bcd88700 log [INF] : osd.521 [2607:f298:4:2243::7088]:6806/53217 boot
2012-04-24 13:23:22.981556 7fa5bc587700 mon.peon5752@0(leader).osd e21668 OSDMonitor::handle_osd_timeouts: never got MOSDPGStat info from osd 521. Marking down!
2012-04-24 13:23:45.245380 7fa5bcd88700 log [INF] : osd.521 [2607:f298:4:2243::7088]:6806/53217 boot
when the pg stats message doesn't arrive quickly enough.
Fixes: #2341
Signed-off-by: Sage Weil <sage@newdream.net>
Reviewed-by: Greg Farnum <gregory.farnum@dreamhost.com>
The indices were set incorrectly when I whipped thi sup. That's what
you get for not testing nor being careful enough in review.
Signed-off-by: Greg Farnum <gregory.farnum@dreamhost.com>
* snap_set to a deleted (and recreated) snapshot
* resizing down (truncating) and back up
* resizing to non-object-aligned sizes
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
This way we can't miss an update if we get a notify during ictx_refresh.
Specifically, a race like this:
Thread 1 Thread 2 Process 2
ictx_refresh()
read_header()
snap_create()
notify()
need_refresh = true
process header...
need_refresh = false
If this happened, we would not re-read the header with the new
snapshot, so the snapshot would not happen at the intended point
in time, but only after we re-read the header again.
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
* snapid should determine whether our mapped snapshot is gone, not snapname
* snap_set(<nonexistent_snap>) shouldn't reset us to CEPH_NOSNAP
* snapname should be set before using the it in the perfcounter name
* snapname and image name don't need to be passed as arguments since an
ImageCtx already contains that info
* ictx_check() doesn't need to check for non-existent snaps - only I/Os care,
so check in check_io() instead
Signed-off-by: Josh Durgin <josh.durgin@dreamhost.com>
Allow for '-h' and other options such as disabling the journal sync tests,
defining it is to be run on a btrfs FS, enabling exit on error (default is
now 'off'), and allow certain env variables to specify additional options
to each store.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
