This python script is making excessive sleep calls while running some
ceph commands.
Waiting up to 5 seconds to get the proper health status can be shorten
to avoid the worst case of waiting almost 5 seconds for nothing.
This patch removes also two sleeps calls after a wait_for_health call
which is already supposed to provides a clean state. Waiting
respectively 20 & 15 seconds after that call is just loosing time which
is precious at make check time.
Signed-off-by: Erwan Velu <erwan@redhat.com>
OSDs are taking some time to be up but waiting 10 secs seems execessive
here between two loops. In the worst case, we can be in a situation of
waiting 10secs for nothing as we are just a few microsecs after the osd
is up.
This patch simply reduce the sleep from 10 to 1 seconds.
Signed-off-by: Erwan Velu <erwan@redhat.com>
It could sounds like nothing but the actual sleeping rampup is counter
productive.
The code does : kill <proc>; sleep 0; kill <proc>; sleep 0; kill <proc;
sleep 1; and then it grows up 120 seconds by a smooth rampup.
But actually there is almost no chance the process dies so fast meaning
that by default we switch to the sleep 1.
Moving from sleep 0 to sleep 1 doesn't seems a big win but as
kill_daemons() is called very often we can save a lot of time by then
end.
This patch offer to sleep first a 1/10th of second instead of 0 and then
1/20th of second instead of 0.
The sleep call is also moved after the kill call as it's not necessary
waiting before executing the command.
This patch makes the running time of a test like osd-scrub-repair.sh
dropping from 7m30 to 7m7.
Saving another ~30seconds is an interesting win at make check level.
Signed-off-by: Erwan Velu <erwan@redhat.com>
osd-scrub-repair is making several similar objectore calls in a
sequential way while they could be easily parallelized.
Each single objectore call can spent up to dozen of seconds so making
the call parallel is saving a lot of time while keeping the code pretty
simple.
This particular patch saves approx. 2 minutes on the actual code on a recent
laptop. The global running time of osd-scrub-repair drops from 9m33 to
7m37 !
Signed-off-by: Erwan Velu <erwan@redhat.com>
wait_for_clean() is a very common call when running the make check.
It does wait the cluster to be stable before continuing.
This script was doing the same calls twice and could be optimized by
making the useful calls only once.
is_clean() function was checking num_pgs & get_num_active_clean()
The main loop itself was also calling get_num_active_clean()
This patch is inlining the is_clean() inside this loop to benefit from a
single get_num_active_clean() call. This avoid a useless call of (ceph +
xmlstarlet).
This patch does move all the 'timer reset' conditions into an else
avoiding spawning other ceph+xmlstarlet call while we already know we
should reset the timer.
The last modification is to reduce the sleeping time as the state of the
cluster is changing very fast.
This whole patch could looks like almost not a big win but for a test
like test/osd/osd-scrub-repair.sh, we drop from 9m56 to 9m30 while
reducing the number system calls.
At the scale of make check, that's a lot of saving.
Signed-off-by: Erwan Velu <erwan@redhat.com>
get_num_active_clean() is called very often but spawn 1 useless process.
The current "grep -v | wc -l" can be easily replaced by "grep -cv" which
do the same while spawning one process less.
Signed-off-by: Erwan Velu <erwan@redhat.com>
The current code of kill_daemons() was killing daemons one after the
other and wait it to actually die before switching to the next one.
This patch makes the kill_daemons() loop being run in parallel to avoid
this bottleneck.
Signed-off-by: Erwan Velu <erwan@redhat.com>
This script had the following performance issue :
- 4 ceph-dencoders spawn sequentialy
- running twice the same dencoder command
This patch is adding parallelism around the 4 sequential calls but also
prevent from testing the deterministic feature twice.
On a recent laptop, this patch drops the running time from 7mn to 3m46
while keeping the loadavg < 2.
Signed-off-by: Erwan Velu <erwan@redhat.com>
The current code was running sequentially two ceph-dencoder calls.
This process is executed pretty fast but adding sequentiality and by the number
of loops to execute, it have a cost.
This patch is just making this two calls being run in parallel.
As a result, the test/encoding/readable.sh test is running in 4m50 instead of 6.
The associate loadavg isn't impacted as it stays at 6 while being run with
nproc=8.
This patch save 1/6th of building time without impact the loadavg.
Signed-off-by: Erwan Velu <erwan@redhat.com>
When running make -j x check, we face a weird situation where the makefile
targets are spawn in parallel up to "x" but one of those target is very very
long and sequential.
The "readable.sh" test is trying to run ~7.9K tests where 5.3K are actually
executed.
The current code is taking 23mn on a recent laptop (Intel(R) Core(TM)
i7-4810MQ CPU @ 2.80GHz, 32GB of RAM & SSD).
This patch implements parallelism to speed up this process which is not really CPU and
neither IO bound.
By default, readable.sh is now using the number of logical processors to determine
the level of parallelism (by using nproc). If needed, defining the MAX_PARALLEL_JOBS
variable will override this default value.
On the same system, where nproc=8, the resulting execution time is 5m55 seconds :
4x faster than the original code.
The global 'make check' is therefore getting faster too and dropped from 30 to
16 minutes : 2x faster than the original code.
Signed-off-by: Erwan Velu <erwan@redhat.com>
This commit introduce two new functions in ceph-helpers.sh to ease
parallelism in tests.
It's based on two functions : run_in_background() & wait_background()
The first one allow you to spawn processes or functions in background and saves
the associated pid in a variable passed as first argument.
The second one waits for thoses pids to complete and report their exit status.
If one or more failed then wait_background() reports a failure.
A typical usage looks like :
pids1=""
run_in_background pids1 bash -c 'sleep 5; exit 0'
run_in_background pids1 bash -c 'sleep 1; exit 1'
run_in_background pids1 my_bash_function
wait_background pids1
The variable that contains pids is local making possible to do nested calls of
thoses two new functions.
Signed-off-by: Erwan Velu <erwan@redhat.com>
Since the merge of pr #7693, 'ceph command' to get the help is invalid.
As a result, 'test/cephtool-test-mon.sh' test was broken
This patch simply change the 'ceph command' by a 'ceph --help command'
Since this change the test is passing again.
Signed-off-by: Erwan Velu <erwan@redhat.com>
This should only happen with a buggy client, but we should avoid crashing,
and send a polite error message back.
Signed-off-by: Sage Weil <sage@redhat.com>
Normal ops do this so they can behave when racing with split; pgnls ops
are no different.
In particular, this fixes a bug where we have an old OSDMap that doesn't
reflect a split, and the OSD replies with a 'next' value of the PG's new
max. If we resend the same value to that PG, it'll be out of bounds,
and BlueStore will notice.
Signed-off-by: Sage Weil <sage@redhat.com>
otherwise vector::push_back() will use the copy ctor if it resizes,
to enforce its strong exception guarantee.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Move initialization to the few tests that actually use it.
Fixes: http://tracker.ceph.com/issues/15225
Signed-off-by: Richard W.M. Jones <rjones@redhat.com>
