If the cluster dies during the rados bench, the maximum running time is
no more considered and all emitted aios are pending.
rados bench never quits and the global testing timeout (3600 sec : 1
hour) have to be reach to get a failure.
This situation is dramatic for a background test or a CI run as it locks
the whole job for too long for an event that will never occurs.
This ideal solution would be having 'rados bench' considering a failure
once the timeout is reached when aios are pending.
A possible workaround here is to put use the system command 'timeout'
before calling rados bench and fail if rados didn't completed on time.
To avoid side effects, this patch is doubling rados timeout. If rados
didn't completed after twice the expected time, it have to fail to avoid
locking the whole testing job.
Please find below the way it worked on a real test case.
We can see no IO after t>2 but despite timeout=4 the bench continue.
Thanks to this patch, the bench is stopped at t=8 and return 1.
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon: timeout 8 rados -p foo bench 4 write -b 4096 --no-cleanup
5: hints = 1
5: Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 4 seconds or 0 objects
5: Object prefix: benchmark_data_mr-meeseeks_184960
5: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
5: 0 0 0 0 0 0 - 0
5: 1 16 1144 1128 4.40538 4.40625 0.00412965 0.0141116
5: 2 16 2147 2131 4.16134 3.91797 0.00985654 0.0109079
5: 3 16 2147 2131 2.77424 0 - 0.0109079
5: 4 16 2147 2131 2.0807 0 - 0.0109079
5: 5 16 2147 2131 1.66456 0 - 0.0109079
5: 6 16 2147 2131 1.38714 0 - 0.0109079
5: 7 16 2147 2131 1.18897 0 - 0.0109079
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon: return 1
5: /home/erwan/ceph/src/test/smoke.sh:18: run: return 1
Signed-off-by: Erwan Velu <erwan@redhat.com>
116cf759c8
will now hide all shadow trees(roots), so this is not applicable anymore
(actually it is misleading).
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
/bin/bash is a Linuxism. Other operating systems install bash to
different paths. Use /usr/bin/env in shebangs to find bash.
Signed-off-by: Alan Somers <asomers@gmail.com>
- stop running via make check
- add teuthology yamls to run them
- disable ceph_objecstore_tool.py for now (too slow for make check, and
we can't use vstart in teuthology via a package install)
- drop cephtool tests since those are already covered by other teuthology
tests
- leave a handful of (fast!) ceph-helpers tests for make check for minimal
integration tests.
Signed-off-by: Sage Weil <sage@redhat.com>
