If the cluster dies during the rados bench, the maximum running time is
no more considered and all emitted aios are pending.
rados bench never quits and the global testing timeout (3600 sec : 1
hour) have to be reach to get a failure.
This situation is dramatic for a background test or a CI run as it locks
the whole job for too long for an event that will never occurs.
This ideal solution would be having 'rados bench' considering a failure
once the timeout is reached when aios are pending.
A possible workaround here is to put use the system command 'timeout'
before calling rados bench and fail if rados didn't completed on time.
To avoid side effects, this patch is doubling rados timeout. If rados
didn't completed after twice the expected time, it have to fail to avoid
locking the whole testing job.
Please find below the way it worked on a real test case.
We can see no IO after t>2 but despite timeout=4 the bench continue.
Thanks to this patch, the bench is stopped at t=8 and return 1.
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon: timeout 8 rados -p foo bench 4 write -b 4096 --no-cleanup
5: hints = 1
5: Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 4 seconds or 0 objects
5: Object prefix: benchmark_data_mr-meeseeks_184960
5: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
5: 0 0 0 0 0 0 - 0
5: 1 16 1144 1128 4.40538 4.40625 0.00412965 0.0141116
5: 2 16 2147 2131 4.16134 3.91797 0.00985654 0.0109079
5: 3 16 2147 2131 2.77424 0 - 0.0109079
5: 4 16 2147 2131 2.0807 0 - 0.0109079
5: 5 16 2147 2131 1.66456 0 - 0.0109079
5: 6 16 2147 2131 1.38714 0 - 0.0109079
5: 7 16 2147 2131 1.18897 0 - 0.0109079
5: /home/erwan/ceph/src/test/smoke.sh:55: TEST_multimon: return 1
5: /home/erwan/ceph/src/test/smoke.sh:18: run: return 1
Signed-off-by: Erwan Velu <erwan@redhat.com>
The wait_for_clean() is using the default timeout aka 300sec = 5mn.
wait_for_clean() is trying to find a clean status within that timeout
_or_ reset its counter if any progress got made in between loops.
In a case where the cluster is sane, the recovery should be made in
shorter than 5mn but it the cluster died, waiting for 5mn for nothing is
unefficient.
This patch is about defining a custom timeout for a wait_for_clean() not
to wait much more that 1m30 (90sec). If no progress is made in that
period, there is very few chance this will read the a valid state
anyhow.
Signed-off-by: Erwan Velu <erwan@redhat.com>
This commit adds the config options stored by the MON database to the
configuration documentation page.
One can filter for these config options by setting the 'Source' filter
to 'mon' on the configuration documentation page.
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
Since we only support Jewel and later releases, which both support
object-map and fast-diff, enabling/disabling object-map should always
enable/disable fast-diff.
Signed-off-by: Mao Zhongyi <maozy.fnst@cn.fujitsu.com>
mon: add 'osd destroy-new' command that only destroys NEW osd slots
Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
ceph-volume may run into a problem and want to clean up, but we do not
want to give it blanket access to the 'osd destroy' command. Instead,
make an 'osd destroy-new' that can only create new OSDs (ones that are
in the process of being created but have never booted yet).
Signed-off-by: Sage Weil <sage@redhat.com>
If there is a stray clone (one that does not appear in the SnapSet) and
we do any sort of recovery on it the OSD will crash. Log an error instead
but continue.
This addresses a problem where a cluster has both (1) an unexpected clone
and (2) the clone is not present on all replicas. Doing repair on that
PG will both not fix the unexpected clone and also cause the remaining
OSDs to crash trying to recover it.
Include a test.
Fixes: https://tracker.ceph.com/issues/24396
Signed-off-by: Sage Weil <sage@redhat.com>
In EC recovery read, if the object's attrs read failed or with errors, we erase the attrs we have read and
try to read it again from left shards. This will make the primary osd get the object's attrs correct and
avoid assert.
Signed-off-by: xiaofei cui <cuixiaofei@sangfor.com>
This reverts commit 886606bfd7.
Signed-off-by: David Zafman <dzafman@redhat.com>
Conflicts:
qa/standalone/scrub/osd-scrub-repair.sh (manually made equivalent changes)
mgr/dashboard: controller infrastructure refactor and new features
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
pg sends pg-stats to mgr every 5 seconds, so we cannot check for the
number of pgs right after creating the pool, at that moment, the number
of pgs could be 0, that's why manger.wait_for_clean() returns right
away, and leaves us with 0 pgs: the pgs serving the pool are still being
created. that's why `manager.get_num_active_clean()` returns `0`
sometimes. so, we should force osd to flush their stats to mgr, and wait
until the pg stats converages.
Fixes: http://tracker.ceph.com/issues/24321
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/22230/head:
test/cli-integration/rbd: fixed unit formatting
qa/workunits/rbd: fix unit formatting tests
librbd: clones should not inherit implicit feature bits from parent
Reviewed-by: Mykola Golub <mgolub@suse.com>
mon/MgrMonitor: change 'unresponsive' message to info level
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
We generate a MGR_DOWN health warning at the appropriate points; having
this at WRN level just triggers failed teuthology runs but doesn't much
value for the user.
Clear out teuthology whitelisting for this message.
Fixes: http://tracker.ceph.com/issues/24222
Signed-off-by: Sage Weil <sage@redhat.com>
rgw: Do not modify email if argument is not set
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
If the bucket is empty or does not have weight-set weights yet, avoid
crashing when populating the parent bucket.
Fixes: http://tracker.ceph.com/issues/23386
Signed-off-by: Sage Weil <sage@redhat.com>
* so we just rely on a single build system instead of two of them, the
other place we use cmake is cmake/modules/BuildRockDB.cmake.
* disable gflags when building rocksdb, it's optional and does not help
in the sense of testing librados support.
* disable prompts when installing on debian, to silence warnings like:
debconf: unable to initialize frontend: Dialog
* drop --force-yes option, as it is deprecated, and is replaced with
--allow-downgrades, --allow-remove-essential,
--allow-change-held-packages, but none of them apply in our case.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Just make caller happy. there is no easy way to support timeout.
Signed-off-by: Yan, Zheng <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24053
