For testing injectargs a configuration option was changed that has side
effects on the cluster. It could introduce random failures later. It is
replaced with a configuration option that cannot have adverse side
effects on the cluster.
http://tracker.ceph.com/issues/9919Fixes: #9919
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
It is incorrect to append the content of CEPH_ARGS to the argument list
when running injectargs. For instance if
CEPH_ARGS='--log-file the.log' \
./ceph tell osd.0 injectargs --no-osd_debug_op_order
translates into
./ceph tell osd.0 injectargs --no-osd_debug_op_order \
--log-file the.log
it ends up changing the log file of osd.0 which is probably unintended.
Instead CEPH_ARGS is inserted before injectargs and it translates into:
./ceph tell osd.0 --log-file the.log \
injectargs --no-osd_debug_op_order
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
The arguments of injectargs being valid ceph arguments, they are.
consumed when the ceph cli calls rados.conf_parse_argv(). It can be
worked around by obsuring them as in:
ceph tell osd.0 injectargs '--osd_debug_drop_ping_probability 444'
where '--osd_debug_drop_ping_probability 444' is a single argument that
does not match any known argument. The trick is that it will be
evaluated again once it reaches the OSD or the MON and translated into
the expected list of arguments. Although it is clear once explained, it
is obscure and leads to strange combinations such as:
ceph tell osd.0 injectargs '--osd_debug_op_order '
(note the extra space at the end) to set boolean parameters. A better
workaround is to add a -- marking the end of the options as in:
ceph tell osd.0 -- injectargs --osd_debug_op_order
this one is unfortunately much less documented and the user does not
usually know the exact semantic of --, let alone where it should be
placed.
The simpler solution is to split the argument list in two if
"injectargs" is found. The arguments that show after the "injectargs"
argument is removed from the list of arguments until parsing is
complete. It implements the more intuitive syntax:
ceph tell osd.0 injectargs --osd_debug_op_order
and the other forms are still valid for backward compatibility.
http://tracker.ceph.com/issues/9372Fixes: #9372
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Suites run with CEPH_TEST_CLI_DUP_COMMAND=1, which will send a duplicate
command for every command issued with the 'ceph' tool. Behavior is to
get a reply from the command and then send a duplicate, looking for the
same outcome (guaranteeing idempotency of the operations). However, it
so happens that if you remove the entity's own key from the keyring and
you happen to be unlucky enough so that the client's connection gets
failed (we also run tests with connection failure injections), the
'ceph' tool won't be able to reconnect to the cluster to send the
duplicate command (as it's entity no longer exists in the cluster's
keyring).
We rewrite the test instead of resorting to ugly hacks to work around
this behavior, simply having a new 'role-definer' added by the existing
'role-definer' (which we weren't testing anyway, so bonus points for
that) and then have one removing the other (to test the procedure) and
finally using 'client.admin' to remove the last 'role-definer'.
Fixes: #9820
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
Wip client flock
Add support for file locking to the userspace client, and improve blocked-lock cancellation so that it doesn't remove locks that succeeded when racing.
Reviewed-by: Greg Farnum <greg@inktank.com>
Assuming they are more likely than others to leave OSD/MON in an
unstable state that could have undefined side effects on the tests
following it. A cleaner solution would be to run them in a separate
script that is run on an independent cluster.
http://tracker.ceph.com/issues/9700Fixes: #9700
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
We added MDS resetting code here a while back,
but the order of operations was such that a
"cluster up" was being run between a fail_all_mds
and the point at which we needed the map not to
be interfered with (testing setmap).
Also the new fs create/destroy cycles for testing
EC pool handling were missing calls to stop the
daemons before fs rm.
Signed-off-by: John Spray <john.spray@redhat.com>
It is expected for ceph tell to fail with ENXIO if the daemon it is
trying to join is not ready for some reason. This should be handled as a
transient error instead of a fatal error.
Add two shell functions to help with retry. They may prove useful if
other cases requiring a few retries show up.
http://tracker.ceph.com/issues/9655Fixes: #9655
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
test creating and entity with blank caps with and without '--force'
being specified. without '--force' they must fail with EINVAL as the
monitor will not be able to parse them.
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
We have variables with the same name that are being shared! We don't
hit any issues with it currently because the code just kind of works
even though that happens. Add a bit of new logic that relies on an
immutable return code (for instance) and we're in the woods.
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>
expect_false does not extend past the pipe and fails because the command
succeeds
introduced in f05c977bbc
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Keep the osd trash test to ensure it is a valid command but make it a
noop by giving it a zero argument (meaning thrash 0 OSD maps).
Remove the loops that were added after the command in an attempt to wait
for the cluster to recover and not pollute the rest of the tests. Actual
testing of osd thrash would require a dedicated cluster because it the
side effects are random and it is unnecessarily difficult to ensure they
are finished.
http://tracker.ceph.com/issues/9620Fixes: #9620
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
ceph --format plain osd find 1 (and metadata) are not implemented and
must fallback to the default (json-pretty).
http://tracker.ceph.com/issues/9538Fixes: #9538
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Add a trivial osd health test at the beginning of each group of
tests. When facing an intermittent failure, it is difficult to diagnose
if the cluster appears to be missing an OSD but there is no indication
as to when the OSDs were last up.
The tests are now only run after all OSDs are up.
These checks can be disabled with --no-sanity-check to allow running
some tests that have less requirements than running all the tests.
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
From the bash man page:
set -e exit immediately ... The shell does not exit ... if the
command's return value is being inverted with !
Add an explicit exit 1 where appropriate.
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
* Removing tiers from a base pool in use by CephFS is forbidden.
* Using CephFS pools as tiers is forbidden.
Signed-off-by: John Spray <john.spray@redhat.com>
Fixes two things:
* EC pools are now permissible if they have a cache overlay
* Pools are not permissible if they are a cache tier.
Fixes: #9435
Signed-off-by: John Spray <john.spray@redhat.com>
This workunit will be used by tests as a placeholder that always return
true. This is helpful in tests when a script from the qa/workunits
directory is mandatory but we do not care about testing anything. For
an example of how it can be used, check
https://github.com/ceph/ceph-qa-suite/pull/120
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Add tests to fail as soon as an unexpected condition is met in
test_mon_osd. Otherwise the actual error will be more difficult find in
the logs.
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Aside from being a bit odd to begin with, using stderr
was causing tests to fail because the output was polluted
by log output which is also on stderr.
Fixes: 9281
Signed-off-by: John Spray <john.spray@redhat.com>
The ceph_erasure_code_benchmark output is converted into a JSON series
suitable to display in HTML with the http://www.flotcharts.org/
library. A self contained copy of the HTML,JS,CSS files is included for
durability and can be used from the source tree with:
CEPH_ERASURE_CODE_BENCHMARK=src/ceph_erasure_code_benchmark \
PLUGIN_DIRECTORY=src/.libs \
qa/workunits/erasure-code/bench.sh fplot jerasure |
tee qa/workunits/erasure-code/bench.js
and display with:
firefox qa/workunits/erasure-code/bench.html
Signed-off-by: Loic Dachary <loic@dachary.org>
Expand the default suite to enumerate all cases that are relevant to the
current code base so that it is easier to consume. Namely it means
* iterating over object sizes of 4KB (what is used by default) and
1MB (what was previous benchmarked)
* grouping results in series that would make sense to plot to get the
behavior of a given technique for a series of K/M values and all
possible erasures.
Instead of specifying the iterations to run, set the size of the total
data set to be exercised and compute the iterations by dividing it by
the object size. Since the object size varies, it is impractical to
preset the number of iterations and get meaningful results.
The PARAMETERS environment variable is added to enable the caller to
inject --parameter jerasure-variant=generic, for instance.
The packets size is calculated based on the other parameters. The
options are limited when packets are small (4KB) and it would not make a
real difference to give control over it. The packet size is capped to
a maximum of 3100 bytes which is roughly what has been found to be an
optimal value for large packets (1MB).
Signed-off-by: Loic Dachary <loic@dachary.org>
Previously this test assumed no pre-existing
filesystem and no MDS running. Generalize it
to nuke any existing filesystems found before
running, so that you can use it inside a vstart
cluster that had MDS>0.
Signed-off-by: John Spray <john.spray@redhat.com>
A sample command to run the test on hadoop 2.x is
TESTDIR=/home/test HADOOP_HOME=/usr/lib/hadoop HADOOP_MR_HOME=/usr/lib/hadoop-mapreduce sh workunits/hadoop-wordcount/test.sh starting hadoop-wordcount test
Signed-off-by: rootfs <hchen@redhat.com>
`cephfs set_layout` was broken and is now deprecated
in favour of using xattrs for layout. Retire the
kclient-specific test.
Fixes: #8773
Signed-off-by: John Spray <john.spray@redhat.com>
Make sure gets and sets of tiering-specific variables succeed on tier
pools and fail on non-tier pools.
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
This reverts commit 29c33f0c05.
We don't need the debugging any more, and having two separate fsx runners
already caused one update-in-the-wrong-place issue.
Signed-off-by: Greg Farnum <greg@inktank.com>
If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 522174b066)
mon: OSDMonitor: 'osd pool' - if we can set it, we must be able to get it
Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage@redhat.com>
Add support to get the values for the following variables:
- target_max_objects
- target_max_bytes
- cache_target_dirty_ratio
- cache_target_full_ratio
- cache_min_flush_age
- cache_min_evict_age
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Avoid possibility that we create multiple OSDs do to retries by passing in
the optional uuid arg. (A stray osd id will make the osd tell tests a
few lines down fail.)
Fixes: #8728
Signed-off-by: Sage Weil <sage@inktank.com>
... that after a fs new on fresh pools, crash_replay_interval
is set to the default on the data pool.
Signed-off-by: John Spray <john.spray@redhat.com>
If the test is run against a cluster started with vstart.sh (which is
the case for make check), the --asok-does-not-need-root disables the use
of sudo and allows the test to run without requiring privileged user
permissions.
Signed-off-by: Loic Dachary <loic@dachary.org>
Accomodate changes:
* data and metadata pools no longer exist by default
* filesystem-using tests must use `fs new` to create
the filesystem first.
Signed-off-by: John Spray <john.spray@inktank.com>
Fail if 'rbd rm' fails - most probably it'd fail with "image still has
watchers" and in that case it's a bug in the kernel client which we do
want to notice. Also nuke the trap-based error handling - cleanup() is
half-baked and not really necessary here.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
Take advantage of the fact that 'rbd map' will now talk to udev and
output the device that got assigned by the kernel to the newly created
mapping. Drop the get_id() cruft, udevadm settle and chown calls.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
Commit 7dc93a9651 fixed an incorrect
behavior with the OSD's 'osd bench' value hard-caps. The test wasn't
appropriately modified unfortunately.
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
The test was a big sequence of commands being run and it has been growing
organically for a while, even though it has maintained a sense of
locality with regard to the portions being tested.
This patch intends to split the commands into functions, allowing for a
better semantic context and easier expansion. On the other hand, this
will also allow us to implement mechanisms to run specific portions of
the test instead of always having to run the whole thing just to test a
couple of lines down at the bottom (or have to creatively edit the test).
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
To provide coverage for the 'import' command
resulting in the same contents as before an
export/import cycle.
Signed-off-by: John Spray <john.spray@redhat.com>
As run_xfstests.sh does no longer accept test lists, wrap it to provide
an expunge file with a list of tests we want to skip for krbd runs.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
Back in 2013 xfstests were rearranged, which also changed the way
./check parses test lists. Catch up with those changes. Note that
tests can no longer be listed in ranges, we only accept individual
tests and test groups (e.g. -g quick).
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
run_xfstests.sh is going to be updated in the next commit to be able to
drive newer xfstests. Among other things, the new xfstests proper
doesn't support listing tests in ranges, which is what the qemu wrapper
(run_xfstests_qemu.sh) relies on. So keep a copy of the old
run_xfstests.sh around until the qemu vm image is regenerated and the
up-to-date exclusion list for that kernel is shaken out.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
Reweight all items within a subtree. For example, you might reweigh tall
OSDs within a rack to a new value.
Signed-off-by: Sage Weil <sage@inktank.com>
It was a noop because it was incorrectly using the variable n. Add a
test to protect against regression.
http://tracker.ceph.com/issues/8440Fixes: #8440
Reported-by: Geoffrey Hartz <hartz.geoffrey@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
When a pool is created with ceph osd pool create, the auid is not
inferred from the session auid and is set to zero. Add the
ceph osd pool set <pool> auid <int>
command to set it after it is created, and the matching get:
ceph osd pool get <pool> auid
Signed-off-by: Loic Dachary <loic@dachary.org>
Modified qemu-iotests workunit script to check for versions
that use the latest qemu (currently only Trusty). Limit the
tests to those that are applicable to rbd.
Fixes: 7882
Signed-off-by: Warren Usui <warren.usui@inktank.com>
Listing objects isn't reliable with cache pools; skip that part of the
test if we see that rbd has tiering enabled.
Signed-off-by: Sage Weil <sage@inktank.com>
There are several perils when splitting a cache pool:
- split invalidstes pg stats, which disables the agent
- a scrub must be manually triggered post-split to rebuild stats
- the pool may fill the OSDs during that period.
- or, the pool may end up beyond the 'full' mark and once scrub does
complete and the agent activate we may block IO for a long time while
we catch up with flush/evict
Make it a bit harder for users to shoot themselves in the foot.
Fixes: #8043
Signed-off-by: Sage Weil <sage@inktank.com>
Create a custom profile with ruleset-failure-domain=osd. (The default
ruleset-failure-domain=host won't do because this script assumes and
works only if all osds are on the same host.) While at it, set k and m
explicitly to avoid troubles in the future.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
OSDs that for some reason get behind on processing their op queue break
expect_alloc_hint_eq(), as it pokes the FS and not the journal. Fix it
by flushing the journal before proceeding with anything else.
Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
The qa and functional tests are adapted to the new command prototype
requiring a profile instead of a list of properties. When possible the
implicit ruleset creation is used to simplify the test setup.
Signed-off-by: Loic Dachary <loic@dachary.org>
A new module, s3_utilities.pm has been created. It contains
subroutines common to at least two of the workunits in this
directory. Code was moved here from the other pl files, and
some minor changes (paramers and scope changes) were needed.
Fixes: 7472
Signed-off-by: Warren Usui <warren.usui@inktank.com>
If I have to touch this again I will remove it. Ugh. This time,
ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-03-11_02:30:01-rados-firefly-distro-basic-plana/125922
hit NXIO a few lines down because one of the OSDs was still down.
Signed-off-by: Sage Weil <sage@inktank.com>
Added port (fixed value for right now in teuthology) to hostname.
Fixes: 7374
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
Signed-off-by: Warren Usui <warren.usui@inktank.com>
(cherry picked from commit 8200b8a025)
- fix the wait check for osds to come back up
- make sure they get marked back in, too
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
'rados cppool' copies the contents but that doesn't make the destination
pool an unmanaged snaps pool. Therefore, we must get an ENOTSUP when
we try to remove an unmanaged snap from a not-unmanaged pool.
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
This wreaks havoc on our QA because it marks osds up and down and then
immediately after that we try to scrub and some osds are still down.
Adjust the CLI test to wait for all OSDs to come back up after thrashing.
Signed-off-by: Sage Weil <sage@inktank.com>
Prevent creation of buckets of type 0 ('osd', 'device', etc.), as they
will confusing the mapping algorithm.
Signed-off-by: Sage Weil <sage@inktank.com>
The cache pools will throttle when they reach the target max size, so it
is important to make the administrator aware when they approach that point.
Unfortunately it is not particularly easy to efficiently keep track of
which PGs have hit their limit and use that for reporting. However, it
is easy to raise a flag when we start to approach the target for the
entire pool, and that sort of early warning is arguably more useful
anyway.
Trigger the warning based on the target full ratio. Not when we hit the
target, but when we are 2/3 between it and completely full.
Implements: #7442
Signed-off-by: Sage Weil <sage@inktank.com>
This is a friendlier interface for setting up a cache tier with some
reasonable defaults (defined via config options). This will simplify
the user experience and documentation.
Signed-off-by: Sage Weil <sage@inktank.com>
In general, users should not use non-empty pools as new tiers or else
things can behave strangely:
- the data sets are unrelated behavior will be... strange.
- if the cache pool is not "new" and does not do the OMAP flag, the OSD
will not know not to flush omap objects to an EC base tier
- probably other random stuff I'm forgetting
Allow a user to shoot themselves in the foot with --force-nonempty.
Implements: #7457
Signed-off-by: Sage Weil <sage@inktank.com>
We would like to get the hit set parameters: hit_set_type |
hit_set_period | hit_set_count | hit_set_fpp via OSDMonitor
Signed-off-by: Kai Zhang <zakir.exe@gmail.com>
comment out erasure pool related tests when an OSD is involved because
it does not work yet. See http://tracker.ceph.com/issues/7360.
Signed-off-by: Loic Dachary <loic@dachary.org>
