Commit Graph

28807 Commits

Author SHA1 Message Date
Sage Weil
b0fee96078 Merge pull request #630 from dachary/master
osd: change warn_interval_multiplier to uint32_t

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-24 10:27:05 -07:00
Loic Dachary
1bce1f009b osd: change warn_interval_multiplier to uint32_t
to prevent overflow in OpTracker::check_ops_in_flight when
multiplying warn_interval_multiplier *= 2

Backport: cuttlefish, dumpling

http://tracker.ceph.com/issues/6370 fixes #6370

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-24 19:12:35 +02:00
Sage Weil
2cbe93b840 Merge pull request #628 from dachary/wip-erasure-fix-thread-test
ErasureCode: fix plugin loading threaded test

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-24 08:42:49 -07:00
Yehuda Sadeh
e1d7c32d8a Merge pull request #627 from dachary/master
rgw: fix rgw test to reflect usage change

Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
2013-09-24 07:38:36 -07:00
Loic Dachary
4bc18189b1 ErasureCode: fix plugin loading threaded test
The TEST_F(ErasureCodePluginRegistryTest, factory_mutex) was bugous and
only succeeded by chance. The sleep was on the factory constructor which
was never called. An erasure code plugin that hangs forever on load is
created instead.

The sleep_forever.detach is replaced by pthread_cancel to interrupt the
thread that hangs forever. If not, gtest will try to join the thread and
never exit.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-24 16:07:31 +02:00
Loic Dachary
e50343e442 rgw: fix rgw test to reflect usage change
otherwise src/test/cli/radosgw-admin/help.t fails when running make
check when run after a configure --with-radosgw

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-24 08:05:38 +02:00
Sage Weil
08c386f542 Merge pull request #588 from dachary/wip-6274
mon: unit tests to protect against some MonCommands.h typos
2013-09-23 16:42:26 -07:00
Sage Weil
286a699190 Merge pull request #566 from ceph/wip-purge-stray
Fixes for purging stray

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 16:16:48 -07:00
Gregory Farnum
6200eeca59 Merge pull request #591 from ceph/wip-misc
Reviewed-by: Greg Farnum <greg@inktank.com>
2013-09-23 15:48:04 -07:00
Greg Farnum
7741de0ba4 Merge branch 'master' of github.com:ceph/ceph 2013-09-23 15:41:14 -07:00
Sage Weil
1425119b9a Merge pull request #625 from ceph/wip-warn-pg
mon: warn when pg_num is too low or appears out of whack wrt the cluster size

Reviewed-by: Greg Farnum <greg@inktank.com>
2013-09-23 15:40:14 -07:00
Greg Farnum
73289b34b0 mds: be more careful about decoding LogEvents
We need to wrap the full decode section or we can abort the process
if there's an issue (which we may want to just skip by).

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 15:31:56 -07:00
Sage Weil
816cbbfea3 Merge pull request #619 from dachary/wip-erasure-pad
ErasureCode: optimize padding calculation 

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 15:14:22 -07:00
Loic Dachary
c32c51a3f3 ErasureCode: optimize padding calculation
Andreas-Joachim Peters pointed out that the ErasureCodeJerasure method
pad_in_length is very inefficient for no reason. It is replaced by the
alignment method and the calculation is inlined in the encode method.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-24 00:08:36 +02:00
Sage Weil
60e9e85461 vstart: set 'mon pg min per osd'
We need to tweak this since we create relatively few pgs per osd.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-23 14:57:23 -07:00
Sage Weil
befdcfd658 mon/PGMonitor: health warn if pool has relatively high objects/pg
If there is a pool that has a high objects/pg relative to the rest of the
cluster, warn, as this suggests this particular pool may have too few
PGs.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-23 14:57:22 -07:00
Sage Weil
a684ac8220 mon/PGMonitor: health warn if pg_num != pgp_num
Users need to adjust pg_num *and* pgp_num for split but may forget to do
both.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-23 14:56:48 -07:00
Sage Weil
dc7114e060 mon: warn if pg to osd ratio is too low
If there are not enough PGs relative to the number of in osds, warn.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-23 14:56:45 -07:00
Loic Dachary
97471e333e ErasureCode: coding style conformance
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:52:50 +02:00
Loic Dachary
bfd4db2525 ceph_argparse: unit tests for validate_command config-key
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
fa396d9461 ceph_argparse: unit tests for validate_command osd
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
c09d01c210 ceph_argparse: unit tests for validate_command mon
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
0ff003065c ceph_argparse: unit tests for validate_command mds
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
00d15049ea ceph_argparse: unit tests for misc validate_command
Contrary to all other classes, this series of command ( Monitor ) does
not have a common prefix.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
253e53f41a ceph_argparse: unit tests for validate_command auth
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
d3bf39e761 ceph_argparse: unit tests for validate_command pg
http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
1d54754c48 pybind: ceph_argparse unit tests foundations
The general idea is to have a series of commands, in the same order as
they show in mon/MonCommands.h, as if they were input to the ceph
client. For each command a valid combination is verified. And at least
one validation error is checked to produce a validation error. For
instance:

    ['pg', 'stat']

is a valid command and the validate_command function is expected to
return a value that is not None or {}. The command

    ['pg', 'stat', 'toomany' ]

is also given to validate_command to check that an error occurs when
an extra argument is given.

The TestArparse class implements a few methods to reduce the verbosity
of the tests. It does not provide many methods : only those that
significantly reduce the verbosity have been implemented. The drawback
of writing too many convenience methods is that they are more difficult
to read and maintain.

The signature dictionary is made a global variable so that
it is only extracted once for all classes. It is immutable.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:44 +02:00
Loic Dachary
f9c7bc697c pybind: catch EntityAddress missing /
If the / is missing in an EntityAddress, an ArgumentValid exception must
be raised so that it can be caught in the same way other argument
validation exceptions are.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
ccf2200841 mon: validate mon dump epoch as a positive integer
All other epochs are validated in the same way

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
8f0bb2f02f pybind: unit tests for ceph_argparse::parse_json_funcsigs
Run parse_json_funcsigs against the builtin commands found
in mon/MonCommands.h. Although it does not provide for a full
validation, it will detect some degenerate cases.

It is expected to raise if a space is missing at the end of an option
specification ( see https://github.com/ceph/ceph/pull/585 ) and this
case is tested. New tests of the same kind can be added whenever an
undetected error is found. Ideally a validation function is implemented
and it would be updated ( with an associated test ) when a new
pathological case is found.

The json string given to parse_json_funcsigs is obtained from
the support program get_command_descriptions.

The python-nose dependencies are added to the build requirements in
debian/control and ceph.spec.in because make check should always be run
at built time.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
96067e0784 .gitignore gtags(1) generated files
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
c16b54fd53 mon: get_command_descriptions support program
The get_command_descriptions function is not designed to be tested in
C++ because all the validation happens in pybind/ceph_argparse.py. The
get_command_descriptions program is designed to be used by python unit
tests as a mean to get a JSON dump of the content of mon/MonCommands.h

      get_command_descriptions --all

      {"cmd000":{"sig":["pg","stat"],"help": ... "avail":"cli,rest"}}

It also provides a way to reproduce and keep track of past errors
( typos etc. ) to ensure the python validation keeps catching it.

      get_command_descriptions --pull585

Add /get_command_descriptions to .gitignore so that

      git ls-files --exclude-standard --others

does not see it which is required for
f018d220f2/build-ceph.sh (L73)

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
e0b5697387 mon: convenience function to convert commands to json
The get_command_descriptions is added to Monitor.h and contains
the code previously inlined in Monitor::handle_command to implement
the get_command_descriptions command. It is intended for tests to
convert command descriptions into json, including error cases.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
9c9d7e6a32 autotools: set noinst_PROGRAMS
to be used by unit test support programs that do not need to be installed

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
ee33ff81f8 autotools: group test scripts in check_SCRIPTS
The check_SCRIPTS is added in Makefile-env.am to list the tests that do
not require compilation. The scripts listed in check-local and in the
TESTS variable use check_SCRIPTS instead.

The PYTHONPATH environment variable is added to Makefile-env.am and
includes the pybind directory so that python unit tests can load the
libraries from sources.

http://tracker.ceph.com/issues/6274 refs #6274

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>
Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-23 23:46:43 +02:00
Loic Dachary
0eb0c3e0f3 Merge pull request #626 from ceph/wip-usage
common/config: include --cluster in default usage message
2013-09-23 14:25:15 -07:00
Sage Weil
104277c836 Merge pull request #617 from dachary/wip-lfnindex
os: fix TestLFNIndex test warnings

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 14:11:36 -07:00
Sage Weil
157754b3a0 common/config: include --cluster in default usage message
Clean it up a bit too.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-23 14:10:36 -07:00
Sage Weil
679f520781 Merge pull request #624 from ceph/automake-fix-common
make: build common/secret only on linux

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 14:07:22 -07:00
Noah Watkins
27fb44b358 make: build common/secret only on linux
libkeyutils is only available on linux. before the automake refactoring
secret.c was compiled into rbd and ceph_mount targets which are linux
only targets. secret.c was moved to libcommon during the refactoring,
but the conditional compilation was lost.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2013-09-23 13:07:12 -07:00
Sage Weil
9adb73ae41 Merge pull request #618 from dachary/wip-jerasure-minimum-to-decode
ErasureCode: minimum_to_decode unit tests and optimization

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 10:53:54 -07:00
Sage Weil
5bc4610855 Merge pull request #605 from dachary/wip-erasure-code-plugin-test
erasure code plugin test
2013-09-23 10:27:47 -07:00
Sage Weil
10f291838e Merge pull request #603 from dachary/wip-erasure-code-example
erasure code example cleanup
2013-09-23 10:26:48 -07:00
Sage Weil
b683005c21 Merge pull request #607 from ceph/prctl-getname-test
code_env: use feature test for PR_GET_NAME support

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-23 10:14:07 -07:00
Sage Weil
6b745c9491 Merge pull request #616 from ceph/automake-flags
automake: fixup am_common_flags to avoid re-definitions

Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com>
2013-09-23 10:12:16 -07:00
Gregory Farnum
2299c4dbf2 Merge pull request #614 from ceph/wip-rados-model
Prevent copy_from against yourself, and fix issues with the testing framework around it

Reviewed-by: Greg Farnum <greg@inktank.com>
2013-09-23 09:55:32 -07:00
Sage Weil
10363daddb Merge remote-tracking branch 'gh/next' 2013-09-23 09:50:10 -07:00
Wido den Hollander
991139bc0d doc: When bootstrapping mon set the correct caps for client.admin
Otherwise client.admin will be privilege-less and the admin can't
access the cluster.
2013-09-23 13:52:51 +02:00
Loic Dachary
9dfac9edfe ErasureCode: minimum_to_decode unit tests and optimization
The minimum_to_decode function simply returns the available chunks if no
recovery is necessary.

Add unit tests covering all minimum_to_decode situations:

* trying to read nothing
* read a chunk if none are available
* reading a subset of the available chunks
* read a missing chunk if there is less than k chunks available.
* reading when a chunk must be recovered

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-22 11:02:39 +02:00
Loic Dachary
9442cfaa39 os: fix TestLFNIndex test warnings
Do not ignore system return value. Wrap emacs compile line to 80
characters.

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-22 10:46:23 +02:00