Commit Graph

32387 Commits

Author SHA1 Message Date
Loic Dachary
381850b161 Merge pull request #1497 from dachary/wip-jerasure
remove gf-complete / jerasure sub modules

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-19 00:56:36 +01:00
Loic Dachary
e9eb641fc7 remove gf-complete / jerasure sub modules
and add the source files in the tree.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-19 00:28:17 +01:00
Sage Weil
8b77995f00 Merge pull request #1494 from dachary/wip-gitmodule-mirrors
build: add gf-complete/jerasure to gitmodule_mirrors
2014-03-18 15:59:26 -07:00
João Eduardo Luís
362d370dd8 Merge pull request #1125 from ceph/wip-mon-docs
wip-mon-docs: Better explain required number of monitors & how to troubleshoot a monitor

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>
2014-03-18 22:26:34 +00:00
Loic Dachary
4221e0dd25 build: add gf-complete/jerasure to gitmodule_mirrors
Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 22:04:13 +01:00
Sage Weil
f1e3bc9a9b doc/dev/cache-pool: describe the tiering agent
Signed-off-by: Sage Weil <sage@inktank.com>
2014-03-18 13:09:29 -07:00
Alfredo Deza
06bfdfc9dd Merge pull request #1491 from dachary/wip-submodules
erasure-code: jerasure and gf-complete moved to ceph namespace

Reviewed-by:  Alfredo Deza <alfredo.deza@inktank.com>
2014-03-18 14:07:09 -04:00
Loic Dachary
1ddae7e2cc erasure-code: jerasure and gf-complete moved to ceph namespace
for maintenance purposes

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 19:00:21 +01:00
Loic Dachary
fcbacff7f9 Merge pull request #1489 from dachary/wip-7757
erasure-code: disable SSE extensions

Reviewed-by: Christophe Courtaut <christophe.courtaut@gmail.com>
2014-03-18 15:47:11 +01:00
Loic Dachary
543c642b1a erasure-code: disable SSE extensions
Until jerasure-2 / gf-complete is able to detect them at runtime.

http://tracker.ceph.com/issues/7757 refs #7757

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 15:46:22 +01:00
Loic Dachary
bd12d376b8 Merge pull request #1488 from dachary/wip-jerasure-2
erasure-code: update jerasure v2 submodule

Reviewed-by: Christophe Courtaut <christophe.courtaut@gmail.com>
2014-03-18 15:45:49 +01:00
Loic Dachary
1c5411a75f erasure-code: update gf-complete v1 submodule
Point to https://bitbucket.org/dachary/gf-complete/branch/v1 which
aggregates the pending pull requests required by Ceph while they are
reviewed.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 11:39:43 +01:00
Loic Dachary
d5e38d4ab3 erasure-code: update jerasure v2 submodule
to include the latest compilation warning fixes.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 09:50:20 +01:00
Sage Weil
7bc9ade26b Merge pull request #1384 from dachary/wip-7599
upgrade jerasure to version 2

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 20:52:57 -07:00
Sage Weil
390f57b4cd Merge pull request #1486 from ceph/wip-cli-err
ceph.in: Better error on bad arg to 'tell'

Reviewed-by: Dan Mick <dan.mick@inktank.com>
2014-03-17 20:51:00 -07:00
Sage Weil
e26ad1351b Merge pull request #1479 from ceph/wip-rados-outfile
tools/rados: Allow binary file output of omap data

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 18:41:18 -07:00
Sage Weil
d0975d17fa Merge pull request #1480 from ceph/wip-remotebit-dump
mds: Fix remotebit::dump for less common types

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 18:39:48 -07:00
Sage Weil
27ed656919 Merge pull request #1477 from dachary/wip-erasure-code-profiles
replace properties with erasure code profiles

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 16:26:53 -07:00
Loic Dachary
eac224a4ab doc: update the erasure-code dev documentation
Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
b273011874 osd,mon: use profile instead of properties
The qa and functional tests are adapted to the new command prototype
requiring a profile instead of a list of properties. When possible the
implicit ruleset creation is used to simplify the test setup.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
8cf85f046e mon: add osd crush create-erasure functional tests
Cleanup the TEST_crush_rule_all function from redundant leftovers.
Explicitly test crush rule rm instead of implicitly.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
b85330eb1d osd: create the erasure-code ruleset in OSDMap::build_simple
Using the "default" profile, add the matching erasure-code ruleset.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
5c1db3b69b osd: create erasure_code_profile in OSDMap::build_simple
The "default" erasure_code_profile is set by OSDMap::build_simple using
the osd_pool_default_erasure_code_profile default configuration option.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
f8aa1edae8 mon: set the profile and ruleset defaults early
The poolstr is removed from the prepare_pool_crush_ruleset prototype
because it no longer decides for the default ruleset, if it is not
omitted by the caller of osd pool create.

If no profile
  profile = default
If no ruleset and profile is default
  ruleset = erasure-code
If no ruleset and profile is not default
  ruleset = the name of the pool

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
5774344756 mon: add crush ruleset name to osd pool create
The ruleset to be used for the new erasure coded pool was expected in
the properties, under the name crush_ruleset. It does not belong to the
erasure code profile and needs to be added to the prototype explicitly.

The crush ruleset name is added to the prototype of the prepare_new_pool
and prepare_pool_crush_ruleset methods.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
4f1b372e54 osd: use erasure code profile when building the PGBackend
The PGBackend::build_pg_backend() prototype is modified to add an
OSDMapRef which is used to get the profile stored in the pg_pool_t
and pass it to the erasure code plugin.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:42 +01:00
Loic Dachary
173e95800b mon: OSDMonitor use erasure_code_profile instead of properties
The prepare_pool_properties is replaced with the
parse_erasure_code_profile method which looks up the same content from
OSDMap::erasure_code_profiles instead of the argument vector.

The type and name substitution is applied in OSDMonitor and the
MonCommand prototypes for osd pool create and osd create-erasure are
updated.

The behavior is not modified.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:41 +01:00
Loic Dachary
063de51e88 osd: obsolete pg_pool_t properties with erasure_code_profile
The generic map properties is only used for erasure code and should be
specialized. It is obsoleted by a string that can be looked up in
OSDMap::erasure_code_profiles to retrieve the key/value pairs that will
be interpreted by the erasure code plugin.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:41 +01:00
Loic Dachary
04d2fd1003 mon: add the erasure-code-profile {set,get,rm,ls} MonCommand
"erasure-code-profile set" parses the key=value pairs given in argument
and stores them in OSDMap::erasure_code_profiles. The
"erasure-code-profile get" supports plain text display if the Formatter
is not set (or invalid).

  erasure-code-profile set myprofile a=b c=d

is stored as

  OSDMap::erasure_code_profile["myprofile"] = {"a":"b", "c":"d"}

"erasure-code-profile ls" displays a list of the profile names from
OSDMap::erasure_code_profiles

"erasure-code-profile rm" removes a profile, if it exists. Displays a
message and succeeds if it does not exist.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-18 00:23:31 +01:00
Sage Weil
fa1d957c11 mon/Paxos: commit only after entire quorum acks
If a subset of the quorum accepts the proposal and we commit, we will start
sharing the new state.  However, the mon that didn't yet reply with the
accept may still be sharing the old and stale value.

The simplest way to prevent this is not to commit until the entire quorum
replies.  In the general case, there are no failures and this is just fine.
In the failure case, we will call a new election and have a smaller quorum
of (live) nodes and will recommit the same value.

A more performant solution would be to have a separate message invalidate
the old state and commit once we have all invalidations and a majority of
accepts.  This will lower latency a bit in the non-failure case, but not
change the failure case significantly.  Later!

Fixes: #7736
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
2014-03-17 16:21:56 -07:00
Sage Weil
1048763e33 Merge remote-tracking branch 'gh/firefly' 2014-03-17 16:19:07 -07:00
Samuel Just
76beb5ce82 Merge pull request #1487 from ceph/wip-7738
os/FileJournal: return errors on make_writeable() if reopen fails

Reviewed-by: Samuel Just <sam.just@inktank.com>
2014-03-17 16:03:49 -07:00
Sage Weil
aed074401d os/FileJournal: return errors on make_writeable() if reopen fails
This is why #7738 is resulting in a crash instead of an error.

Signed-off-by: Sage Weil <sage@inktank.com>
2014-03-17 15:37:44 -07:00
Sandon Van Ness
f0944003cc Add file to store mirror location of module's.
Signed-off-by: Sandon Van Ness <sandon@inktank.com>
2014-03-17 12:15:32 -07:00
Sage Weil
ec362e8996 Merge pull request #1481 from ceph/wip-tmap2omap-warn
mds: avoid spurious TMAP2OMAP warning

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 09:49:54 -07:00
John Spray
a9f8a9e458 ceph.in: Better error on bad arg to 'tell'
Previously would get a rather enigmatic error:
  UnboundLocalError: local variable 'ret' referenced before assignment

Now give something sensible:
  ceph_argparse.ArgumentValid: Bad target type 'mds'

Also update a couple of the other catch-all exception handlers
so that they will let the (nicer) ArgumentError exception through
for humans to see instead of munging them into RuntimeErrors.

Signed-off-by: John Spray <john.spray@inktank.com>
2014-03-17 16:21:24 +00:00
Loic Dachary
1a451f2e03 mon: functional tests teardown must be run on error
otherwise daemons will keep running and interfere with tests happening
afterwards.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 16:48:53 +01:00
Loic Dachary
514b5e3ed5 mon: add helper to selection functions implementing tests
The call_TEST_functions will lookup SHARE_MON_TEST_* shell functions and
run them after creating a mon. Each is expected to clean up and avoid
interferences. It will also lookup TEST_* shell functions and include them
in setup/teardown to provide them with a clean environment.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 16:48:53 +01:00
Loic Dachary
e4b4b1f172 osd: OSDMap::erasure_code_profile accessors
For both the incremental and the OSDMap fields.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 16:48:43 +01:00
Loic Dachary
c4f8f26595 osd: add OSDMap::erasure_code_profile
A map of key=value properties to be interpreted by the erasure code
plugins. For instance profile["default"] = { "k": 2, "m": 1 }

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 16:48:30 +01:00
John Spray
2b9bd26749 mds: avoid spurious TMAP2OMAP warning
The message "one or more OSDs do not support TMAP2OMAP" was printed
incorrectly when zero OSDs were up (and therefore the feature was
absent).  Don't issue this prompt until at least one OSD is up.

Signed-off-by: John Spray <john.spray@inktank.com>
2014-03-17 15:42:38 +00:00
Sage Weil
35aed15f9b Merge pull request #1482 from ceph/wip-7611
ceph.in: do not allow using 'tell' with interactive mode

Reviewed-by: Sage Weil <sage@inktank.com>
2014-03-17 08:17:53 -07:00
Joao Eduardo Luis
e39c213c1d ceph.in: do not allow using 'tell' with interactive mode
This avoids a lot of hassle when dealing with to whom tell each command
on interactive mode, and even more so if multiple targets are specified.

As so, 'tell' commands should be used while on interactive mode instead.

Backport: dumpling,emperor

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
2014-03-17 14:44:50 +00:00
John Spray
b2af217b1d mds: Fix remotebit::dump for less common types
This was hitting an assertion when used on anything
other than a regular/symlink/dir.  Now covers
all types.

Signed-off-by: John Spray <john.spray@inktank.com>
2014-03-17 14:28:51 +00:00
John Spray
9d77ce170b tools/rados: Allow binary file output of omap data
Extends getomapval and getompaheader to take an output
file argument the same way 'get' does.  Removes misleading
'-o' line from usage(), there doesn't appear to be any
such
option.

Signed-off-by: John Spray <john.spray@inktank.com>
2014-03-17 13:56:26 +00:00
Loic Dachary
be319983e6 erasure-code: make libcommon include liberasure-code
So that it can be used in OSDMap which is part of libcommon.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 09:48:03 +01:00
Loic Dachary
e6d9066f94 erasure-code: add ostream to factory()
Modify calls to ErasureCodePlugin::factory() to provide the new
ostream argument.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 09:48:03 +01:00
Loic Dachary
de62572c61 erasure-code: remove dependency to the global context
Instead of relying on derr to display error messages, add them to an
ostream parameter given in argument to load() and factory(). The erasure
code convenience library no longer depends on the global context that is
indirectly referenced by debug.h

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 09:48:03 +01:00
Loic Dachary
bcc74879a1 common,erasure-code,mon: s/erasure-code-//
The parameters to erasure code do not need to be prefixed with the
erasure-code- string. There only are erasure-code parameters and the
prefix was originaly intended to desambiguate the erasure-code
properties, assuming that the properties map could be used for other
purposes.

Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 09:48:03 +01:00
Loic Dachary
d4d6ff0706 mon,osd,erasure-code: update copyright notices
Signed-off-by: Loic Dachary <loic@dachary.org>
2014-03-17 09:48:03 +01:00