1
0
mirror of https://github.com/ceph/ceph synced 2025-02-16 07:17:21 +00:00
Commit Graph

28762 Commits

Author SHA1 Message Date
Sage Weil
c4260fad29 osd/ReplicatedPG: drop unused issue_repop() arguments
These are cruft from the old parallelexec mode and can be removed.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:19:18 -07:00
Sage Weil
27ca5d2e06 osd/ReplicatedPG: drop dead code from parallelexec issue_repop
This is unused and references otherwise unused variables we can remove!

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:19:18 -07:00
Sage Weil
c9885e7487 osd/ReplicatedPG: generate one-off unique temp object names
Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:19:18 -07:00
Sage Weil
6cecd0db56 osd/osd_types: move coll_t::META_COLL definition
Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:19:18 -07:00
Sage Weil
17c5d765d7 os/FileStore: implement collection_move_rename
This is similar to a collection_add + collection_move sequence in that we
apply the same replay guards.  The difference is that we roll it up into
a single operation, change the filename, and make the omap content carry
over by calling DBObjectMap->clone (as there is no rename function or
collection awareness in the DBObjectMap).

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:19:18 -07:00
Sage Weil
ef7cffc34f os/ObjectStore: add collection_move_rename
Add method to move an object between collections *and* change its name.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-11 15:09:14 -07:00
Yehuda Sadeh
73e626ccea Merge pull request from kri5/master
rgw-admin: Adds --metadata-key option to help

Reviwed-by: Yehuda Sadeh <yehuda@inktank.com>
2013-09-11 09:01:18 -07:00
Christophe Courtaut
288bef3329 rgw-admin: Adds --metadata-key option to help
Signed-off-by: Christophe Courtaut <christophe.courtaut@gmail.com>
2013-09-11 17:49:04 +02:00
Yehuda Sadeh
0499948aad rgw: when failing read from client, return correct error
Fixes: 
When getting a failed read from client when putting an object
we returned the wrong value (always 0), which in the chunked-
upload case ended up in assuming that the write was done
successfully.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 20:10:03 -07:00
Greg Farnum
558d9fc956 osd: bind objecter messenger to cluster address
We don't want it binding to whatever willy-nilly, and as an OSD even
its "client" traffic should go on the cluster address.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 16:15:23 -07:00
Greg Farnum
1d1bf41e50 osd: name the objecter's messenger "ms_objecter" instead of "hbclient"
Bad copypasta on initial create?

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 16:15:12 -07:00
Greg Farnum
14c71ea39b Objecter: rename cancel_op -> cancel_linger_op
This makes it possible to converse about op_cancel and cancel_linger_op
without getting too confused.

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 16:15:03 -07:00
Gregory Farnum
383d8a199e Merge pull request from ceph/wip-6033-redirects
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 15:49:50 -07:00
Greg Farnum
ab2506dbfa test: update cache test since OSDs support redirects
Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:50 -07:00
Greg Farnum
76c343ba17 osd: implement basic caching policies in ReplicatedPG
Right now these are very basic and aren't as sophisticated as we
want them to end up, but we have a skeleton for where to put the
decision-making logic.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:50 -07:00
Greg Farnum
fbbe3ad122 Objecter: follow redirect replies from the OSD
If we get back a redirect reply, we clean up the Op's external references
and re-send using the target_oloc and target_oid. To facilitate this,
recalc_op_target() now only fills them in and overrides them with pool
cache semantics if they're empty.

Keep in mind that this is a pretty simple redirect formula -- the
Objecter will keep following redirects forever if that's what the OSDs
send back. The client is not providing any synchronization right now.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:49 -07:00
Greg Farnum
29133fdae5 Objecter: write a helper function to clean up ops that need to be retried
We have a little block to clean them up if we get back EAGAIN, but it's
actually leaking map references; we will also use this for redirects
from the OSDs.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:49 -07:00
Greg Farnum
ac1c92173c MOSDOpReply: add a redirect member
When present, clients must send the request to the location specified
by the redirect (by using the combine_with_locator() function on
request_redirect_t).
A separate mechanism must be used to ensure that clients see and respect
the redirect, as we do not bump up the minimum required version to
decode.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:49 -07:00
Greg Farnum
a45612c202 Objecter: add an Op::target_oid, and use it when submitting Ops
For now it's just a copy of base_oid, but soon we will allow it
to be overwritten for OSD-driven redirects.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:48 -07:00
Greg Farnum
79f02d6b8f Objecter: rename Op::oid -> Op::base_oid
Analagous to the oloc->base_oloc rename we did in
e2fcad09d9, we may specify a different
target name for a redirect. Rename the existing oid to base_oid to
avoid any confusion.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:48 -07:00
Greg Farnum
aeec0c6476 osd: create a new request_redirect_t
We'll use this so that the OSD can tell the Objecter to redirect a
request to a different object somewhere else.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-10 15:33:48 -07:00
Sage Weil
e3ca59c397 Merge pull request from ceph/wip-osd-pool-create-syntax
fix ceph tool (syntax error in command specification prevents anything from working)

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 15:04:42 -07:00
Josh Durgin
c24e170627 mon: fix syntax error in osd pool create args
Just missing a space between pgp_num and properties specification.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-09-10 14:47:31 -07:00
Sage Weil
dedfccd904 Merge pull request from dachary/master
autoconf: use $(UNITTEST_CXXFLAGS) in tests

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 09:17:25 -07:00
Sage Weil
277d266713 Merge pull request from dachary/wip-6113
mon: add key[=value] ... to osd pool create

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-10 09:15:42 -07:00
Loic Dachary
780954e21f autoconf: use $(UNITTEST_CXXFLAGS) in tests
As suggested by Roald van Loon
https://github.com/ceph/ceph/pull/538/files#r6260751

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 17:00:08 +02:00
Loic Dachary
647188c4e1 ErasureCodeJerasure: plugin
Create the class matching the string found in the
erasure-code-technique parameter, using the same strings are the
original {encoder,decoder}.c examples from Jerasure-1.2A. Registers
the plugin in ErasureCodePluginRegistry.

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:23 +02:00
Loic Dachary
e9e5391250 ErasureCodeJerasure: define technique Liber8tion
technique == "liber8tion"

       ErasureCodeInterface (abstract)
       |
       -> ErasureCodeJerasure (abstract)
          |
          -> ErasureCodeJerasureLiberation
             |
             -> ErasureCodeJerasureLiber8tion
                | == liber8tion

Derived from Liberation it overloads the parse and prepare methods.

parse : default to K=2 and packetsize = 8.
If any of the following constraints is not satisfied, revert to the
default:

  * K <= 8
  * packetsize must not be zero

prepare uses liber8tion_coding_bitmatrix

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
63867e941f ErasureCodeJerasure: define technique BlaumRoth
technique == "blaum_roth"

       ErasureCodeInterface (abstract)
       |
       -> ErasureCodeJerasure (abstract)
          |
          -> ErasureCodeJerasureLiberation
             |
             -> ErasureCodeJerasureBlaumRoth
                | == blaum_roth

Derived from Liberation it only overloads the prepare method to use
blaum_roth_coding_bitmatrix.

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
9a82010584 ErasureCodeJerasure: define technique Liberation
technique == "liberation"

parse : default to K=7, M=2 and W=7 and packetsize = 8.
If any of the following constraints is not satisfied, revert to the
default:

 * K > W
 * W > 2
 * W is a prime number
 * packetsize must not be zero
 * packetsize must be a multiple of sizeof(int)

pad_in_length : pad to a multiple of k*w*packetsize*sizeof(int)

prepare, jerasure_encode, jerasure_decode map directly to the matching
jerasure functions

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
530fb8a51f ErasureCodeJerasure: define techniques CauchyOrig and CauchyGood
The technique Cauchy has two variants:

       ErasureCodeInterface (abstract)
       |
       -> ErasureCodeJerasure (abstract)
          |
          -> ErasureCodeJerasureCauchy (abstract)
          |  |
          |  -> ErasureCodeJerasureCauchyOrig
          |     | == cauchy_orig
          |  -> ErasureCodeJerasureCauchyGood
          |     | == cauchy_good

ErasureCodeJerasureCauchy defines the prepare_schedule method to be used
by prepare method, which is the only one overloaded by
ErasureCodeJerasureCauchyOrig (calling cauchy_original_coding_matrix)
and ErasureCodeJerasureCauchyGood ( calling
cauchy_good_general_coding_matrix).

The schedule is retained for encoding and the bitmatrix for decoding.

parse : default to K=7, M=3, W=8 and packetsize = 8.

pad_in_length : pad to a multiple of k*w*packetsize*sizeof(int)

jerasure_encode, jerasure_decode map directly to the matching
jerasure functions

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
b70cb93a68 ErasureCodeJerasure: define technique ReedSolomonRAID6
technique == reed_sol_r6_op

parse : default to K=7 and W=8 . If W is not 8, 16 or 32, it
reverts to 8.

pad_in_length : pad to a multiple of k*w*sizeof(int)

prepare, jerasure_encode, jerasure_decode map directly to the matching
jerasure functions

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
65f1970a72 ErasureCodeJerasure: define technique ReedSolomonVandermonde
technique == reed_sol_van

parse : default to K=7, M=3 and W=8 . If W is not 8, 16 or 32, it
reverts to 8.

pad_in_length : pad to a multiple of k*w*sizeof(int)

prepare, jerasure_encode, jerasure_decode map directly to the matching
jerasure functions

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:22 +02:00
Loic Dachary
c8def86049 ErasureCodeJerasure: unit test common to all techniques
A typed unit test is defined and must run regardless of the technique.
When a new technique is derived from ErasureCodeJerasure, it is added
to the JerasureTypes typedef and the test will validate that:

  * it provides reasonable defaults for the technique specific
    parameters
  * it modifies the k, m and w to reasonable defaults depending
    on the imposed constraints ( for instance Liber8tion requires
    that w == 8 but the test sets it to 7 )
  * the encoding of K=2, M=2 produces 4 chunks, the first two
    of which contains the original buffer data showing the
    code is systematic
  * decoding when all 4 chunks are available indeed retrieves
    the original buffer content
  * decoding when the two data chunks are are missing indeed
    retrieves the original buffer content

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 16:46:10 +02:00
Loic Dachary
454c116d38 mon: add key[=value] ... to osd pool create
With the introduction of the erasure code pool, arguments to be
interpreted depending on the pool type must be introduced.
For instance the erasure code pool loads a plugin at run time will
use easure-code-k=10 to split each object in 10.

The arguments are described as

  name=properties,type=CephString,n=N,req=false,goodchars=[A-Za-z0-9-_.=]

If key=value it is stored in the new properties data member of pg_pool_t
as properties[key] = value, otherwise the value is the empty string.

The pg_pool_t version is bumped to 10 and the encode/decode methods
modified to take the properties into account. The
generate_test_instances method creates a two entries map, one of which
is the empty string to cover the case when no value is specified.

http://tracker.ceph.com/issues/6113 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 13:53:57 +02:00
David Zafman
8c76f3a0f9 crushtool: do not dump core with non-unique bucket IDs
Return -EEXIST on duplicate ID
BUG FIX: crush_add_bucket() mixes error returns and IDs
  Add optional argument to return generated ID

Fixes: 

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-09 21:50:32 -07:00
Sage Weil
bde2772b93 doc/release-notes: v0.67.3
Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-09 21:47:57 -07:00
Greg Farnum
397b4c209f osd: add empty() function to object_locator_t
Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-09 18:12:48 -07:00
Greg Farnum
0b24be281c MOSDOpReply: more const-correctness
get_map_epoch() is a non-reference getter -- mark it as const!

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-09 17:21:32 -07:00
Greg Farnum
82e40a7d86 ReplicatedPG: do not meaninglessly fill in the reqid on make_writeable() cloning
This reqid is used to fill in a map that is used for giving clients
the correct version on replayed ops, unless the reqid is blank (in
which case it doesn't go into the map). Indirect clones are not ops
that the client wants to observe and the version is incorrect right now,
so don't fill it in.
Note that this should not have actually caused any buggy behavior, because
the map would have simply been overwritten by the real requested event
a short time later (while still protected by locks and things). But it's
very confusing.

Signed-off-by: Greg Farnum <greg@inktank.com>
2013-09-09 17:21:32 -07:00
Loic Dachary
e435468725 ErasureCodeJerasure: base class for jerasure ErasureCodeInterface
The ErasureCodeJerasure class is derived from ErasureCodeInterface and
is meant to be derived to implement each jerasure technique (
Reed-Solomon, Cauchy ... ).

The parameters K ( number of data chunks ), M ( number of coding chunks
) and W ( word size ) are data members common to all techniques. The
technique data member is expected to be set to a string describing the
technique for debugging purposes.

minimum_to_decode_with_cost ignores the cost and calls minimum_to_decode.

minimum_to_decode returns the first K chunks or an error if there are
not enough. Since all codes are systematic, when all chunks are
available returning the first K allows for concatenation and is the best
choice.

The encode method converts bufferlist into char* as expected by the
jerasure functions. The padding of the incoming buffer depends on the
technique and is computed by the pad_in_length method. Encoding is done
with the jerasure_encode method.

The decode method converts the char* returned by the jerasure functions
into bufferlists to be consumed by the caller. The decoding is done by
the jerasure_decode method.

The to_int convenience method is used to convert parameters. The
is_prime convenience method will be used by some techniques to validate
parameters.

Immediately after creating an ErasureCodeJerasure derived object, the
init method must be called. It will call the parse method to interpret
the parameters required by the technique and set the k, m and w data
members. The prepare method is expected to compute the matrix ( and
schedule if necessary ) and store it in a data member. The init method
will be called while holding the ErasureCodePluginRegistry mutex. The
encode and decode methods will not be protected by a mutex and may be
called by different threads for the benefit of different placement
groups. They will not have any side effect on the object.

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 02:01:21 +02:00
Loic Dachary
9b4048aabc ErasureCodeJerasure: fix jerasure compilation
Add the imported jerasure-1.2A to the Makefile.am and fix the
compilation warnings.

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 02:01:21 +02:00
Loic Dachary
8ab29e951f ErasureCodeJerasure: import jerasure-1.2A
The files are copied verbatim from
http://web.eecs.utk.edu/~plank/plank/papers/Jerasure-1.2A.tar and a
section is added to the top level COPYING file to reflect the BSD
license.

https://github.com/dachary/ceph/tree/wip-5879 refs 

Signed-off-by: Loic Dachary <loic@dachary.org>
2013-09-10 02:01:21 +02:00
Sage Weil
661b377c28 os/FileStore: pass old + new object name to lfn_link
For now, the only caller leaves the name the same.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-09 16:55:21 -07:00
Dan Mick
7281354f72 Merge pull request from ceph/wip-osd-stat
osd: expose bytes used/avail via perf / asok
2013-09-09 16:42:06 -07:00
Sage Weil
deea63f7a8 osd: expose bytes used/avail via perf / asok
This values are already sent to the monitor.  Expose them via the admin
socket too so collectd/diamond/whatever can pick them up.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-09 16:36:22 -07:00
Sage Weil
63901a4f9f Merge pull request from ceph/wip-6230-workunit
qa: workunits: mon: crush_ops: test 'ceph osd crush move' 

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-09 16:31:28 -07:00
Joao Eduardo Luis
3bc618b7b4 qa: workunits: mon: crush_ops: test 'ceph osd crush move'
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
2013-09-10 00:29:19 +01:00
Sage Weil
a5a3510ea8 Merge pull request from ceph/wip-6230
mon: MonCommands: expect a CephString as 1st arg for 'osd crush move'

Reviewed-by: Sage Weil <sage@inktank.com>
2013-09-09 16:01:06 -07:00
Joao Eduardo Luis
7d3799fde1 mon: MonCommands: expect a CephString as 1st arg for 'osd crush move'
Fixes: 

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
2013-09-09 23:44:57 +01:00