Commit Graph

31705 Commits

Author SHA1 Message Date
Samuel Just
40bdcb8850 osd/,mon/: add (up|acting)_primary to pg_stat_t
We use pg_stat_t information to determine pg create targeting.

Fixes: #7481
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
2014-02-19 14:14:18 -08:00
Josh Durgin
72f260e07c Merge pull request #1256 from ceph/wip-librados-ops-and-rvals
librados c api for object operations and a few bug fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Sage Weil <sage.weil@inktank.com>
2014-02-19 11:18:57 -08:00
Samuel Just
314116e250 PG: insert into stray set if !us_up() && !is_acting()
This is simpler than checking actingbackfill
since it may not yet be filled in.

Fixes: #7470
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
2014-02-18 22:20:40 -08:00
Sage Weil
a33874d036 Merge pull request #1257 from ceph/wip-agent
osd: a few agent fixes

Reviewed-by: Samuel Just <sam.just@inktank.com>
2014-02-18 21:10:04 -08:00
Sage Weil
a5c5d924be osd/ReplicatedPG: EOPNOTSUPP on hit_set_get on non-replicated pools
We need to use the async read stuff for this to work.

Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 21:05:34 -08:00
Sage Weil
a40cd5075c osd/ReplicatedPG: load older HitSets into memory
If our evict_mode is non-idle, load older HitSets into memory in the agent
work thread.

Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 21:05:34 -08:00
Sage Weil
0af73755fa osd/ReplicatedPG: strengthen agent_work assertions
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 21:05:34 -08:00
Samuel Just
1d907c60d7 OSD: consider up/acting primary for new intervals
Both acting_up_affected and start_peering_interval need
to consider primary changes as well as acting/up changes.

Fixes: #7469
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
2014-02-18 19:46:33 -08:00
Josh Durgin
1ff0eefeb1 Merge pull request #1259 from ceph/wip-6677
rgw: add compatibility for MultipartUpload

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 15:38:34 -08:00
Josh Durgin
a2c195d85b Merge pull request #1258 from ceph/wip-rgw-gc
radosgw-admin: gc list --include-all

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 15:37:58 -08:00
Yehuda Sadeh
b0dcc798e8 radosgw-admin: gc list --include-all
A new option to the radosgw-admin gc list command that dumps *all* gc
entries, and not just the expired ones. This is useful for debugging.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-02-18 15:43:44 -08:00
Mike Lundy
609f4c5671 Throw a Timeout exception on timeout.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 13:25:49 -08:00
Josh Durgin
c5d3bdf7fa Merge pull request #1214 from ceph/wip-rgw-manifest-2
Wip rgw manifest 2
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 13:07:53 -08:00
Yehuda Sadeh
197544137e dencoder: fix for new rgw manifest code
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-02-18 13:11:35 -08:00
Yehuda Sadeh
b3ce18891f cls/rgw: fix debug output
print the correct error value

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-02-18 13:11:35 -08:00
Yehuda Sadeh
3fb6e25846 test/rgw: manifest unitest
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-02-18 13:11:35 -08:00
Yehuda Sadeh
da64bf70be rgw: scalable manifest object
Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
 - head: the head part of the object, which is the part that contains
   the first chunk of data. An object might not have a head (as in the
   case of multipart-part objects.
 - stripe: data portion of a single rgw object that resides on a single
   rados object.
 - part: a collection of stripes that make a contiguous part of an
   object. A regular object will only have one part (although might have
   many stripes), a multipart object might have many parts. Each part
   has a fixed stripe size, although the last stripe of a part might
   be smaller than that. Consecutive parts may be merged if their stripe
   value is the same.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-02-18 13:11:34 -08:00
Josh Durgin
7a019b386c test/librados: remove unused lines and fix return code for cmp guards
Now that return values are actually set by the osd and client, fix up
the tests that were checking the old behavior.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
bfbfcd6ead ReplicatedPG: fix successful write condition
ctx->modify no longer implies that the operation is a write,
if it ever did. These days op->may_write() is checked reliably
on the OSD, so just use that instead.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
18f08cd9c0 ReplicatedPG: clear osd op reply output for writes
Since the vector of OSDOps used by the reply is the same
as the one processed by do_osd_ops() now, any output data
needs to be cleared for writes. To be compatible with current behavior,
allow writes that aren't applying anything or have failed to return data
still.

Add a new parameter to the MOSDOpReply constructor to determine
whether the output data should be cleared. Clear it for successful
writes, and remove a redundant result < 0 -> result > 0 check in the
process.  This was caught by ceph_test_cls_hello and its
writes_dont_return_data method.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
1d661ca972 ReplicatedPG: set individual osd op rvals
This takes care of things that can fail before transaction is
executed, like omap comparison. Getting individual rvals from a
transaction requires more refactoring, so I'm leaving it for a later
cleanup.

Fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
7743698907 ReplicatedPG: don't copy the ops vector into OpContext
There's no need to copy it, and if we refer to the same vector from
the original MOSDOp, we can modify rvals for individual ops, since the
MOSDOp ops vector is copied to the MOSDOpReply when it is constructed.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
e550ad7915 ReplicatedPG: avoid undefined behavior in xattr comparison
Reading past the end of a pointer returned by string.data() in c++98
is undefined. While we're fixing this, also allow comparison of xattrs
containing null bytes.

Fixes: #7250
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
4b229f71ca qa: add librados c object operations tests to librados test script
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
7a760337e2 packaging: add test programs for librados c object operations
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
bac7bb6480 librados: bump version number
This covers the many recent changes leading up to the firefly release,
including the c object operations api.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
995fea9a44 librados: add omap object operations to the c api
Create an iterator type similar to XattrIter for returning results.
It just wraps the std::map to preserve sorting.

When getting omap keys or values, use an extra callback to set up the
iterator, and default optional parameters to the empty string.

Since it's hard to test read or write omap ops in isolation, combine
them all in the c read ops test case.

One difference between this and the c++ api is that omap_cmp only
allows one assertion per sub-op, for simplicity of the
interface. Multiple omap_cmp operations can still be added to the same
op to get the same effect.

Fixes: #7194
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
242a6c8523 librados: add read xattr functions for c object operations
Only implement string xattr comparison by since integer xattr
comparison assumes ceph-encoded integers in little-endian 64 bit
format, which should not be exposed to librados users.

Add an extra callback for getxattr so that we can initialize the
iterator.

Fixes: #7193
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
f68b706e5e Objecter: add a way to chain callbacks
The librados C api needs to do extra things like converting c++ data
structures or setting lengths, but some objecter operations already
have out_handlers that librados shouldn't override.

Add a method that chains contexts together by using a new context that
calls both the original and the newly added one.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
2543cde793 librados: add exec to the c object operations api
Add two versions: one that allocates a buffer of the appropriate
length for the user, but relies on the user to free it, and one that
uses a user-supplied buffer but may fail if it is too short.

Reuse the bufferlist -> buffer conversion context added for reads into
the user supplied buffer.

The librados-allocated buffer can be handled just like librados
allocated buffers used by the various command functions, so just reuse
do_out_buffer() for them.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
0dc4309b08 librados: add read to c object operation api
Do the usual bufferlist to buffer conversion in a callback from the
objecter before the librados user gets called.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:33 -08:00
Josh Durgin
6503f21a26 Objecter: add a context argument to ObjectOperation::read()
The c api will need this to be able to set the output buffer length
and potentially copy from bufferlist to buffer.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
6094e437ba librados: add simple atomic read operations to c api
stat, assert_exists, and set_flags are simple and need no extra
infrastrucutre.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
770942aa9c librados: add exec to the c write operations api
Nothing special needed here, just copying the input buffer and passing
things through. Don't allow output data since it's not usually
available for writes and unreliable when it is.

Fixes: #7195
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
bc03fbb409 librados: add individual op flags for c write operations
Move flag validation to a static function so it can be shared with the
c++ api. Refer to the new C constants from the c++ api so that it's
easy to keep them in sync.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
4f4121ae19 librados: add global op flags to the c api
The c++ api did not include these in the sync versions of the calls,
so add a flags argument to the IoCtxImpl functions for those, with
default arguments to avoid changing the many many callers.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
7f1f0b550f librados: remove constant for CMPXATTR_NOP
This constant was just added in the public header to match the
operations in rados.h, and was not included in a stable release yet.
NOP is misleading, since it's actually uninterpreted anywhere, and
using it would always return EIO from the OSD. It's not particularly
useful, so just remove it. Adding a cmpxattr with FAIL_OK is
equivalent.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
50ca88a166 librados: add some comments about constants in librados.h
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
e255bf5813 librados: add section docs for object operations
The doxygen end section of watch notify was accidentally moved when
these were added, so fix that at the same time.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
c61ba410eb test/librados: add ObjectReadOperation tests
These will help prevent regressions and demonstrate what should be
expected by the c api.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
5276e3cc08 test/librados: refactor setup/teardown in tiering tests
These tests can either use the standard RadosTest, with one pool, or a
new test case with two pools.

The new test case has to copy a couple lines in static methods
(working around lack of virtual static methods is more complex than
it's worth here), but the rest is straight forward. Rename
base_pool_name and base_ioctx to pool_name and ioctx to match the
member variables of the parent class RadosTest.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
ae072ac2fc test/librados: use standard rados test case for snapshot tests
Split these into two categories, self-managed and pool snapshots,
since they are mutually exclusive for a single pool. Remove snapshots
and objects at the end of tests, regardless of pass/fail, so the same
pool can be reused.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
5497e1cbd3 test/librados: use a test case instead of duplicating code everywhere
These tests don't have any special requirements on their ioctx and
pools, so they can also run much faster by using different namespaces
per test instead of new pools. For the few tests that do depend on
namespace, reset the ioctx to the default namespace.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
c6d8d0ef26 test/librados/cmd: remove unnecessary pool creation
Several tests here don't use the pool at all.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
d3c6f1712f test/librados: use connect_cluster_pp() instead of duplicating it
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
8d4a5fd5d6 test/librados: add a connect_cluster() helper
This mirrors the c++ version, connect_cluster_pp(), and removes the
same code from create_one_pool().

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
9630f2f02d test/librados: create general test case classes
Using a test case allows us to remove a boatload of boilerplate code
in all the tests, and focus them more on what they're actually
testing.

Create one for C tests, and one for C++ tests, with the same
functionality:

- create a pool when the test case starts
- between individual tests, create an ioctx and set its namespace uniquely
- delete objects from the default namespace during individual test teardown
- delete the pool only when the whole test case is finished

In gtest, a test case is the whole set of tests declared as part of
the same class using TEST_F(). Many tests create and delete individual
pools, but this is unnecessary for independent operation in most
cases, since we can use namespaces for that now. Since pool creation
and deletion dominates test run time, using these test cases makes
running many of the tests much faster.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
6273ba4a9f test/librados: move test.cc into its own library
This way it doesn't have to be compiled many times, and it's easier
to add new general functionality to new files without adding those
.cc files to every test program.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:31 -08:00
Josh Durgin
abca34aa59 Objecter: keep ObjectOperation rval pointers for writes
Just before sending an op, prepare_mutate_op() is called, creating a
new Op. prepare_read_op() already copied over all the out-params
correctly, but for write operations the individual op return value
pointers were not copied, so they would not be filled in. With this
fixed, librados users can get the per-op return codes again.

Partially fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:31 -08:00
Gregory Farnum
393a202198 Merge pull request #1227 from ceph/wip-7394
Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>
2014-02-18 12:34:29 -08:00