Commit Graph

31717 Commits

Author SHA1 Message Date
Josh Durgin
e255bf5813 librados: add section docs for object operations
The doxygen end section of watch notify was accidentally moved when
these were added, so fix that at the same time.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
c61ba410eb test/librados: add ObjectReadOperation tests
These will help prevent regressions and demonstrate what should be
expected by the c api.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
5276e3cc08 test/librados: refactor setup/teardown in tiering tests
These tests can either use the standard RadosTest, with one pool, or a
new test case with two pools.

The new test case has to copy a couple lines in static methods
(working around lack of virtual static methods is more complex than
it's worth here), but the rest is straight forward. Rename
base_pool_name and base_ioctx to pool_name and ioctx to match the
member variables of the parent class RadosTest.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
ae072ac2fc test/librados: use standard rados test case for snapshot tests
Split these into two categories, self-managed and pool snapshots,
since they are mutually exclusive for a single pool. Remove snapshots
and objects at the end of tests, regardless of pass/fail, so the same
pool can be reused.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
5497e1cbd3 test/librados: use a test case instead of duplicating code everywhere
These tests don't have any special requirements on their ioctx and
pools, so they can also run much faster by using different namespaces
per test instead of new pools. For the few tests that do depend on
namespace, reset the ioctx to the default namespace.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
c6d8d0ef26 test/librados/cmd: remove unnecessary pool creation
Several tests here don't use the pool at all.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
d3c6f1712f test/librados: use connect_cluster_pp() instead of duplicating it
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
8d4a5fd5d6 test/librados: add a connect_cluster() helper
This mirrors the c++ version, connect_cluster_pp(), and removes the
same code from create_one_pool().

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
9630f2f02d test/librados: create general test case classes
Using a test case allows us to remove a boatload of boilerplate code
in all the tests, and focus them more on what they're actually
testing.

Create one for C tests, and one for C++ tests, with the same
functionality:

- create a pool when the test case starts
- between individual tests, create an ioctx and set its namespace uniquely
- delete objects from the default namespace during individual test teardown
- delete the pool only when the whole test case is finished

In gtest, a test case is the whole set of tests declared as part of
the same class using TEST_F(). Many tests create and delete individual
pools, but this is unnecessary for independent operation in most
cases, since we can use namespaces for that now. Since pool creation
and deletion dominates test run time, using these test cases makes
running many of the tests much faster.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:32 -08:00
Josh Durgin
6273ba4a9f test/librados: move test.cc into its own library
This way it doesn't have to be compiled many times, and it's easier
to add new general functionality to new files without adding those
.cc files to every test program.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:31 -08:00
Josh Durgin
abca34aa59 Objecter: keep ObjectOperation rval pointers for writes
Just before sending an op, prepare_mutate_op() is called, creating a
new Op. prepare_read_op() already copied over all the out-params
correctly, but for write operations the individual op return value
pointers were not copied, so they would not be filled in. With this
fixed, librados users can get the per-op return codes again.

Partially fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-02-18 12:34:31 -08:00
Gregory Farnum
393a202198 Merge pull request #1227 from ceph/wip-7394
Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>
2014-02-18 12:34:29 -08:00
Samuel Just
43b5cf977c Merge remote-tracking branch 'upstream/wip-ecbackend-for-review'
Reviewed-by: Sage Weil <sage@inktank.com>
2014-02-18 10:56:12 -08:00
Sage Weil
45a782ff56 Merge pull request #1255 from ceph/wip-cache-perf
osd: add perfcounters for new cache and agent events

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
2014-02-18 08:22:17 -08:00
Sage Weil
4bee6ff1bc osd/ReplicatedPG: clean up agent skip debug output
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:03 -08:00
Sage Weil
d1a185bd4b osd: l_osd_agent_{wake,skip,flush,evict}
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:03 -08:00
Sage Weil
dbec1096f4 osd: l_osd_tier_[dirty|clean]
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:03 -08:00
Sage Weil
150e87a163 osd: l_osd_tier_whiteout
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:03 -08:00
Sage Weil
2d5371ddaf osd: l_osd_tier_evict
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:02 -08:00
Sage Weil
c45a477efc osd: l_osd_tier_[try_]flush[_fail]
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:02 -08:00
Sage Weil
4b0a0a1023 osd: l_osd_copyfrom
Signed-off-by: Sage Weil <sage@inktank.com>
2014-02-18 04:51:02 -08:00
Samuel Just
5e727bfe1a RadosModel: only output if missing header is actually a problem
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:17 -08:00
Samuel Just
11f288e5d5 Objecter: track primary explicitly to detect changing primaries
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:17 -08:00
Samuel Just
bc31c4b32a ReplicatedPG: add some debugging if waiting_for_commit is non-empty
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
b90584a837 osd/: instantiate the right backend based on pool
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
792f0a85d0 ECBackend: flesh out ECBackend implementation
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
45b6c59e5b osd/ECTransaction: add type encapsulating EC transaction
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
33c3830158 osd/ECUtil: add ec encode/decode util helpers
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
7731a9490c ObjectStore: add bufferlist/string getattr
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
258af5ebe6 buffer: allow caller to initialize bufferhash state
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
ae827cfe4d osd_types: add swap, claim_append to ObjectModDesc
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
7772014a4a OSD,PG: pass through EC messages
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:16 -08:00
Samuel Just
6cd64a507d messages,osd: add EC messages and associated types
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
e8820ac42a src/osd/: add ECBackend stubs
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
0661d8025a PGBackend: make getattrs virtual
This way, the backend can choose to filter private attrs.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
ca323ff627 ReplicatedPG: fix size checking for ECBackend pools
Allow the backend to adjust the logical object size before
checking against the ondisk size.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
510585b795 PGBackend/ReplicatedBackend: factor out logical to ondisk size mapping
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
26745ca899 PGBackend/ReplicatedBackend: move the backend agnostic code into PGBackend for scrub
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
1835c29f6d ReplicatedBackend: factor out be_deep-scrub
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
f0b69b384b osd/: replace bufferptr with bufferlist for attrs in recovery
Now we won't have to keep converting back and forth!

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:15 -08:00
Samuel Just
0b180e6a62 PG::discover_all_missing: request a FULLLOG
Otherwise, we cannot correct for divergent objects in the missing
set.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
84e2f39c55 PG,ReplicatedPG: Generalize missing_loc for ECBackend
Prior to EC pools, unfound => missing.  Now, unfound (unreadable,
really) is dependent on the PGBackend requirements for reconstituting
an object.  This also means recovering an object missing on a replica
but not the primary requires tracking the missing_loc set.

Thus, rather than maintaining missing_loc only for objects missing
on the primary, the MissingLoc structure will track all missing
objects actingbackfill-wide until each object is recovered.

For simplicity, since we don't really know what objects need recovery
until activation (and since we can't do anything with that information
prior to activation anyway), we defer populating the missing_loc
information until activation.

We need peers to rollback divergent log entries before we attempt to
read the relevant objects.  The simplest way to accomplish this seems to
be the simply choose to always activate peers if search_for_missing
turns up missing objects.

Due to EC pools, missing is necessary, but not sufficient for readability.
Thus, we instead check is_unreadable for cases where we need to read the object
and reserve is_missing for cases where we need the object context.

wait_for_missing_object becomes waiting_for_unreadable_object in order to avoid
having another layer of waiting_for_* maps.  These ops may be requeued
either when the primary is recovered or when the object is no longer degraded,
depending on when the object becomes readable.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
8b33d60fc4 PG: allow PGBackend to set criteria for PG up-ness
ECBackend needs to be able to require that a readable
set of the most recent interval to write be available
in order to ensure that it rolls back the log far
enough.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
facec7654c PGBackend: add some additional helpers.
ECBackend's primary specific logic mostly won't treat the
primary shard specially, so it'll be handy to have primary
agnostic helpers for get_shard_info and get_shard_missing.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
74b8a0c05c PG,PGBackend: expose acting and backfill_targets
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
5a8c0bb952 os/: allow objects with NO_GEN but non-NO_SHARD shard
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
2d3205ff1a PG: skip CRUSH_ITEM_NONE in acting set in various places
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
5db3b2dc2c osd/: extend pg_interval_t to include primary
Otherwise, we cannot correctly determine up_from/up_thru for
old intervals.  Also, we need this information to determine
when a new interval starts due to a new primary without a
change in the acting set.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:14 -08:00
Samuel Just
9d8be1b082 Objecter: handle primary other than acting[0]
EC pool particularly might have CRUSH_ITEM_NONE as acting[0] which
is not a valid osd.

Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:13 -08:00
Samuel Just
904704ac7f OSDMap: handle CRUSH_ITEM_NONE in acting sets
Signed-off-by: Samuel Just <sam.just@inktank.com>
2014-02-17 20:12:13 -08:00