Commit Graph

72280 Commits

Author SHA1 Message Date
Sage Weil
aed09380ca ceph-objectstore-tool: handle both new and legacy SnapSet format
If we encounter legacy snaps, add to snapmapper then; otherwise, use the
clone_snaps field in SnapSet to add all clones to snapmapper when we
process the head object.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:39:13 -04:00
Sage Weil
c048f6affb test/osd/osd-scrub-snaps.sh: fix test
- objects come in a different order, meh
- ss is on head, always, not snapdir.
- error messages on head, not snapdir

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:13 -04:00
Sage Weil
376015fd9e osd/PrimaryLogPG: only clear SnapSet::head_exists for legacy snapsets
In the new world, head_exists is always true.  Stop clearing it in
_delete_oid (except for legacy compatibility), and fix the various callers
and assertions to compensate.  Note that we now use head_exists as a flag
to guide us into the snapdir branch of finish_ctx() (which will go away
post-luminous).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f063506b21 osd/osd_types: pass legacy flag to from_snap_set
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
bbc63bf204 osd/osd_types: document snaps vectors are descending
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
138a808afa osd/ReplicatedBackend: set recovry_info.ss on non-head push
Replicas need this in order to store the clones in SnapMapper.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f0db7db65f osd/ECBackend: populate recovery_info
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
e24d5024c1 osd/ECBackend: whitespace
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f016d4f389 osd/PrimaryLogPG: clear whiteout on exists=false objects in obc cache
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
e3e8caec5f osd/PrimaryLogPG: remove head if snapset is empty and head is whiteout
We include a check to make sure we do not delete a dirty whiteout if this
is a tier pool and the object is dirty.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
48e3f0eeed osd/PrimaryLogPG: maintain pessimistic estimate of num_legacy_snapsets
- Assume that any snapset we update, if not require_luminous, is a net-new
legacy SnapSet.  (It might be an existing one, which would be net-0, but
that is harder to tell.)

Then, during scrub,

- Any unreadable oi is assumed to include a legacy snapset
- Any snapset we encounter if !require_luminous is legacy
- Any object that should have a snapset but doesn't (corrupt or missing)
is assumed to be legacy.
- If were trying to update a legacy Snapset but have to abort, then it is
still legacy.

We could assume that a missing/broken snapset is not legacy since it has
to be repaired anyway (and therefore shouldn't block upgrade), but I'm
not sure.  For now, we'll take the conservative approach of blocking the
upgrade if the snapset metadata is missing/corrupt.

Signed-off-by: Sage Weil <sage@redhat.com>

# Conflicts:
#	src/osd/PrimaryLogPG.cc
2017-05-05 13:38:12 -04:00
Sage Weil
3862309d50 osd/osd_types: add num_legacy_snapsets to stats struct
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
7911950d85 osd/PrimaryLogPG: convert snapdir during scrub
During scrub, we assemble the clone legacy_snaps values and put them
in the SnapSet.

Note that we do not bother clearing legacy_snaps in the clones; this is
more complexity to do correctly and not worth the effort; we will always
have the SnapSet to indicate whether we need to look at the oldf ield
or not.

If the rebuild clone_snaps is not the correct size we bail out; this can
happen if some clones are missing oi or missing entirely, or if there are
extra clones.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
7f90c72394 osd: store clone list in SnapSet
Store the per-clone snaps list in SnapSet if (1) REQUIRE_LUMINOUS is set
in the OSDMap and (2) the SnapSet isn't a 'legacy' SnapSet that hasn't
been converted yet by scrub.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
bdaf7740da osd/osd_types: rename object_info_t::snaps -> legacy_snaps
This will make it easier to identify users of this field that need to
conditionally use either the old legacy_snaps or the new
SnapSet::clone_snaps vector.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
a7b5922169 osd/osd_types: add clone_snaps to SnapSet
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
de6f09f43a osd/PrimaryLogPG: delete + ignore_cache is a soft hint
We may still need to create a whiteout because clones still exist.

Arguably delete+ignore_cache is not the right way to remove whiteouts and
we should have a separate RADOS operation for this.  But we don't.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
2640101e9a osd/PrimaryLogPG: keep snapset on whiteout head if luminous
If REQUIRE_LUMINOUS is set on the OSDMap, put the SnapSet on the head
and make it a whiteout.  This is simpler and will eventually (once all
the old snapdir objects have been fixed up) let us remove a ton of
snapdir-related headaches.

This is surprisingly simple.  Everywhere else we work with snapdir is
already set up to handle the snapset on the head.  The only difference is
that we are preventing outselves from moving from the snapset-on-head
state to the snapset-on-snapdir-with-no-head state.  The only time this
happens is when the object is logically deleted in _delete_oid.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
10f8615e8f osd/PrimaryLogPG: filter snapc.snaps up front
No reason to wait for make_writeable(); ensure we have a valid snapc
from the start.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
1d87287d45 osd/osd_types: remove unused snapcolls from ScrubMap::object
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
27d50fa157 osd/osd_types: drop ScrubMap::object::nlink; drop some decode compat cruft
We moved to v7 encoding in ca81563d8d71052a2040eda74e7ac418f1616fc7, which
was pre-jewel, so that's the oldest version we need to decode.

Drop the nlink field, which has not been used in a long time.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
6f2e82b997 osd/osd_types: move SnapSetContext to osd_internal_types
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
5484fe13c0 osd/SnapMapper: add some debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
3f2cc3d248 osd/PrimaryLogPG: debug obs.exists from get_object_context
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
be4d92ca2e osd/PrimaryLogPG: improve make_writeable debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Jason Dillaman
965cfb524d Merge pull request #14834 from runsisi/wip-fix-double-free
librbd: minor fixes for image trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 13:16:48 -04:00
Jason Dillaman
bcae20d662 Merge pull request #14830 from runsisi/wip-open-cleanup
librbd: remove redundant check for image id emptiness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 13:16:35 -04:00
Kefu Chai
6e1e5407d8 Merge pull request #14927 from tchaikov/wip-19427
logclient: use the seq id of the 1st log entry when resetting session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2017-05-06 00:31:33 +08:00
Kefu Chai
8ecdf9a57d Merge pull request #14920 from tchaikov/wip-14579
mgr: add machinery for python modules to send MCommands to daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-05-06 00:29:12 +08:00
Kefu Chai
e1d435b196 Merge pull request #13681 from liewegas/wip-epochs
osd: tag fast dispatch messages with min_epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-05-06 00:27:58 +08:00
Sage Weil
897566070a osd/OSDMapMapping: add assert
This is easier to debug than walking off into memory we don't own!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 11:47:03 -04:00
Sage Weil
ab7df6d573 mon/OSDMonitor: fix prime_pg_temp condition
We have to make sure the *pg* exists, not just the pool, in the previous
map, or else we'll get a funky overrun in OSDMapMapping::get().

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 11:43:59 -04:00
Casey Bodley
9667761ecf Merge pull request #14273 from cbodley/wip-rgw-beast-new-parser
rgw: update Beast for streaming reads in asio frontend

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
2017-05-05 11:38:54 -04:00
Dan van der Ster
da9f1d8294 client: client_quota no longer optional
Remove the possibility to disable client quotas.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
2017-05-05 16:48:20 +02:00
Mykola Golub
8e912b9a05 rbd-nbd: relax size check for newer kernel versions
Fixes: http://tracker.ceph.com/issues/19871
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
2017-05-05 16:44:51 +02:00
Casey Bodley
4d076dd565 rgw: rename frontend from asio to beast
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Casey Bodley
5b41fb55e4 rgw: update Beast for streaming reads in asio frontend
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Casey Bodley
78cd41d599 rgw: update Beast submodule
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Sage Weil
99928c9e0d Merge pull request #14931 from tchaikov/wip-19771
qa/tasks/ceph_manager: always fix pgp_num when done with thrashosd task

Reviewed-by: Sage Weil <sage@redhat.com>
2017-05-05 08:53:38 -05:00
Loic Dachary
dca1ae1e0a crush: update documentation for negative choose step
And remove an obsolete comment.

Signed-off-by: Loic Dachary <loic@dachary.org>
2017-05-05 15:39:26 +02:00
Sage Weil
d21ea85f31 Merge pull request #14973 from ifed01/wip-bluestore-kvsync-cond
os/bluestore: remove unused condition variable

Reviewed-by: Sage Weil <sage@redhat.com>
2017-05-05 08:20:35 -05:00
Igor Fedotov
71d60ba8c4 os/bluestore: remove unused condition variable
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
2017-05-05 15:33:01 +03:00
Mykola Golub
dff49227fa Merge pull request #14963 from dillaman/wip-19858
rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-05 15:27:07 +03:00
Jason Dillaman
8c9971b718 Merge pull request #14965 from dmick/wip-19865
test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 07:30:52 -04:00
Spandan Kumar Sahu
6f10373b7a mgr: remove non-existent MDS daemons from FSMap
Fixes: http://tracker.ceph.com/issues/17453
Signed-off-by: Spandan Kumar Sahu <spandankumarsahu@gmail.com>
2017-05-05 16:54:58 +05:30
Mykola Golub
a20c62760b Merge pull request #14465 from dillaman/wip-librbd-devstack-tempest
qa/workunits: switch to OpenStack Ocata release for RBD testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-05 14:12:53 +03:00
Jin Cai
1dc3e4f23a osd/OSD.cc: lock the pg only when all conditional checks are passed when
scheduling a  scrub

Signed-off-by: Jin Cai <caijin.caij@alibaba-inc.com>
2017-05-05 17:53:22 +08:00
Kefu Chai
561cbded0c mon: check is_shutdown() in timer callbacks
introduce a helper class: C_MonContext, and initialize all timer events
using it, to ensure that they do check is_shutdown() before doing their
work.

Fixes: http://tracker.ceph.com/issues/19825
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-05-05 13:54:45 +08:00
Dan Mick
73bcaef349 test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset
If RBD_FEATURES is not in the environment, set *features to 0 in
get_features(); callers rely on a valid return value.  (This was
breaking on arm64.)

Fixes: http://tracker.ceph.com/issues/19865
Signed-off-by: Dan Mick <dan.mick@redhat.com>
2017-05-04 20:15:58 -07:00
Kefu Chai
12139ae529 mon/Elector: call cancel_timer() in shutdown()
instead of doing it manually.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-05-05 09:50:47 +08:00