Commit Graph

71912 Commits

Author SHA1 Message Date
Sage Weil
aed09380ca ceph-objectstore-tool: handle both new and legacy SnapSet format
If we encounter legacy snaps, add to snapmapper then; otherwise, use the
clone_snaps field in SnapSet to add all clones to snapmapper when we
process the head object.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:39:13 -04:00
Sage Weil
c048f6affb test/osd/osd-scrub-snaps.sh: fix test
- objects come in a different order, meh
- ss is on head, always, not snapdir.
- error messages on head, not snapdir

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:13 -04:00
Sage Weil
376015fd9e osd/PrimaryLogPG: only clear SnapSet::head_exists for legacy snapsets
In the new world, head_exists is always true.  Stop clearing it in
_delete_oid (except for legacy compatibility), and fix the various callers
and assertions to compensate.  Note that we now use head_exists as a flag
to guide us into the snapdir branch of finish_ctx() (which will go away
post-luminous).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f063506b21 osd/osd_types: pass legacy flag to from_snap_set
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
bbc63bf204 osd/osd_types: document snaps vectors are descending
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
138a808afa osd/ReplicatedBackend: set recovry_info.ss on non-head push
Replicas need this in order to store the clones in SnapMapper.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f0db7db65f osd/ECBackend: populate recovery_info
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
e24d5024c1 osd/ECBackend: whitespace
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
f016d4f389 osd/PrimaryLogPG: clear whiteout on exists=false objects in obc cache
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
e3e8caec5f osd/PrimaryLogPG: remove head if snapset is empty and head is whiteout
We include a check to make sure we do not delete a dirty whiteout if this
is a tier pool and the object is dirty.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
48e3f0eeed osd/PrimaryLogPG: maintain pessimistic estimate of num_legacy_snapsets
- Assume that any snapset we update, if not require_luminous, is a net-new
legacy SnapSet.  (It might be an existing one, which would be net-0, but
that is harder to tell.)

Then, during scrub,

- Any unreadable oi is assumed to include a legacy snapset
- Any snapset we encounter if !require_luminous is legacy
- Any object that should have a snapset but doesn't (corrupt or missing)
is assumed to be legacy.
- If were trying to update a legacy Snapset but have to abort, then it is
still legacy.

We could assume that a missing/broken snapset is not legacy since it has
to be repaired anyway (and therefore shouldn't block upgrade), but I'm
not sure.  For now, we'll take the conservative approach of blocking the
upgrade if the snapset metadata is missing/corrupt.

Signed-off-by: Sage Weil <sage@redhat.com>

# Conflicts:
#	src/osd/PrimaryLogPG.cc
2017-05-05 13:38:12 -04:00
Sage Weil
3862309d50 osd/osd_types: add num_legacy_snapsets to stats struct
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
7911950d85 osd/PrimaryLogPG: convert snapdir during scrub
During scrub, we assemble the clone legacy_snaps values and put them
in the SnapSet.

Note that we do not bother clearing legacy_snaps in the clones; this is
more complexity to do correctly and not worth the effort; we will always
have the SnapSet to indicate whether we need to look at the oldf ield
or not.

If the rebuild clone_snaps is not the correct size we bail out; this can
happen if some clones are missing oi or missing entirely, or if there are
extra clones.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
7f90c72394 osd: store clone list in SnapSet
Store the per-clone snaps list in SnapSet if (1) REQUIRE_LUMINOUS is set
in the OSDMap and (2) the SnapSet isn't a 'legacy' SnapSet that hasn't
been converted yet by scrub.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
bdaf7740da osd/osd_types: rename object_info_t::snaps -> legacy_snaps
This will make it easier to identify users of this field that need to
conditionally use either the old legacy_snaps or the new
SnapSet::clone_snaps vector.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
a7b5922169 osd/osd_types: add clone_snaps to SnapSet
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
de6f09f43a osd/PrimaryLogPG: delete + ignore_cache is a soft hint
We may still need to create a whiteout because clones still exist.

Arguably delete+ignore_cache is not the right way to remove whiteouts and
we should have a separate RADOS operation for this.  But we don't.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:12 -04:00
Sage Weil
2640101e9a osd/PrimaryLogPG: keep snapset on whiteout head if luminous
If REQUIRE_LUMINOUS is set on the OSDMap, put the SnapSet on the head
and make it a whiteout.  This is simpler and will eventually (once all
the old snapdir objects have been fixed up) let us remove a ton of
snapdir-related headaches.

This is surprisingly simple.  Everywhere else we work with snapdir is
already set up to handle the snapset on the head.  The only difference is
that we are preventing outselves from moving from the snapset-on-head
state to the snapset-on-snapdir-with-no-head state.  The only time this
happens is when the object is logically deleted in _delete_oid.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
10f8615e8f osd/PrimaryLogPG: filter snapc.snaps up front
No reason to wait for make_writeable(); ensure we have a valid snapc
from the start.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
1d87287d45 osd/osd_types: remove unused snapcolls from ScrubMap::object
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
27d50fa157 osd/osd_types: drop ScrubMap::object::nlink; drop some decode compat cruft
We moved to v7 encoding in ca81563d8d, which
was pre-jewel, so that's the oldest version we need to decode.

Drop the nlink field, which has not been used in a long time.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
6f2e82b997 osd/osd_types: move SnapSetContext to osd_internal_types
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
5484fe13c0 osd/SnapMapper: add some debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
3f2cc3d248 osd/PrimaryLogPG: debug obs.exists from get_object_context
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Sage Weil
be4d92ca2e osd/PrimaryLogPG: improve make_writeable debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2017-05-05 13:38:11 -04:00
Jason Dillaman
965cfb524d Merge pull request #14834 from runsisi/wip-fix-double-free
librbd: minor fixes for image trash move

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 13:16:48 -04:00
Jason Dillaman
bcae20d662 Merge pull request #14830 from runsisi/wip-open-cleanup
librbd: remove redundant check for image id emptiness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 13:16:35 -04:00
Kefu Chai
6e1e5407d8 Merge pull request #14927 from tchaikov/wip-19427
logclient: use the seq id of the 1st log entry when resetting session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2017-05-06 00:31:33 +08:00
Kefu Chai
8ecdf9a57d Merge pull request #14920 from tchaikov/wip-14579
mgr: add machinery for python modules to send MCommands to daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-05-06 00:29:12 +08:00
Kefu Chai
e1d435b196 Merge pull request #13681 from liewegas/wip-epochs
osd: tag fast dispatch messages with min_epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-05-06 00:27:58 +08:00
Casey Bodley
9667761ecf Merge pull request #14273 from cbodley/wip-rgw-beast-new-parser
rgw: update Beast for streaming reads in asio frontend

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
2017-05-05 11:38:54 -04:00
Casey Bodley
4d076dd565 rgw: rename frontend from asio to beast
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Casey Bodley
5b41fb55e4 rgw: update Beast for streaming reads in asio frontend
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Casey Bodley
78cd41d599 rgw: update Beast submodule
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-05-05 10:12:35 -04:00
Sage Weil
99928c9e0d Merge pull request #14931 from tchaikov/wip-19771
qa/tasks/ceph_manager: always fix pgp_num when done with thrashosd task

Reviewed-by: Sage Weil <sage@redhat.com>
2017-05-05 08:53:38 -05:00
Sage Weil
d21ea85f31 Merge pull request #14973 from ifed01/wip-bluestore-kvsync-cond
os/bluestore: remove unused condition variable

Reviewed-by: Sage Weil <sage@redhat.com>
2017-05-05 08:20:35 -05:00
Igor Fedotov
71d60ba8c4 os/bluestore: remove unused condition variable
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
2017-05-05 15:33:01 +03:00
Mykola Golub
dff49227fa Merge pull request #14963 from dillaman/wip-19858
rbd-mirror: failover and failback of unmodified image results in split-brain

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-05 15:27:07 +03:00
Jason Dillaman
8c9971b718 Merge pull request #14965 from dmick/wip-19865
test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-05-05 07:30:52 -04:00
Mykola Golub
a20c62760b Merge pull request #14465 from dillaman/wip-librbd-devstack-tempest
qa/workunits: switch to OpenStack Ocata release for RBD testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-05 14:12:53 +03:00
Dan Mick
73bcaef349 test/librbd/test_librbd.cc: set *features even if RBD_FEATURES is unset
If RBD_FEATURES is not in the environment, set *features to 0 in
get_features(); callers rely on a valid return value.  (This was
breaking on arm64.)

Fixes: http://tracker.ceph.com/issues/19865
Signed-off-by: Dan Mick <dan.mick@redhat.com>
2017-05-04 20:15:58 -07:00
Jason Dillaman
4031555dda librbd: add no-op event when promoting an image
The rbd-mirror process needs an event in the journal
to properly detect the transition between primary and
non-primary state between peers.

Fixes: http://tracker.ceph.com/issues/19858
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2017-05-04 20:26:03 -04:00
Jason Dillaman
23ad14377f librbd: do not delay propagation of demote events
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2017-05-04 19:47:01 -04:00
Jason Dillaman
3f179bf862 rbd-mirror: prevent infinite loop when computing replay status
If the image had a non-primary predecessor epoch whose tag tid
duplicates an epoch within its own journal, an infinite loop
would result.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2017-05-04 18:00:48 -04:00
Yehuda Sadeh
77a43573ab Merge pull request #14373 from rzarzynski/wip-rgw-19520
rgw: fix broken /crossdomain.xml, /info and /healthcheck of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2017-05-04 11:25:50 -07:00
Casey Bodley
ff724c9498 Merge pull request #9950 from weiqiaomiao/wqm-wip-fetchobj-req
rgw: should delete in_stream_req  if conn->get_obj(...) return not zero value

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-05-04 14:13:01 -04:00
Casey Bodley
9ddbcfd85e Merge pull request #14601 from Jing-Scott/fix-bulkupload-multisite
rgw: fix forward request for bulkupload to be applied in multisite

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
2017-05-04 14:01:35 -04:00
Mykola Golub
2a71c8f9da Merge pull request #14945 from dillaman/wip-19811
rbd-mirror: ensure missing images are re-synced when detected

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-04 20:47:49 +03:00
Mykola Golub
285c4a15a2 Merge pull request #14935 from dillaman/wip-rbd-cli-mirror-pool
rbd: removed spurious error message from mirror pool commands

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-05-04 20:47:20 +03:00
Tamilarasi Muthamizhan
a189b61095 Merge pull request #14400 from ceph/wip-cd-1node
qa/tasks: few fixes to get ceph-deploy 1node to working state
2017-05-04 10:42:50 -07:00