Commit Graph

81238 Commits

Author SHA1 Message Date
Kefu Chai
dbe33502cb qa/suites/hadoop: switch from trusty to ubuntu latest
use the ubuntu_face.yaml facet for the distro used for testing. since we
are dropping the support for trusty.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-04 14:06:42 +08:00
Kefu Chai
88311be439 qa/suites/fs: use ubuntu_latest for multiclient tasks
instead of using ubuntu 14.04, since we want to drop the support of this
release.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-04 14:06:40 +08:00
Kefu Chai
aa89bb2f93 qa/suites/fs: use ubuntu_latest for libcephfs_java
* instead of using ubuntu 14.04, use ubuntu_latest.since we want
  to drop the support of this release.
* refactor this test to use the facet of ubuntu_latest.yaml.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-04 14:06:37 +08:00
Kefu Chai
582f04508f qa/tasks: remove test_buildpackages
the buildpackages feature is not part of ceph. and its test was moved to
teuthology in 5a43f8d5. let's remove the leftovers from ceph/ceph. BTW,
we have not run this test suite since then. and its dependency cannot be
fulfilled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-04 14:06:18 +08:00
Yan, Zheng
f10cd45952 mds: don't create snap inode during trans-authority rename
MDCache::cow_inode() may add auth pin to frozen inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-12-04 12:02:55 +08:00
Yan, Zheng
28d2b20df3 mds: trim non-auth snap dentry during cache rejoin
Because corresponding dentry in auth mds may have been pruned (snap deleted)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-12-04 12:02:54 +08:00
Yan, Zheng
5d4b440430 mds: detach unlinked inode's snaprealm from its parent
When replaying slave update, inode may gets unlinked, but preserved
in memory in case of rollback. In this case, we need to detach the
inode's snaprealm from its parent.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-12-04 12:02:54 +08:00
Yan, Zheng
da12abbd0f mds: fix CInode::find_snaprealm()
If inode has no parent dentry (newly created), CInode::find_snaprealm()
should try the oldest projected parent.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2017-12-04 12:01:18 +08:00
Sage Weil
8c44dabe4b osd/PG: ignore purged_snaps inconsistencies for now
These are possible because we update purged_snaps, part of the pg_info_t,
but we do not bump the pg version or match it with a log entry, which
means that the change does not reliably propagate to new OSDs during
peering etc.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-03 11:36:27 -06:00
Kefu Chai
8ad1a5b642
Merge pull request #19196 from shun-s/wip-cephdisk-fix-major-error
ceph-disk: fix signed integer is greater than maximum when call major

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-12-03 10:34:02 +08:00
Sage Weil
46abd50f8b ceph-volume: use --keyfile instead of --key
We do not want the key to show up on the command line (it may appear in
the process list or sudo log file).

Fixes: http://tracker.ceph.com/issues/22283
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-03 10:15:51 +08:00
Sage Weil
18aff37ee3 osd: accept 'keyfile' as well as 'key' during mkfs
Also, don't print the raw key to the log.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kefu@redhat.com>
2017-12-03 10:15:51 +08:00
Sage Weil
55d499bdef
Merge pull request #19296 from shinobu-x/strange_double_space
osd: Remove double space
2017-12-02 11:44:15 -06:00
Shinobu Kinjo
0fb7f9754d osd: Remove double space
Should not break anything but just strange...

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>
2017-12-02 11:57:39 -05:00
Kefu Chai
f8df9280e5
Merge pull request #19161 from xiexingguo/wip-pg-log
osd/PGLog: get rid of ineffective container operations

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-12-02 22:36:19 +08:00
xie xingguo
d98aa399b0 osd/PGLog: get rid of ineffective container operations
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2017-12-02 15:16:28 +08:00
Kefu Chai
52bb6e8c6e
Merge pull request #19242 from liewegas/wip-21147
qa/suites/rados/thrash: extend mgr beacon grace when many msgr failures injected

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-12-02 14:14:29 +08:00
Kefu Chai
9185109cb9
Merge pull request #19185 from liewegas/wip-trim-mgrmap
mon/MgrMonitor: limit mgrmap history

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-12-02 14:13:44 +08:00
Sage Weil
f2d602acb8 mon/OSDMonitor: propagate new_removed_snaps to other tiers
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:27:02 -06:00
Sage Weil
9607a2db46 mon/OSDMonitor: prune purged snaps
Be a bit careful here because the mon has to do some bookkeeping to avoid
pruning things twice.  If the PGMapDigest set appears obviously stale,
skip some work (looking at this particular interval) until it is not
obviously stale--move onto the next interval instead.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:27:02 -06:00
Sage Weil
fd6a59ebf4 mon/OSDMonitor: convert removed_snaps on first mimic map
On the first mimic map, consider previously removed_snaps to be removed
in that epoch (since we don't easily know when it happened).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
16c5bcc021 osd/osd_types: pg_pool_t: add FLAG_{SELFMANAGED,POOL}_SNAPS flags
Explicitly track whether we are a pool snaps pool or a selfmanaged
snaps pool.  This was inferred from removed_snaps.empty() before, but
that was fragile and kludgey and removed_snaps is going away.

The upgrade/compat behavior is a bit tricky:

- on decode, we set the flags based on the legacy condition.  This lets us
use and rely on the flags in memory.
- on encode, we exclude the flags if decoding an older pg_pool_t

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
6e1b7c4c14 osd/PG: use new mimic osdmap structures for removed, pruned snaps
- update snap_trimq and purged_snaps based on new mimic OSDMap fields
- improve debug output to include both trimq and purged

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
231ec67b7a osd/PG: simplify replica purged_snaps update
This dependency on the ondisk version dates back before argonaut, and no
longer makes sense.  Once the snap is trimmed by the primary, and
purged_snaps is updated, the replica can (must!) blindly follow suit.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
f04729cede osd/PG: break out of Active AdvMap handler if interval change
If we are about to lose our primary status, we don't want to do *any*
of this stuff... especially share_pg_info(), which would get tagged with
the current epoch but confuse our peers!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
33c9907662 osd/PG: some whitespace
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
e5f62fb8ac osd/PG: move debug_verify_cached_snaps check into PGPool::update
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
86f0b81188 mon/PGMap: add purged_snaps map to PGMapDigest
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
6df912b18b osd/PG: share purged_snaps with mgr at mimic
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
345d3b655a osd/osd_types: add purged_snaps to pg_stat_t
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:49 -06:00
Sage Weil
a53ba7314c osd,mon: add 'nosnaptrim' osd flag
Now

 ceph osd {set,unset} nosnaptrim

will suspend or resume snap trimming.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
192a8dc786 osdc/Objecter: apply removed_snaps from gap to in-flight requests
If we are so laggy that we aren't contiguous with the mon's latest
map, the mon will provide a summary of removed_snaps for the gap.
Apply those to our in-flight ops.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
b1b8fc6738 osdc/Objecter: rename _scan_requests force_resend -> skipped_map
This is what the caller is passing.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
32d7538a50 osdc/Objecter: prune new_removed_snaps from active op snapc's
If an in-flight Op has a snapc referencing a deleted snap, remove it
from the snapc.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
38e96ec279 mon/MgrStatMonitor: dump PGMapDigest at debug level 20
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
49833c3bb2 mon/OSDMonitor: share snaps removed during a map gap
If a client requests a map older than the mon's oldest, share with
them snaps deleted during the gap too.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
9d606c587e mon/OSDMonitor: record removed_snaps by epoch outside of the osdmap
Index by snap and by epoch; separate out pools.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:26:48 -06:00
Sage Weil
553048fbf9 osd/OSDMap: track newly removed and purged snaps in each epoch
Instead of maintaining a set of snapids that have been removed over
all time, instead note just the newly removed and newly purged snaps
in each OSDMap epoch.  This is easier to consume for both the Objecter
and OSD.

Also keep the interval of snaps that have been removed but not perged
in each OSDMap.  This is extremely convenient because it frees the OSDs
from having to maintain this information in parallel even when they may
not have PGs belonging to those pools.  These structures will be large
right when the ugprade happens and the pg_pool_t::removed_snaps gets copied
to the new fields, but in the steady state it will be relatively small,
reflecting only the set of snaps that are currently being removed.

This also provides convenient visibility into the "trimming snaps" set
that the cluster is working on.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:23 -06:00
Sage Weil
37c4affa25 mon/OSDMonitor: clear pending_metadata* in create_pending
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:23 -06:00
Sage Weil
1f133a2350 mon/OSDMonitor: reset OSDMap state before decode
This ensures we don't have any cruft left over in fields that decode()
assumes are initialized from the ctor (and not a previous instance).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
e89649dca5 mds/SnapServer: fix reset()
pg_pool_t's snap_seq is an upper bound on any allocated (or removed)
snapids.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
b9c5a24395 osd/osd_types: SnapSet: remove get_first_snap_after()
No users.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
1b1eec29ae include/types: flat_set operator<<
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
3119cf5cee include/mempool: add flat_set alias
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
ea308ad54e include/interval_set: add get_end() to iterator
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
df7523b882 qa/suites/rados/singleton/all/thrash-eio: more whitelist
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
81d63f2994 osd/OSDMap: improve osdmap flag dumping in json
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
c8bfe3fa53 osd/PG: share_pg_info shares past_itnervals, not PastIntervals()
In reality we only call this when the PG is peered and thus past_intervals
is empty, but this is more defensive in case that changes someday!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:16:22 -06:00
Sage Weil
c536d4c294 osd/osd_types: note about removed_snaps hack
We add in the new snap_seq just to try to keep the interval_set
contiguous.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-01 21:15:02 -06:00
Vasu Kulkarni
95fb43b54d qa/tests: add tests for ceph-volume
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
2017-12-01 14:19:41 -08:00