Commit Graph

89109 Commits

Author SHA1 Message Date
Sage Weil
de747b0dc1 ceph_test_objectstore: queue split on parent pg sequencer
This matches the OSD.

It also ensure that we drain transactions that precede the split, which
will include any finish_write calls that might otherwise attach the
Buffer to the wrong cache shard: _txc_finish() calls finish_write and
passes the cache shard without the cache shard lock held, but may block
waiting for split_cache() which then changes the destination collection's
shard.  Once it gets the lock and proceeds it would operate on the wrong
cache shard, leading to a failed assert later when the sharedblob is
trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-14 09:08:12 -05:00
Sage Weil
f21f1f14f2 os/bluestore: fix split vs finish_write race
In _tcx_finish(), we were looking at the right Cache for the collection,
and then calling finish_write with that Cache and taking the lock.  This
could race with a split_cache() such that after we got the lock the
collection was not on a different cache.  This would in turn lead to a
failed assertion later on in _rm_buffer when the sharedblob was trimmed.

Fixes: http://tracker.ceph.com/issues/24439
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-13 12:10:59 -05:00
Sage Weil
8f3560d4ed Merge PR #22879 into master
* refs/pull/22879/head:
	doc/rados/operations/user-management: document 'network' clause of moncap and osdcap
	mds/SessionMap: make Session::connection private
	mds/MDSAuthCaps: parse and enforce network restriction
	mds/MDSAuthCaps: pass addr to is_capable
	mds/SessionMap: track socket_addr for each Session
	osd/OSDCap: enforce network restriction
	osd/OSDCap: take addr arg to is_capable
	osd/OSDCap: parse 'network' clause in grant
	mon: adapt MonCap network checks to addrvecs
	mon/MonCap: enforce network constraint (if present)
	mon/MonCap: take addr for MonCap::is_capable
	mon/MonCap: parse 'network ...' suffix
	osd: add peer_socket_addr to Session
	mon: use addrvec for RoutedRequest and MForward
	msg/Connection: add get_peer_socket_addr()
	common/ipaddr: add network_contains() helper
	common/ipaddr: add parse_network helper
	common/ipaddr: expose netmask_ipv[46]

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2018-08-13 08:48:14 -05:00
Kefu Chai
e2e6a77b38
Merge pull request #23510 from tchaikov/wip-cmake-boost-build
cmake: fix build WITH_SYSTEM_BOOST=ON

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-08-13 21:42:10 +08:00
Patrick Donnelly
418db4617b
Merge PR #23542 into master
* refs/pull/23542/head:
	qa: fix mon config section

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 19:15:20 -07:00
Patrick Donnelly
47e60908a3
qa: fix mon config section
Fixes: fc6e683832

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 19:13:06 -07:00
Patrick Donnelly
0b879f4401
Merge PR #23444 into master
* refs/pull/23444/head:
	cephfs-shell: avoid sys.argv modification
	tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 18:29:34 -07:00
Sage Weil
d7596a74e1 doc/rados/operations/user-management: document 'network' clause of moncap and osdcap
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:03:20 -05:00
Sage Weil
d3af166d6e mds/SessionMap: make Session::connection private
Ensure all setters go through set_connection()

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:03:20 -05:00
Patrick Donnelly
e6c33aa09b
cephfs-shell: avoid sys.argv modification
In multiple locations in the code.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 15:01:34 -07:00
Sage Weil
6651b49863 mds/MDSAuthCaps: parse and enforce network restriction
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
09bf0cd9a4 mds/MDSAuthCaps: pass addr to is_capable
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
37568982e3 mds/SessionMap: track socket_addr for each Session
Make sure socket_addr is populated when connection is set.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
43f812d15d osd/OSDCap: enforce network restriction
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
7ae14b5688 osd/OSDCap: take addr arg to is_capable
Not enforced yet; just plumbing!

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
b1418387df osd/OSDCap: parse 'network' clause in grant
This mirrors what we do in the MonCap (but, naturally, is a slightly
different since we're using a the spirit parse in a totally different way).

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
759a902ace mon: adapt MonCap network checks to addrvecs
Breaking this out in a separate commit to make the backport a bit easier.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
556426e66a mon/MonCap: enforce network constraint (if present)
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
1081e5f691 mon/MonCap: take addr for MonCap::is_capable
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
5038627c9a mon/MonCap: parse 'network ...' suffix
Allow an optional network constraint to any grant.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
7ebdbb7672 osd: add peer_socket_addr to Session
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
388b82f461 mon: use addrvec for RoutedRequest and MForward
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
4985d068aa msg/Connection: add get_peer_socket_addr()
Return the addr that is actually being used for this socket (among the
potentially many in the addrvec).

Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
be0d1c027a common/ipaddr: add network_contains() helper
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
e961633428 common/ipaddr: add parse_network helper
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Sage Weil
52bed1e72b common/ipaddr: expose netmask_ipv[46]
Signed-off-by: Sage Weil <sage@redhat.com>
2018-08-12 17:01:05 -05:00
Patrick Donnelly
0ef08e1ba9
Merge PR #23066 into master
* refs/pull/23066/head:
	include/utime.h: remove improper comments

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-08-12 14:51:26 -07:00
Patrick Donnelly
aab25e41d9
Merge PR #23513 into master
* refs/pull/23513/head:
	doc: updated contact email for Ceph pull requests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 14:49:05 -07:00
Patrick Donnelly
8a5adec8d5
Merge PR #23280 into master
* refs/pull/23280/head:
	cephfs: LeaseStat versioning
	cephfs: DirStat versioning
	mds: make has_feature const function
	cephfs: InodeStat versioning
	mds: add new feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-08-12 14:25:01 -07:00
Patrick Donnelly
45e96a2f46
Merge PR #23538 into master
* refs/pull/23538/head:
	mds: mark beacons as high priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-08-12 14:12:02 -07:00
Patrick Donnelly
fe9a1b5c58
Merge PR #23527 into master
* refs/pull/23527/head:
	mds: use fast dispatch to handle MDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 14:11:58 -07:00
Patrick Donnelly
358432308d
Merge PR #23457 into master
* refs/pull/23457/head:
	qa: ignore normal slow IO warnings for cephfs

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-08-12 14:10:09 -07:00
Patrick Donnelly
fc6e683832
qa: ignore normal slow IO warnings for cephfs
Generally the slow warnings we get are just over the threshold. These warnings
are related to deploying multiple Ceph daemons side-by-side. Let's see how we
do with two minutes.

Ignoring the warnings entirely is unsatisfactory as they serve as a useful
canary in the coal mine when you see warnings for ops > some unreasonably large
amount of time.

Fixes: http://tracker.ceph.com/issues/26900

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-12 13:59:59 -07:00
Casey Bodley
c954417ec3
Merge pull request #23416 from ZVampirEM77/wip-mfa-cmd-and-options
radosgw-admin: add mfa related command and options

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-08-12 14:10:33 -04:00
Pavani Rajula
ab70dffdb2 tools/cephfs-shell: added support for batch file processing and to execute commands from arguments.
Fixes:http://tracker.ceph.com/issues/26853
Fixes:http://tracker.ceph.com/issues/26855

Signed-off-by: Pavani Rajula <rpavani1998@gmail.com>
2018-08-12 22:08:38 +05:30
Kefu Chai
e0abbb7d1c
Merge pull request #22563 from ifed01/wip-ifed-bdev-minsize
os/bluestore: fix bluefs extent miscalculations on small slow device

Reviewed-by: Sage Weil <sage@redhat.com>
2018-08-12 17:30:28 +08:00
Kefu Chai
7fada44d9a
Merge pull request #23294 from noahdesu/timely-mgr-health-updates
mgr: timely health updates between monitor and manager

Reviewed-by: John Spray <john.spray@redhat.com>
2018-08-12 17:29:29 +08:00
Kefu Chai
bf33564177
Merge pull request #23257 from ifed01/wip-ifed-fix-rcoll-race
os/bluestore: fix race between remove_collection and object removals

Reviewed-by: Sage Weil <sage@redhat.com>
2018-08-12 17:28:28 +08:00
Kefu Chai
84ef860caf
Merge pull request #23492 from liewegas/wip-mgrc-warning
mgr/MgrClient: make some noise for a user if no mgr daemon is running

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
2018-08-12 17:26:17 +08:00
Jos Collin
2de7e88e4a
cephfs: LeaseStat versioning
* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped LeaseStat::encode().
* The old format is maintained for backward compatibility.
* Created Locker::encode_lease() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>
2018-08-12 06:42:18 +05:30
Jos Collin
b576cebb65
cephfs: DirStat versioning
* Use the new feature bit CEPHFS_FEATURE_REPLY_ENCODING for encoding.
* encode/decode in the new format.
* Dropped DirStat::encode().
* The old format is maintained for backward compatibility.
* Created CDir::encode_dirstat() for encoding and dropped the duplicates.

Fixes: http://tracker.ceph.com/issues/24444
Signed-off-by: Jos Collin <jcollin@redhat.com>
2018-08-12 06:38:41 +05:30
Patrick Donnelly
4a9fcb6c6f
Merge PR #23500 into master
* refs/pull/23500/head:
	debian: mark python-ceph-argparse Arch = all
	rpm: package cephfs-shell for fedora
	tools/cephfs,deb: package cephfs-shell
	cmake: install script and egg-info files of cephfs-shell
	tools/cephfs: add setup.py for cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-11 16:33:47 -07:00
Patrick Donnelly
7b766ab48e
Merge PR #23480 into master
* refs/pull/23480/head:
	tools/cephfs: fix mkdirs(exist_ok) call
	tools/cephfs: cephfs-shell cleanup
	tools/cephfs: run cephfs-shell thru autopep8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-11 16:27:48 -07:00
Patrick Donnelly
ec3fa7b565
mds: mark beacons as high priority
The mons already mark beacon replies as high priority (via default mon message
priority). We should expect that the mons handle our beacons at the same
priority so the MDS doesn't wrongly get marked laggy.

Fixes: http://tracker.ceph.com/issues/26899

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-08-11 16:11:34 -07:00
Casey Bodley
6bb38deb35
Merge pull request #22868 from iliul/modify-log-level
rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-08-11 10:48:24 -04:00
iliul
0a3f3d0e34
rgw: Fix log level of gc_iterate_entries
Fixes: http://tracker.ceph.com/issues/23801

Signed-off-by: iliul <liul.stone@gmail.com>
2018-08-11 11:17:15 +08:00
Yan, Zheng
7fc04be933 mds: use fast dispatch to handle MDSBeacon
Current mds handles MDSBeacon messages through MDSRank::ms_dispatch().
MDSRank::ms_dispatch() locks mds_lock at the very beginning. This means
that long running task (such as processing finished contexts) can delay
MMDSBeacon. Using fast dispatch for MDSBeacon can avoid this issue.

Fixes: http://tracker.ceph.com/issues/23519
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2018-08-11 07:33:13 +08:00
Sage Weil
c6eb1c4a5a Merge PR #22838 into master
* refs/pull/22838/head:
	kv/RocksDBStore: Handle nullptr if clock cache is chosen.
	kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri pool.
	src/kv: Initial import of a custom RocksDB cache.
	src/rocksdb: switch back to master branch.

Reviewed-by: Sage Weil <sage@redhat.com>
2018-08-10 16:32:34 -05:00
Sage Weil
bdf1653ec9 Merge PR #22970 into master
* refs/pull/22970/head:
	os/bluestore: limit OSD memory usage by tuning the cache size.

Reviewed-by: Sage Weil <sage@redhat.com>
2018-08-10 16:32:29 -05:00
Enming.Zhang
fd47c91a45 radosgw-admin: add mfa related command and options
command:
+ mfa create
+ mfa list
+ mfa get
+ mfa remove
+ mfa check
+ mfa resync

options:
+ --totp-serial
+ --totp-seed
+ --totp-second
+ --totp-window
+ --totp-pin

Signed-off-by: Enming.Zhang <enming.zhang@umcloud.com>
2018-08-10 12:16:49 -04:00