Commit Graph

91619 Commits

Author SHA1 Message Date
Casey Bodley
cdbc18254d rgw: remove RGWPutObj::select_processor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
8e030b35f1 rgw: RGWBulkUploadOp uses AtomicObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
ee8bc41329 rgw: PostObj uses AtomicObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
522e2718ce rgw: fetch_remote_obj() uses AtomicObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
1ecd78b2da rgw: prepare RGWRadosPutObj for fetch_remote_obj
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
a7eab94ba8 rgw: RGWObjectCtxImpl::invalidate takes const ref
also removes the default implementation, which is duplicated in the
specialization for rgw_obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
dd472ab0cf rgw: rgw_file uses AtomicObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
47b1754e47 rgw: PutObj::execute() uses new ObjectProcessors
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
d6b28cdb39 rgw: compression filter uses rgw::putobj::Pipe
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
70c8c8110c rgw: encryption filter uses rgw::putobj::Pipe
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
d0c9798eb5 rgw: temporarily disable putobj filters
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
6907e85d9d rgw: read multipart params in RGWPutObj_ObjStore_S3::get_params
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
f6a6aedab5 rgw: PutObj avoids some static_casts to RGWObjectCtx
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
572d47bfdb rgw: remove gotos from PutObj
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
6d94a5d204 rgw: PutObj uses unique_ptr to manage PutObjProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
6b6002a802 rgw: PutObj uses scope_guard to update perf counter on return
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
5de7718f45 rgw: add rgw::putobj::MultipartObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
9ff79e39dc rgw: add rgw::putobj::AtomicObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
5c0ee28731 rgw: add const to some RGWBucketInfo helpers
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
65a1340c58 rgw: add rgw::putobj::ManifestObjectProcessor
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
93a0024982 rgw: add const to some manifest generator functions
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Casey Bodley
53a22e8862 rgw: add rgw::putobj::RadosWriter adapter
implements the DataProcessor interface by writing its buffers with Aio,
and tracks the set of successful writes so they can be deleted on
failure/cancelation

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Patrick Donnelly
299fc82ded
Merge PR #24505 into master
* refs/pull/24505/head:
	mds: wait shorter intervals if beacon not sent

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-10-16 07:52:54 -07:00
Marcin Juszkiewicz
ee3af68dd9 cmake: mark gcc 7 as minimum C++ compiler (due to std++17)
Ceph requires C++17 support from compiler. This means gcc 7.x being
minimal version supported.

This also allows to fail quick on Debian 'stretch' with it's gcc 6.3
compiler.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
2018-10-16 16:34:03 +02:00
Kefu Chai
dd460032dd
Merge pull request #24373 from mogeb/build-cls-rbd
osd: add required cls_* libraries as dependencies of osd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-10-16 22:12:53 +08:00
Tiago Melo
eb86ecda08 mgr/dashboard: Remove unnecessary imports/providers
Signed-off-by: Tiago Melo <tmelo@suse.com>
2018-10-16 14:44:59 +01:00
Patrick Donnelly
4014441b8d
mds: wait shorter intervals if beacon not sent
MDS beacon upkeep always waits mds_beacon_interval seconds even when laggy.
Check more frequently for when we stop being laggy to reduce likelihood that
the MDS is removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-16 06:29:35 -07:00
Yan Jun
d24cac1ba4 global: fix err_to_stderr can't work when it was configured to false
function `set_stderr_level(-1, -1)` set m_stderr_log and m_stderr_crash to -1,
regardless of whether `err_to_stderr` is set to false or not, so logs will be
 always written to stderr. fix it as the same way as handle_conf_change does.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
2018-10-16 20:32:10 +08:00
Yan Jun
0cb8ef3c09 osd: replace cout/derr with dout(0) to avoid writing trivial logs to syslog file
Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
2018-10-16 20:31:50 +08:00
Sage Weil
f79bfa0544 PendingReleaseNotes: note about mgr/balancer/max_misplaced change
Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:23:03 -05:00
Sage Weil
3365c5d40b mgr/DaemonServer: remove stale/misleading check
The PGMap::last_osdmap_epoch is always 0.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
1617473f10 mgr/DaemonServer: throttle pgp_num changes based on misplaced %
- block pgp_num increase if pg_num hasn't increased yet
- make no changes if there are inactive or unknown pgs
- make no changes if there are degraded pgs either.  this might be a bit
  conservative...
- calculate the magnitude of our adjusted based on the max_misplaced
  target.  this assumes a uniform distribution of objects across pgs,
  so not perfectly accurate, but hopefully close enough.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
d6022dd87b mgr/DaemonServer: block pg_num decrease(merge) until pgp_num is reduced
We can't merge until the PGs are stored together.  (The mon would stop
us if we tried, but let's not waste time trying.)

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
155d9a2c00 mgr/DaemonServer: adjust_pgs(): cosmetic change to debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
128437c579 mon/PGMap: add get_recovery_stats()
This is some of the same info we get in the json dump from
print_summary -> overall_recovery_summary -> recovery_summary.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
edbd592ee4 mgr/balancer: mgr/balancer/max_misplaced -> pg_max_misplaced
Make this a compiled-in option that is consumed by other components as
well as the balancer.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
4fc2af95ce pybind/mgr/mgr_module: add get_option()
get_config() gets a mgr-style config option.  Add get_option which
fetches a compiled-in config option.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
b4c17cca15 mgr/DaemonServer: allow pg_num increases that abort pending merges
If we are waiting for a PG to merge we can't decrease more, but if we
were in the process of merging and an increase is requested, we can
abort the merge by increase pg_num_actual whenever we want.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
b84fa91368 mon/OSDMonitor: resent pre-nautilus client ops on aborted merge
If we are in premerge (pg_num_pending == pg_num - 1) and abort by
increasing pg_num, we the last_force_op_resend_prenautilus since it will
be an interval change for nautlius+.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
48412ce4e4 mon/OSDMonitor: make pgp_num track pg_num more consistently
Previously we were automatically adjusting pgp_num_target on a
pg_num_target change *only* when decreasing pg_num.  Instead, make
pgp_num (continue to) track pg_num if it currently matches.  If it ever
is set differently than pg_num, leave it different (unless/until it
matches again).

This is still slightly weird, but I think in practice it is good enough.
In the rare case that the admin manually sets pgp_num to something
different than pg_num, they probably won't also be using automagic
pg_num adjustment that might make them match and start tracking again.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
747c9dcf0a Merge PR #24095 into master
* refs/pull/24095/head:
	osd: do not authenticate heartbeat connections until nautilus
	msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat
	osd: add missing space in heartbeat debug output
	mgr/DaemonServer: add missing return
	msg/async: drop verify_authorizer wrapper
	osd: authenticate ping sessions
	msg/simple: remove verify_authorizer wrapper
	msg: remove unused ms_verify_authorizer
	msg/async: remove get_authorizer wrapper
	msg/simple: remove get_authorizer wrapper
	msg/Messenger: pull authenticator validation into Messenger
	msg/Messenger: uninline ms_deliver_verify_authorizer
	mgr/DaemonServer: expose keyring for authenticator verification
	mon: expose keyring for msgr1 authentication
	mds: expose keyring for authenticater verification
	osd: expose keyring for authenticater verification
	msg/Dispatcher: add ms_get_auth1_authorizer_keystore
	mon: fix ref cycle breakage in handle_forward
	mon: use MonOpRequest get_session() instead of PaxosServiceMessage's
	mon: get session from MonOpRequest in handle_command
	messages/MForward: drop unused ctor
	mon: use ms_handle_authentication to parse caps
	mon: kill Session::global_id and use Connection member instead
	mgr/DaemonServer: move session setup into ms_handle_authentication
	mds: move session setup into ms_handle_authentication
	osd: move session setup into ms_handle_authentication
	msg: new ms_handle_authentication, add fields to Connection

Reviewed-by: Ricardo Dias <rdias@suse.com>
2018-10-16 07:20:17 -05:00
Sage Weil
05faeb4d12 Merge PR #24579 into master
* refs/pull/24579/head:
	qa/osd: fixup osd-rep-recov-eio.sh fails to parse pg dump

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-16 07:17:59 -05:00
John Spray
e6a26aeff7
Merge pull request #24597 from batrick/i36450
qa: fix run call args

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
2018-10-16 13:09:08 +01:00
Kefu Chai
67777bbe7a
Merge pull request #23983 from theanalyst/vstart-rgw-asok
vstart: set admin socket for RGW in conf

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 19:15:35 +08:00
Tatjana Dehler
31863f9638 mgr/dashboard: configs textarea disallow horizontal resize
The textarea allows horizontal and vertical resize by default. Only the
vertical resize is appropriate for this form.

Fixes: http://tracker.ceph.com/issues/36452
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2018-10-16 12:28:16 +02:00
Patrick Nawracay
807f431123 mgr/dashboard: Fix spaces around status labels on OSD list
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-10-16 12:03:48 +02:00
kungf
d9a66ea410 OSD: fix a typo err when increase recvoery bytes statistics
Signed-off-by: kungf <yang.wang@easystack.cn>
2018-10-16 17:22:42 +08:00
Kefu Chai
52706830d6
Merge pull request #24576 from cyx1231st/wip-seastar-msgr-refactor
crimson/net: seastar-msgr refactoring

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-10-16 16:44:16 +08:00
Zhi Zhang
552090d71a common: assert if buffer advance length overflow
In the old ceph version, buffer advance length was defined as int, but
in async msg, the real length of data buffer was defined as unsigned.

Occassionly some MDS message back from OSD was too large, which caused
this length overflow and made MDS crash.

For compatibility reason, add an assertion here if buffer advance length
is overflow.

Fixes: http://tracker.ceph.com/issues/36340

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
2018-10-16 14:26:14 +08:00
Kefu Chai
e1fd799b75
Merge pull request #24255 from gregsfortytwo/wip-crush-doc
doc: explain 'firstn v indep' in the CRUSH docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2018-10-16 13:49:19 +08:00