1
0
mirror of https://github.com/ceph/ceph synced 2025-02-19 08:57:27 +00:00
Commit Graph

91481 Commits

Author SHA1 Message Date
Casey Bodley
53a22e8862 rgw: add rgw::putobj::RadosWriter adapter
implements the DataProcessor interface by writing its buffers with Aio,
and tracks the set of successful writes so they can be deleted on
failure/cancelation

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 11:06:14 -04:00
Patrick Donnelly
299fc82ded
Merge PR into master
* refs/pull/24505/head:
	mds: wait shorter intervals if beacon not sent

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-10-16 07:52:54 -07:00
Marcin Juszkiewicz
ee3af68dd9 cmake: mark gcc 7 as minimum C++ compiler (due to std++17)
Ceph requires C++17 support from compiler. This means gcc 7.x being
minimal version supported.

This also allows to fail quick on Debian 'stretch' with it's gcc 6.3
compiler.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
2018-10-16 16:34:03 +02:00
Kefu Chai
dd460032dd
Merge pull request from mogeb/build-cls-rbd
osd: add required cls_* libraries as dependencies of osd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-10-16 22:12:53 +08:00
Tiago Melo
eb86ecda08 mgr/dashboard: Remove unnecessary imports/providers
Signed-off-by: Tiago Melo <tmelo@suse.com>
2018-10-16 14:44:59 +01:00
Patrick Donnelly
4014441b8d
mds: wait shorter intervals if beacon not sent
MDS beacon upkeep always waits mds_beacon_interval seconds even when laggy.
Check more frequently for when we stop being laggy to reduce likelihood that
the MDS is removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-16 06:29:35 -07:00
Yan Jun
d24cac1ba4 global: fix err_to_stderr can't work when it was configured to false
function `set_stderr_level(-1, -1)` set m_stderr_log and m_stderr_crash to -1,
regardless of whether `err_to_stderr` is set to false or not, so logs will be
 always written to stderr. fix it as the same way as handle_conf_change does.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
2018-10-16 20:32:10 +08:00
Yan Jun
0cb8ef3c09 osd: replace cout/derr with dout(0) to avoid writing trivial logs to syslog file
Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>
2018-10-16 20:31:50 +08:00
Sage Weil
f79bfa0544 PendingReleaseNotes: note about mgr/balancer/max_misplaced change
Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:23:03 -05:00
Sage Weil
3365c5d40b mgr/DaemonServer: remove stale/misleading check
The PGMap::last_osdmap_epoch is always 0.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
1617473f10 mgr/DaemonServer: throttle pgp_num changes based on misplaced %
- block pgp_num increase if pg_num hasn't increased yet
- make no changes if there are inactive or unknown pgs
- make no changes if there are degraded pgs either.  this might be a bit
  conservative...
- calculate the magnitude of our adjusted based on the max_misplaced
  target.  this assumes a uniform distribution of objects across pgs,
  so not perfectly accurate, but hopefully close enough.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
d6022dd87b mgr/DaemonServer: block pg_num decrease(merge) until pgp_num is reduced
We can't merge until the PGs are stored together.  (The mon would stop
us if we tried, but let's not waste time trying.)

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
155d9a2c00 mgr/DaemonServer: adjust_pgs(): cosmetic change to debug output
Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
128437c579 mon/PGMap: add get_recovery_stats()
This is some of the same info we get in the json dump from
print_summary -> overall_recovery_summary -> recovery_summary.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
edbd592ee4 mgr/balancer: mgr/balancer/max_misplaced -> pg_max_misplaced
Make this a compiled-in option that is consumed by other components as
well as the balancer.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
4fc2af95ce pybind/mgr/mgr_module: add get_option()
get_config() gets a mgr-style config option.  Add get_option which
fetches a compiled-in config option.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
b4c17cca15 mgr/DaemonServer: allow pg_num increases that abort pending merges
If we are waiting for a PG to merge we can't decrease more, but if we
were in the process of merging and an increase is requested, we can
abort the merge by increase pg_num_actual whenever we want.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
b84fa91368 mon/OSDMonitor: resent pre-nautilus client ops on aborted merge
If we are in premerge (pg_num_pending == pg_num - 1) and abort by
increasing pg_num, we the last_force_op_resend_prenautilus since it will
be an interval change for nautlius+.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
48412ce4e4 mon/OSDMonitor: make pgp_num track pg_num more consistently
Previously we were automatically adjusting pgp_num_target on a
pg_num_target change *only* when decreasing pg_num.  Instead, make
pgp_num (continue to) track pg_num if it currently matches.  If it ever
is set differently than pg_num, leave it different (unless/until it
matches again).

This is still slightly weird, but I think in practice it is good enough.
In the rare case that the admin manually sets pgp_num to something
different than pg_num, they probably won't also be using automagic
pg_num adjustment that might make them match and start tracking again.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-16 07:22:48 -05:00
Sage Weil
747c9dcf0a Merge PR into master
* refs/pull/24095/head:
	osd: do not authenticate heartbeat connections until nautilus
	msg,osd: enable unauthenticated Dispatcher for pre-nautilus OSD compat
	osd: add missing space in heartbeat debug output
	mgr/DaemonServer: add missing return
	msg/async: drop verify_authorizer wrapper
	osd: authenticate ping sessions
	msg/simple: remove verify_authorizer wrapper
	msg: remove unused ms_verify_authorizer
	msg/async: remove get_authorizer wrapper
	msg/simple: remove get_authorizer wrapper
	msg/Messenger: pull authenticator validation into Messenger
	msg/Messenger: uninline ms_deliver_verify_authorizer
	mgr/DaemonServer: expose keyring for authenticator verification
	mon: expose keyring for msgr1 authentication
	mds: expose keyring for authenticater verification
	osd: expose keyring for authenticater verification
	msg/Dispatcher: add ms_get_auth1_authorizer_keystore
	mon: fix ref cycle breakage in handle_forward
	mon: use MonOpRequest get_session() instead of PaxosServiceMessage's
	mon: get session from MonOpRequest in handle_command
	messages/MForward: drop unused ctor
	mon: use ms_handle_authentication to parse caps
	mon: kill Session::global_id and use Connection member instead
	mgr/DaemonServer: move session setup into ms_handle_authentication
	mds: move session setup into ms_handle_authentication
	osd: move session setup into ms_handle_authentication
	msg: new ms_handle_authentication, add fields to Connection

Reviewed-by: Ricardo Dias <rdias@suse.com>
2018-10-16 07:20:17 -05:00
Sage Weil
05faeb4d12 Merge PR into master
* refs/pull/24579/head:
	qa/osd: fixup osd-rep-recov-eio.sh fails to parse pg dump

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-16 07:17:59 -05:00
John Spray
e6a26aeff7
Merge pull request from batrick/i36450
qa: fix run call args

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
2018-10-16 13:09:08 +01:00
Kefu Chai
67777bbe7a
Merge pull request from theanalyst/vstart-rgw-asok
vstart: set admin socket for RGW in conf

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-10-16 19:15:35 +08:00
Tatjana Dehler
31863f9638 mgr/dashboard: configs textarea disallow horizontal resize
The textarea allows horizontal and vertical resize by default. Only the
vertical resize is appropriate for this form.

Fixes: http://tracker.ceph.com/issues/36452
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2018-10-16 12:28:16 +02:00
Patrick Nawracay
807f431123 mgr/dashboard: Fix spaces around status labels on OSD list
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-10-16 12:03:48 +02:00
kungf
d9a66ea410 OSD: fix a typo err when increase recvoery bytes statistics
Signed-off-by: kungf <yang.wang@easystack.cn>
2018-10-16 17:22:42 +08:00
Kefu Chai
52706830d6
Merge pull request from cyx1231st/wip-seastar-msgr-refactor
crimson/net: seastar-msgr refactoring

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-10-16 16:44:16 +08:00
Zhi Zhang
552090d71a common: assert if buffer advance length overflow
In the old ceph version, buffer advance length was defined as int, but
in async msg, the real length of data buffer was defined as unsigned.

Occassionly some MDS message back from OSD was too large, which caused
this length overflow and made MDS crash.

For compatibility reason, add an assertion here if buffer advance length
is overflow.

Fixes: http://tracker.ceph.com/issues/36340

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
2018-10-16 14:26:14 +08:00
Kefu Chai
e1fd799b75
Merge pull request from gregsfortytwo/wip-crush-doc
doc: explain 'firstn v indep' in the CRUSH docs

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2018-10-16 13:49:19 +08:00
Patrick Donnelly
2760ddde89
Merge PR into master
* refs/pull/24286/head:
	client: fix fuse client can't read or write data due its caps is invalid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-15 21:34:31 -07:00
Patrick Donnelly
6a4cc58a9d
Merge PR into master
* refs/pull/24292/head:
	qa: add test for rctime on root inode
	mds: set rctime on new system inode
	mds: small refactor

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-10-15 21:31:04 -07:00
Patrick Donnelly
35843a31c8
Merge PR into master
* refs/pull/24486/head:
	client: explicitly show blacklisted state via asok status command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-15 21:23:19 -07:00
Patrick Donnelly
14f23d602b
Merge PR into master
* refs/pull/24508/head:
	cephfs-shell: fixup 'str' object has no attribute 'decode'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-15 21:12:56 -07:00
Alexey Sheplyakov
2eeb48dd33 fix compilation with distcc and other compiler wrappers
When building with ccache, distcc, and other compiler wrappers (such
as STLFilt):

CC='ccache gcc' CXX='ccache g++' cmake /path/to/ceph
make

python modules fail to compile since distutils try to execute the
wrapper itself without specifying the actual compiler.
Although cmake has a special magic switch for compiling with ccache
(cmake -DWITH_CCACHE=ON) other tools (distcc) are not supported, and
specifying the compiler as

CC=/whatever/compiler/is

used to work for decades, and it's a good idea to keep it working

Signed-off-by: Alexey Sheplyakov <asheplyakov@mirantis.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-10-16 11:33:29 +08:00
xie xingguo
d7ff33e9fd crush: get "ceph osd crush class create/rm" back
This reverts a27fd9d25c and
b863883ca7.

Quote form Sébastien Han:
> IIRC at some point, we were able to create a device class from the CLI.
Now it seems that the device class gets created when at least one OSD
of a particular class starts.
In ceph-ansible, we create pools after the initial monitors are up and
we want to assign a device crush class on some of them.
That's not possible at the moment since there no device class available yet.
Also, someone might want to create its own device class.
Something as crazy as running Filestore with a tmpfs osd store and
might want to isolate them.
I know it's a very limited use case, but still, it could be desired.

See also https://www.spinics.net/lists/ceph-devel/msg41152.html

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-10-16 08:45:49 +08:00
Sage Weil
585d21df65 Merge PR into master
* refs/pull/24596/head:
	ptl-tool.py: move githubmap update into merge commit

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2018-10-15 18:41:53 -05:00
Sage Weil
46f32747bd .github/stale.yml: configure probot/stale to automatically close stale issues
Initially only warn about closing issues, but do not close them.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-15 17:37:03 -05:00
Patrick Donnelly
d491227956
qa: fix run call args
Fixes: http://tracker.ceph.com/issues/36450
Introduced-by: 95746ecce9
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-15 14:45:18 -07:00
Patrick Donnelly
28b9b4ea2b
Merge PR into master
* refs/pull/24562/head:
	removed warning for resolved errata

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-15 14:34:22 -07:00
Gregory Farnum
929a9201c8
Merge pull request from libingyang-zte/master
doc: fix typos in doc/releases
2018-10-15 14:17:36 -07:00
Gregory Farnum
9b099583ca
Merge pull request from LenzGr/doc-leads-update
doc/dev: Updated component leads table
2018-10-15 14:15:43 -07:00
Casey Bodley
2c71d4a9ef rgw: add rgw::putobj::HeadObjectProcessor
a striped object processor with special handling for the first chunk of
data

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:06:26 -04:00
Casey Bodley
492dbe3f6b rgw: add rgw::putobj::ObjectProcessor
an abstract interface that deals with logical objects and their update
semantics

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:05:12 -04:00
Casey Bodley
9e1292e86e rgw: add rgw::putobj::AioThrottle
implement the throttling algorithm in terms of rgw::putobj::Aio. this
differs from RGWPutObjProcessor_Aio in that it doesn't wait on the
first pending write to complete before making process on later ones

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:05:12 -04:00
Casey Bodley
dfb16c3fe9 rgw: add rgw::putobj::Aio interface
the Aio operations return a ResultList of previous completions that can
be inspected for the error code and object name (the latter is needed to
track created objects that need to be removed on cancelation)

returning results in a list avoids the extra locking that may be
required to poll/wait for a single completion at a time

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:05:12 -04:00
Casey Bodley
4ba0348611 rgw: add rgw::putobj::StripeProcessor
a pipe that stripes data over multiple objects according to an
abstract StripeGenerator

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:05:06 -04:00
Casey Bodley
594dc4cceb rgw: add rgw::putobj::ChunkProcessor and test
ChunkProcessor turns the input stream into a series of discrete chunks
before forwarding to the wrapped DataProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:01:36 -04:00
Casey Bodley
504b7d9c21 rgw: add rgw::putobj::DataProcessor interface and Pipe
adds an abstract DataProcessor interface (analogous to
RGWPutObjDataProcessor) that allows processors to be composed into
pipelines, and a Pipe class to support the existing filters for
compression and encryption

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:00:08 -04:00
Casey Bodley
b33328ff9b rgw: rgw_user_sync_all_stats uses bucket entrypoint
when bucket reshard completes, rgw_link_bucket() passes the new bucket
instance id down to cls_user, but cls_user_set_buckets_info() does not
change the instance id when it's updating an existing bucket. so when
rgw_user_sync_all_stats() looks up each of the user's buckets, it uses
the original bucket instance id instead of the resharded one and
calculates user stats that may not match the current bucket stats

as a workaround, rgw_user_sync_all_stats() no longer relies on the
bucket instance id it gets from rgw_read_user_buckets(), and instead
calls get_bucket_info() to look up the current instance in the bucket
entrypoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 16:45:16 -04:00
Casey Bodley
e1b97fc2e8 rgw: cls_user_set_buckets_info can update bucket id
if cls_user_set_bucket_info() finds an existing bucket entry, it does
not update its bucket id

Fixes: https://tracker.ceph.com/issues/24505

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 16:45:13 -04:00