Commit Graph

97212 Commits

Author SHA1 Message Date
Kefu Chai
c961e002d2 cmake: check for libpython of the same version of interpreter
actually cython or python3-cython's dependencies should take care of it.
but we should at least get it right on our side if we check it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-04 13:30:48 +08:00
Kefu Chai
d8c13e8604 cmake: use specified python3 version if any
use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.

put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.

this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-04 13:30:48 +08:00
xie xingguo
6a8aedc107 qa: add new test case for pulling error
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-04-04 11:04:43 +08:00
David Zafman
11f072fee1 Add checking of num_shards_repaired in osd stats
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-04-04 11:04:42 +08:00
xie xingguo
202606c26d osd: automatically repair replicated replica on pulling error
However this is not a very complete solution since the broken object
info may still get lost if we switch primaries or simply power off nodes.
I think a better idea would be also adding these kind of broken objects
back into replica's own missing set simultaneously, e.g., like we handling
primary reading errors.

But for now I am not sure if that should be a concern?

Fixes: http://tracker.ceph.com/issues/39101
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-04-04 11:04:42 +08:00
Sage Weil
30e49ae28b common/common_init: start log from common_init_finish (if not yet started)
This captures any non-global_init users who created their cct but haven't
started up the log thread yet.  As long as common_init_finish() happens
after we have all of our config options (from the mon config or whatever),
we will log (or not log) to the right location(s).

Fixes a regression in cd6a5b9c40

Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 21:07:06 -05:00
Patrick Donnelly
c3dfc6811d
msg/async: use faster clear method to delete containers
Erasing each element is unnecessarily slower than just doing a clear after any
necessary iteration.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 17:10:32 -07:00
Patrick Donnelly
49930ad8a3
mds: add Octopus cephfs feature bit
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 15:18:10 -07:00
Patrick Donnelly
a00151ac60
mds: verify current release has a cephfs feature bit
Fixes: http://tracker.ceph.com/issues/39077
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 15:16:42 -07:00
Patrick Donnelly
dcd6e97944
mds: add cephfs feature bit for Nautilus
Fixes: http://tracker.ceph.com/issues/39078
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 15:14:38 -07:00
Patrick Donnelly
50792a0533
mon: generalize and refactor lambda use
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 14:59:07 -07:00
Patrick Donnelly
40c6319a55
qa: test featureful client with mimic base
Fixes: http://tracker.ceph.com/issues/39020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 14:58:57 -07:00
Patrick Donnelly
6168791373
qa: remove obsolete snap upgrade tests
Direct upgrades from Luminous to Octopus are not supported. These snap format
upgrade tests are now only going to be run in the mimic/nautilus branches.

Fixes: http://tracker.ceph.com/issues/39020
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 14:58:11 -07:00
Sage Weil
a6f5bc2c2d Merge PR #27353 into master
* refs/pull/27353/head:
	include/ceph_features: fix typo

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
2019-04-03 16:07:19 -05:00
Ricardo Marques
908c1ac11d
Merge pull request #27293 from tspmelo/wip-nfsv4
mgr/dashboard: Fix NFS pseudo validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
2019-04-03 21:39:37 +01:00
J. Eric Ivancich
444df54099
Merge pull request #26439 from taoCH/wip-fix-librgw-empty-etag
rgw_file: save etag and acl info in setattr

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-04-03 16:11:38 -04:00
Sage Weil
56756c08c8 include/ceph_features: fix typo
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 15:00:16 -05:00
Sage Weil
3c9db396ae Merge PR #27141 into master
* refs/pull/27141/head:
	mon/OSDMonitor: fix osd boot feature vs require_osd_release check
	include/ceph_features: retire 7 other old features
	include/ceph_features: retire ERASURE_CODE_PLUGINS_V2
	include/ceph_features: retire OSD_ERASURE_CODES
	include/ceph_features: update comment to align with N+2 upgrades
	include/ceph_features: adjust whitespace for retired and now usable features
	mon: remove check for jewel mons
	mds/FSMap: remove support for encoding jewel FSMap
	include/ceph_features: enable SERVER_OCTOPUS
	test/cli/osdmaptool/feature-set-unset-list: add octopus to output
	test/cli/osdmaptool/feature-set-unset-list: change unknown feature bit
	qa/releases/octopus.yaml: add octopus upgrade final step
	osd/OSDMap: octopus encoding features
	mon/OSDMonitor: add mon_debug_no_require_octopus
	mon/OSDMonitor: allow 'osd require-osd-release octopus'
	mon: add ondisk incompat octopus feature
	mon/mon_types: add mon feature for octopus
	include/ceph_features: SERVER_O -> SERVER_OCTOPUS

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-04-03 14:59:03 -05:00
Sage Weil
16dd435939 libcephfs: common_init_finish after getting monmap and config
This is how librados behaves.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 14:47:42 -05:00
Tiago Melo
4139d4b58e mgr/dashboard: Fix NFS pseudo validation
Fixes: http://tracker.ceph.com/issues/39063

Signed-off-by: Tiago Melo <tmelo@suse.com>
2019-04-03 19:16:33 +00:00
Sage Weil
1e7629983f Merge PR #27291 into master
* refs/pull/27291/head:
	qa/suites/upgrade/nautilus-x-singleton: upgrade mgrs early
	qa/suites/upgrade/mimic-x-singleton: upgrade mgrs later
	qa/suites: add upgrade/nautilus-x-singleton and symlink from rados/upgrade
	qa/suites: move rados/upgrade/mimic-x-singleton to upgrade/ and symlink

Reviewed-by: Sage Weil <sage@redhat.com>
2019-04-03 14:07:27 -05:00
Tiago Melo
a304f5db9f mgr/dashboard: Fix deletion of NFS transports properties
Fixes: http://tracker.ceph.com/issues/39090

Signed-off-by: Tiago Melo <tmelo@suse.com>
2019-04-03 18:55:37 +00:00
Sage Weil
7bea41cdcf qa/distros/supported/ubuntu_latest: 16.04 -> 18.04
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 13:52:45 -05:00
Sage Weil
60470a3d21 qa/distros/supported/centos_latest: 7.5 -> 7.6
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 13:52:20 -05:00
Sage Weil
7cc6962fda qa/distros: add centos 7.6
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 13:51:49 -05:00
Sage Weil
d667228c2e Merge PR #27146 into master
* refs/pull/27146/head:
	mon/MonMap: add min_quorum_size() helper
	mon/MDSMonitor: add 'mds ok-to-stop' command
	mon: add 'mon ok-to-{stop,add-offline,rm}' commands

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-04-03 13:49:19 -05:00
Ricardo Marques
7f150021a1
Merge pull request #26572 from rhcs-dashboard/wip-37337-master
mgr/dashboard: unify button/URL actions naming

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2019-04-03 19:46:38 +01:00
J. Eric Ivancich
a33ae8c97a
Merge pull request #26183 from letterwuyu/wip-rgw-get-detail-user-info
rgw: Make rgw admin ops api get user info consistent with the command line
2019-04-03 12:36:23 -04:00
J. Eric Ivancich
00ec1eeccb
Merge pull request #26412 from theanalyst/rgw-orphan-fixes
rgw: orphan fixes
2019-04-03 12:34:20 -04:00
Sage Weil
05f56ab67f msg/async/ProtocolV[12]: add ms_learn_addr_from_peer
Optionally learn our address based on what our peer (usually the mon) thinks we are
connecting from, as opposed to what our local socket used for the outbound connection.
This enables us to detect what our NATed address is (as a client) in those environments.

Default to true to match the v1 behavior.  This is a net change in
behavior for v2.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 09:28:02 -05:00
Sage Weil
54d0c2b19a Merge PR #27343 into master
* refs/pull/27343/head:
	mon/LogMonitor: add mon_cluster_log_to_file bool option

Reviewed-by: Sébastien Han <seb@redhat.com>
2019-04-03 09:22:19 -05:00
Ernesto Puerta
b8dc3b5c8a
mgr/dashboard: add wording section to hacking.rst
Fixes: http://tracker.ceph.com/issues/37337
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2019-04-03 16:09:36 +02:00
Ernesto Puerta
3346688394
mgr/dashboard: unify button/URL actions naming
- Mappings (actually an Enum) created for actions (buttons and other UI elements) and URLs: ActionLabels and URLVerbs.
  - An alternative would be to fix/improve the current i18n-polyfill, which only works with literal strings (not even with 'const enums' which become literals after Typescript transpiling).
  - Additionally having a predefined file with some strings to translate (actions, verbs, etc) could improve on the 1st of the 2-stage i18n process (as extraction tool has a lot of limitations).
- A corresponding ActionLabelsI18n service with translated labels (it's a service as I haven't found the way to either translate no-const strings (ngx-translate/AST parser failure) or get a static translator).
  - This services could/should be extended to cover all strings that are defined in static/globally scoped objects before any I18n provider has been initialized.
- Breadcrumbs are not translated (neither were they before this change). This part remains untackled: using 'proxy' static objects and performing live translation could deal with the issue.
- New URLBuilder service created (following a established pattern in the Java/.NET world) . This should avoid the need of messing with literal URLs and string composition/parsing, and while the front-end is not meant to be consumed by anyone, Angular does not provide any other way for the app to navigate between components, so the URLs are a de-facto interface contract. Unlike this approach is not flawless, it's easier to enforce, while issues coming from free-from strings are really hard to catch.
  - This could be further improved by using a router registry/dynamic routing. Most of the routes are trivial.
- As a side effect of these changes, routing module has been refactored and some routes moved to their specific modules (pool, rbd, rgw), via loadChildren and routes.forChild() magic. Now the above mentioned components are lazy-loaded/pre-loaded (it means right after the main code is loaded). This should also decrease the loading time (though probably this is not biggest time eater here).
  - As now modules can be loaded multiple times, not only from App module by means of lazy loading, but also from other ones (as PoolModule loads BlockModule to get QoS widgets in Pool windows), now lazy loaded modules include 2 NgModules (one with imports: RouterModule.forChild(routes), meant for lazy-loading, and another without routes).
- Caveat: Some parts might not be (fully) translated (NFS, iSCSI, mirroring), as there's been ongoing work on them and it's hard to keep up with the new code.
These changes will be a waste of time if the new code does not take benefit from/adheres to it, so I'm still figuring out how to spread this (nothing really fancy to demo). Maybe adding some checks/harnessing to enforce the new naming convention (ideas greatly welcome here).

Fixes: http://tracker.ceph.com/issues/37337
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2019-04-03 16:09:36 +02:00
Sage Weil
3046d17f61 mon/MonmapMonitor: clean up empty created stamp in monmap
Some old clusters have an empty created timestamp.  This is mostly
harmless, but it is confusing/wrong, and it does currently break the
telemetry module with errors like

 ValueError: time data '0.000000' does not match format '%Y-%m-%d %H:%M:%S.%f'

from 'ceph telemetry show'.

If we detect an empty created stamp, look at old monmap and use the oldest
modified stamp we can find.

Fixes: http://tracker.ceph.com/issues/39085
Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 08:52:04 -05:00
Yan, Zheng
1f0dcf2b7f mds: open import bounding dirfrags in batch
Fixes: http://tracker.ceph.com/issues/38679
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 07de3126a8f14f668b3f24a32f3062196e14805d)
2019-04-03 21:41:00 +08:00
Yan, Zheng
f51857772f mds: remove superfluous error in StrayManager::advance_delayed()
Fixes: http://tracker.ceph.com/issues/38679
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2019-04-03 21:40:52 +08:00
Sage Weil
978ac6e2f8 mon/LogMonitor: add mon_cluster_log_to_file bool option
Allow cluster logging to a file to be disabled via a boolean.  Default
to true to avoid any change in behavior.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-04-03 07:18:50 -05:00
Yuval Lifshitz
15ae9c72d2 Merge branch 's2_pubsub_api_new' of https://github.com/yuvalif/ceph into s2_pubsub_api_new 2019-04-03 14:30:29 +03:00
Yuval Lifshitz
98e6a04346 rgw/pubsub: wrong link in S3 doc
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-04-03 14:27:49 +03:00
Kefu Chai
a9b7822d1b osdc/Objecter: use list::push_back(T&&) for less memcpy
also use range-based loop for better readablity.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-03 18:57:26 +08:00
Kefu Chai
f35fa1c58c osdc/Objecter: always add `\0' after strncpy()
strncpy() does not ensure that the dest str is nul terminated, so we
need to add the nul terminator here. another option is strlcpy(), but it
need to link against libbsd if glibc is used.

this change also silences warning like

In function ‘char* strncpy(char*, const char*, size_t)’,
    inlined from ‘virtual void
ObjectOperation::C_ObjectOperation_decodewatchers::finish(int)’ at
../src/osdc/Objecter.h:534:15:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:34: warning:
‘char* __builtin_strncpy(char*, const char*, long unsigned int)’
specified bound 256 equals destination size [-Wstringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos
(__dest));
      |
~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-03 18:56:49 +08:00
Kefu Chai
42f777e67d
Merge pull request #27280 from xiexingguo/wip-39039
mgr: more GIL fixes

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-04-03 18:53:55 +08:00
Yuval Lifshitz
604669ea65
Merge branch 'master' into s2_pubsub_api_new 2019-04-03 13:34:30 +03:00
Yuval Lifshitz
dc904249a0 rgw/pubsub: fix documentation link errors
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-04-03 13:29:47 +03:00
Kefu Chai
148e43ba50 ceph-monstore-tool: fix a typo in error message
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-03 18:06:11 +08:00
Kefu Chai
2b931aaf3f ceph-monstore-tool: print out caps when rebuilding monstore
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-03 18:06:11 +08:00
Zengran Zhang
b039e63c68 kv: make delete range optional on number of keys
Actually, we may only wanna let the objects with real big number of omap to
use delete range. if not, we find too many tombstones will have side-effects
on performance of kv.

refer: https://github.com/facebook/rocksdb/wiki/DeleteRange-Implementation

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
2019-04-03 16:47:36 +08:00
Ricardo Marques
f40c51a1af
Merge pull request #27245 from tspmelo/wip-nfs-typo
mgr/dashboard: Fix typo in NFS form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-04-03 09:44:24 +01:00
Kefu Chai
c3c21bdd0c crimson/osd: lower debug level on i/o path
hopefully it can reduce the contribution of cpu cycles of
PG::wait_for_active()

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-04-03 16:10:38 +08:00
xie xingguo
1d464221d9 msg: rename ms_tcp_read_timeout to ms_connection_idle_timeout
The old naming is confusing, e.g., it actually indicates we should tear
down the underlying connection which has no read/write activities
at both sides (namely connection is idle) for over 15 minutes.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-04-03 15:10:13 +08:00