Commit Graph

116896 Commits

Author SHA1 Message Date
Zac Dover
19d34666a8 doc/dev: update unit test 1 of 7
This commit updates the language at the beginning
of the Unit Test section of the developer guide.
The language in this edit is more grammatical than
it used to be, and should now reduce the reader's
cognitive load.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2020-11-18 18:25:05 +10:00
Kefu Chai
fbf83bb377
Merge pull request #38102 from tchaikov/wip-mgr-boost-cleanup
mgr: do not include unused headers

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-11-18 10:02:08 +08:00
Kefu Chai
7ab70a6466
Merge pull request #38139 from batrick/doc-label
.github: add labeler for documentation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-11-18 10:00:28 +08:00
Kefu Chai
739550f365
Merge pull request #38004 from tchaikov/wip-48058
qa/tasks/ceph: update_archive_setting() only if ctx.archive is valid

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-11-18 09:58:47 +08:00
Josh Durgin
c2b9152eac
Merge pull request #37488 from kamoltat/wip-mgr-progress-turn-off-option
mgr/progress: progress module on/off option

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-11-17 13:37:30 -08:00
Josh Durgin
b6adebdbe7 Merge remote-tracking branch 'wip-resurrect-authorizer-challenges'
Bring back CEPHX_V2 authorizer challenges

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-11-17 15:24:07 -05:00
Patrick Donnelly
127f6f2166
.github: add labeler for documentation
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-11-17 10:36:39 -08:00
Ernesto Puerta
d7d35c2b3b
Merge pull request #38084 from WarriorXK/disable-sso-without-python3-saml
mgr/dashboard: Disable sso without python3-saml

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-11-17 17:05:52 +01:00
Patrick Donnelly
f6639c80ed
Merge PR #35749 into master
* refs/pull/35749/head:
	Update Release notes for multimds scrub
	qa/cephfs: log-ignorelist scrub errors
	qa/cephfs: Add more tests for multimds scrub
	qa/cephfs: add tests for multimds scrub
	qa/cephfs: update existing scrub test cases
	mds: don't skip validating disk state of symlink
	mds: abort/pause/resume scrubs in multiple mds
	mds: track scrub status in multiple mds
	mds: remove on_finish from {CInode,CDir}::scrub_info_t
	Continuation: don't delete self while there are in-processing stages
	mds: auth pin CInode when validating its disk state
	mds: rdlock file/nest lock when accumulating stats of subtree dirfrags
	mds: multiple mds scrub support
	include/frag: add encode/decode functions for fragset_t
	mds: remove object can't be scrubbed immediately from scrub stack
	mds: prevent dirfrag scrub/fragment from running at the same time
	mds: change scrub traverse from post-order to breadth-first search
	mds: make both CInode and CDir as entities of scrub
	mds: remove ScrubStack::scrubstack

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-11-17 07:33:20 -08:00
Yan, Zheng
11a199707d Update Release notes for multimds scrub
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-17 13:12:11 +08:00
Kefu Chai
ff227345eb
Merge pull request #38119 from neha-ojha/wip-core-labels
github/labeler.yml: add bluestore, cephadm labels and more core rules

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-11-17 10:52:43 +08:00
Kefu Chai
72e193cb03
Merge pull request #38096 from tchaikov/wip-cmake-boost-1.74
cmake: add 1.74 to known versions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-11-17 10:48:59 +08:00
Kefu Chai
b9335a1004
Merge pull request #38097 from tchaikov/wip-clang-cleanup
message,log:  mark final classes "final", fix clang related warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2020-11-17 10:47:45 +08:00
Kefu Chai
d5e73a9e93
Merge pull request #38099 from tchaikov/wip-mgr-python3.9
mgr/PyModuleRegistry: do not call PyEval_InitThreads() on python3.9

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-11-17 10:47:07 +08:00
Michael Fritch
884cdd5a1a
Merge pull request #38052 from mgfritch/cephadm-validate-ssh-config
mgr/cephadm: validate user provided ssh_config

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2020-11-16 19:32:43 -07:00
Michael Fritch
75ceac6d99
Merge pull request #38053 from mgfritch/cephadm-placement-err
mgr/cephadm: show failure cause during placement

Reviewed-by: Sebastian Wagner <swagner@suse.com>
2020-11-16 19:31:22 -07:00
Michael Fritch
35d6fde51f
Merge pull request #37901 from votdev/issue_48041_device_light
mgr/cephadm: Allow customizing mgr/cephadm/lsmcli_blink_lights_cmd per host

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2020-11-16 19:21:06 -07:00
Neha Ojha
b552684364 .github/labeler.yml: add cephadm
Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-11-17 01:23:19 +00:00
Neha Ojha
2ecef028a0 .github/labeler.yml: add bluestore and more core rules
more to come

Signed-off-by: Neha Ojha <nojha@redhat.com>
2020-11-17 01:19:13 +00:00
Jason Dillaman
8d4a8735c3
Merge pull request #37939 from trociny/wip-rbd-nbd-wait-for-terminate
rbd-nbd: fixes and improvements for unmap/detach wait for process terminate

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-11-16 17:10:17 -05:00
Jason Dillaman
b8ca929ea5
Merge pull request #38081 from wjwithagen/wjw-fix-test_mock_CryptoObjectDispatch.cc
test/librbd/crypto: Fixup include order to prevent implicit definitions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-11-16 17:09:25 -05:00
Casey Bodley
c10a7240b6
Merge pull request #38105 from ofriedma/wip-barbican-qa-issue
qa/rgw: fix "cannot create secret" on barbican test

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2020-11-16 09:53:08 -05:00
Kevin Meijer
0c18437d2c mgr/dashboard: Disable sso without python3-saml
Removed the requirement for the python3-saml package when wanting to disable SSO for the dashboard, this is currently relevant since the official container that runs Ceph mgr does not have this package installed.
So when upgrading from an older, non-containerized version, you would be stuck using a non-functional dashboard.

This pull requests changes that and allows the ceph dashboard sso disable command without the requirement of the library so that we SSO can always be disabled again.

Fixes: https://tracker.ceph.com/issues/48237 Signed-off-by: Kevin Meijer <admin@kevinmeijer.nl>
2020-11-16 15:03:02 +01:00
Kefu Chai
9c1614cc22
Merge pull request #38072 from tchaikov/wip-github-labeler-crimson
github/labeler: add "crimson" to labeler.yml

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-11-16 21:22:03 +08:00
Kefu Chai
6441771be5
Merge pull request #38041 from tchaikov/wip-mon-trim-osdmap
mon/OSDMonitor: remove osd_epochs if osd is marked in or out

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
2020-11-16 19:04:16 +08:00
Kefu Chai
64972101f7
Merge pull request #38103 from ybwang0211/link-to-balancer
doc: Fix the broken link to the "balancer"

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-11-16 18:30:53 +08:00
Or Friedmann
7951391ce2 qa/rgw: fix "cannot create secret" on barbican test
fix "cannot create secret" on barbican test

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

Fixes: https://tracker.ceph.com/issues/47799
2020-11-16 11:53:54 +02:00
Kefu Chai
cc3eeef188 log/Log: cast typed pointer to integer before printing it
libfmt does not print pointer if it's not "void*", but "thread_t" is
defined as a pointer pointing to "struct pthread" on FreeBSD, so we need
to cast it either to "void*" or an integer". let's cast it to an integer
so it's more consistent with the output on Linux where thread_t is
defined as an integer.

this change addresses the FTBFS on FreeBSD likee:

In file included from /home/jenkins/workspace/ceph-master-compile/src/log/Log.cc:27:
In file included from /usr/local/include/fmt/format.h:44:
/usr/local/include/fmt/core.h:1043:20: error: invalid application of 'sizeof' to an incomplete type 'pthread'
    static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
                   ^~~~~~~~~
/usr/local/include/fmt/core.h:1259:32: note: in instantiation of function template specialization
'fmt::v7::detail::arg_mapper<fmt::v7::basic_format_context<std::__1::back_insert_iterator<fmt::v7::detail::buffer<char> >, char> >::map<pthread>' requested here
  return arg_mapper<Context>().map(val);
                               ^
/usr/local/include/fmt/core.h:1408:23: note: in instantiation of function template specialization 'fmt::v7::detail::make_arg<true,
fmt::v7::basic_format_context<std::__1::back_insert_iterator<fmt::v7::detail::buffer<char> >, char>, fmt::v7::detail::type::int_type, pthread *, 0>' requested here
        data_{detail::make_arg<
                      ^
/usr/local/include/fmt/core.h:1764:10: note: in instantiation of member function 'fmt::v7::format_arg_store<fmt::v7::basic_format_context<std::__1::back_insert_iterator<fmt::v7::detail::buffer<char>
>, char>, pthread *const, char *>::format_arg_store' requested here
  return {args...};
         ^
/usr/local/include/fmt/core.h:1835:31: note: in instantiation of function template specialization 'fmt::v7::detail::make_args_checked<pthread *const &, char *, char [10], char>' requested here
  const auto& vargs = detail::make_args_checked<Args...>(format_str, args...);
                              ^
/home/jenkins/workspace/ceph-master-compile/src/log/Log.cc:376:23: note: in instantiation of function template specialization 'fmt::v7::format<char [10], pthread *const &, char *, char>' requested
here
    _log_message(fmt::format("  {} / {}", pthread_id, (char*)pthread_name), true);
                      ^
/usr/include/sys/_pthreadtypes.h:46:8: note: forward declaration of 'pthread'
struct pthread;
       ^
1 error generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 17:24:44 +08:00
ybwang0211
1144ebdf53 doc:Fix the web link that jumps to the "balancer"
In "https://docs.ceph.com/en/latest/rados/operations/upmap/#upmap", the link to "balancer" was not set correctly, so we fixed it.

Signed-off-by: wangyingbin <wangyingbin@inspur.com>
2020-11-16 16:59:24 +08:00
Kefu Chai
662e64ef5d mgr: do not include unused header files
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 16:37:17 +08:00
Kefu Chai
24e57a211b mgr: only include necessary headers
for two reasons:

* faster compilation with less source files to process
* avoid including "boost/detail/iterator.hpp", so we can
  silence the warning like:

boost/detail/iterator.hpp:13:1: note: ‘#pragma message: This header is
deprecated. Use <iterator> instead.’
   13 | BOOST_HEADER_DEPRECATED("<iterator>")
      | ^~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 16:37:17 +08:00
Kefu Chai
28985555f8 mgr/PyModuleRegistry: do not call PyEval_InitThreads() on python3.9
PyEval_InitThreads() and PyEval_ThreadsInitialized() are deprecated
since python3.9, and GIL is initialized by Py_Initialize() in python3.9.
so do not call them unless we are using python < 3.9.

see https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 15:42:37 +08:00
Kefu Chai
7ba8a611e9 log/Log: cast parameter of syslog() from unsigned to int
silences warning like:

../src/log/Log.cc:325:34: warning: field precision should have type 'int', but argument has type 'std::basic_string_view::size_type' (aka 'unsigned long') [-Wformat]
    syslog(LOG_USER|LOG_INFO, "%.*s", s.size(), s.data());
                               ~~^~   ~~~~~~~~
1 warning generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 15:38:26 +08:00
Kefu Chai
c39590ba40 message: mark message classes "final"
silences warnings like:

src/messages/MOSDPGUpdateLogMissingReply.h:74:34: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~MOSDPGUpdateLogMissingReply() final {}
                                 ^
../src/messages/MOSDPGUpdateLogMissingReply.h:21:7: note: mark 'MOSDPGUpdateLogMissingReply' as 'final' to silence this warning
class MOSDPGUpdateLogMissingReply : public MOSDFastDispatchOp {
      ^
1 warning generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 15:38:26 +08:00
Kefu Chai
b6a94da614 cmake: add 1.74 to known versions
sync with
685fa8bec0/Modules/FindBoost.cmake

for boost 1.74 support

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 15:36:09 +08:00
Kefu Chai
8eeb7da944 cmake: enable CMP0074
to search prefixes specified by the <PackageName>_ROOT CMake
variable and the <PackageName>_ROOT environment variable. find_package()
in CMake 3.12 started to use these prefixes if it is enabled.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-11-16 15:36:08 +08:00
Sridhar Seshasayee
b637909a91
Merge pull request #38007 from sseshasa/wip-qa-rados-perf-scheduler
qa/suite/rados: Introduce "scheduler" directory within rados perf suite.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2020-11-16 12:56:09 +05:30
Kamoltat
993bb02b30 mgr/progress: introduce turn off/on feature
progress module can be turned off/on by using
the commands: 'progress off' and 'progress on'

As well as refractoring teuthology test suite
to prevent future bugs that can possibly occur

fixes: https://tracker.ceph.com/issues/47238

Signed-off-by: kamoltat <ksirivad@redhat.com>
2020-11-16 03:46:42 +00:00
Yan, Zheng
c42570a329 qa/cephfs: log-ignorelist scrub errors
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:10:03 +08:00
Sidharth Anupkrishnan
ba9b85c481 qa/cephfs: Add more tests for multimds scrub
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
904c959df4 qa/cephfs: add tests for multimds scrub
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
31deca1fec qa/cephfs: update existing scrub test cases
Now scrub is always async.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
fbac6bc62e mds: don't skip validating disk state of symlink
we can check if backtrace is valid and check if inode number is in-use

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
ff10bdbe16 mds: abort/pause/resume scrubs in multiple mds
Limit scrub abort/pause/resume commands to mds.0. mds.0 sends messages
to other mds, asks them to abort/pause/resume scrubs.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
ad5471f4cc mds: track scrub status in multiple mds
Scrubs are always initialized from mds.0. So mds.0 can ensure that scrub
tags are unique globally. mds.0 periodically gathers scrubs running in
itself and in other mds. A scrub is finished only if it's not running in
any mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
03908aa043 mds: remove on_finish from {CInode,CDir}::scrub_info_t
A CInode/CDir is scrubbed no longer means corresponding subtree is fully
scrubbed. The on_finish in {CInode,CDir}::scrub_info_t become useless.
This patch also removes code that flushs journal if scrub has repaired
anything.  Later patch will add the code back at different place.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:18 +08:00
Yan, Zheng
86fb5b48a8 Continuation: don't delete self while there are in-processing stages
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:17 +08:00
Yan, Zheng
f01ebadeda mds: auth pin CInode when validating its disk state
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:17 +08:00
Simon Gao
c8c3ba439a mds: rdlock file/nest lock when accumulating stats of subtree dirfrags
Signed-off-by: Simon Gao <simon29rock@gmail.com>
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:17 +08:00
Simon Gao
e5c4e8c0b0 mds: multiple mds scrub support
If a non-auth object is encountered during scrubbing, forward scrub
to the object's auth mds.

Fixes: https://tracker.ceph.com/issues/12274
Signed-off-by: Simon Gao <simon29rock@gmail.com>
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
2020-11-16 09:02:17 +08:00