Commit Graph

125544 Commits

Author SHA1 Message Date
Zac Dover
d27bb21cc1 doc/rados: rewrite index.rst
This PR makes minor alterations to the
text at the beginning of the RADOS Guide.

Most notably, the monitor daemon has been
added to the list of types of daemons that
constitute a Ceph cluster.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2021-07-27 05:16:41 +10:00
Michael Fritch
d9198d8668
cephadm: ceph-volume should raise fsid mismatch
raise an fsid mismatch error when passed differing fsids via `--fsid` and `--config`:

```
self = <tests.test_cephadm.TestCephVolume object at 0x7f1c711961f0>, cephadm_fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f1c713addc0>

    def test_fsid(self, cephadm_fs):
        cv_cmd = ['--', 'inventory', '--format', 'json']
        fsid = '00000000-0000-0000-0000-0000deadbeef'

        cmd = ['ceph-volume', '--fsid', fsid] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        s = get_ceph_conf(fsid=fsid)
        f = cephadm_fs.create_file('ceph.conf', contents=s)

        cmd = ['ceph-volume', '--fsid', fsid, '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        cmd = ['ceph-volume', '--fsid', '10000000-0000-0000-0000-0000deadbeef', '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            err = 'fsid does not match ceph.conf'
            with pytest.raises(cd.Error, match=err):
                cd.command_ceph_volume(ctx)
>               assert ctx.fsid == None
E               AssertionError: assert '10000000-0000-0000-0000-0000deadbeef' == None
E                +  where '10000000-0000-0000-0000-0000deadbeef' = <cephadm.CephadmContext object at 0x7f1c7121c1c0>.fsid
```

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Michael Fritch
2d2bb9f96d
cephadm: add ceph-volume tests
add basic ceph-volume tests for `--fsid`, `--config`, and `--keyring`

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Michael Fritch
b589ae68fa
cephadm: remove get_parm mock
fixture does not need to patch the `get_parm` func

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Kalpesh Pandya
3be2b4bcc5 rgw/notifications: Changing the Multipart Upload notification behavior
Changing the notification behavior in case of Multipart Upload, updating
the related test cases and adding the documentation changes for the same.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2021-07-26 21:22:33 +05:30
Guillaume Abrioux
d69ba6ce58
Merge pull request #42443 from dsavineau/ceph_volume_tests_functional_ssh_args
ceph-volume/tests: update ansible environment variables in tox
2021-07-26 17:08:10 +02:00
Patrick Donnelly
83d252cc30 qa: fold frag confs into conf/mds.yaml
These overrides are standard for all configurations. The config to
enable fragmentation is also long removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-26 07:14:38 -07:00
Kefu Chai
54c3e1b688 *: drop strict_iecstrtoll(const char *str,..)
replace strict_iecstrtoll(const char *str,..) with
strict_iecstrtoll(std::string_view, ..). which is more convenient.
and both of them share the same implementation:

strict_iec_cast<uint64_t>(str, err);

so they are interchangeable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
5d22af9316 common,mon: drop strict_sistrtoll()
strict_sistrtoll() is but an alias of strict_si_cast<uint64_t>(..),
let's just drop the former. there are way too many thin wrappers in
strtol.{h,cc}. they don't offer lots of benefit to us.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
650a0c9363 common/strtol: drop strict_strto*(const char*,..)
they are but proxies to strict_strto*(string_view, ...). and their
callers can just call the latter without any change. so drop these
proxies.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
55ed86bb98 common, mon: use strict_si_cast<>(std::string_view,..)
simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
b6f29f4c1b common/strtol: expose strict_si_cast(std::string_view, ..)
this variant is better than strict_si_cast(const char*), because:

* we can just pass std::string to it, as std::string_view can be
  constructed from a std::string implicitly
* strict_si_cast(std::string_view, ..) is the underlying
  implementation of strict_si_cast(const char*,..), so less
  indirection helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
58c4ece8f3 rgw,mon,common/strtol: use strict_iec_cast(std::string_view, ..)
this variant is better than strict_iec_cast(const char*), because:

* we can just pass std::string to it, as std::string_view can be
  constructed from a std::string implicitly
* strict_iec_cast(std::string_view, ..) is the underlying
  implementation of strict_iec_cast(const char*,..), so less
  indirection helps with the readability.
* rgw,mon: use strict_iec_cast(std::string_view, ..) instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
f608782797
Merge pull request #42403 from BlaineEXE/ceph-volume-use-safer-check-for-has-bluestore-label
ceph-volume: use safer check for bluestore label

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
2021-07-26 18:31:13 +08:00
Kefu Chai
0dfab38c01
Merge pull request #42474 from tchaikov/wip-kv-cleanup
kv/RocksDBStore: always initialize local variable

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-26 18:29:01 +08:00
Igor Fedotov
8c49d56b93
Merge pull request #42456 from ifed01/wip-ifed-alloc-histogram2
os/bluestore: implement allocation perf histogram.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-26 13:21:28 +03:00
Akanksha Chaudhari
2f785b1004 mgr/volumes/fs/operations/group.py: add extra blank line
Fixes:https://tracker.ceph.com/issues/51393
Signed-off-by: Akanksha Chaudhari <astroakanksha24@gmail.com>
2021-07-26 12:23:17 +05:30
Mingxin Liu
c5b38306d7 osd/PGLog: set acceptable rollback_info_trimmed_to for pg of replicated pool
if this PG belongs to a replica pool, since replica pool does not
require rollback, see `pg_pool_t::require_rollback()`, we don't
update the omap entry for noting down the
`on_disk_rollback_info_trimmed_to`. see
`PGLog::_write_log_and_missing()` and `PGLog::_write_log_and_missing()`.
in this case, `read_log_and_missing()` should not be able to find
the value for "rollback_info_trimmed_to" in the omap entries.

because `roll_foward_to()` replays all entries starting at
`rollback_info_trimmed_to`, if it is empty, when the primary osd
is merging the pglog from its peers while peering, it would have
to walk through *all* pglog entries. this would incur unnecessary
overhead, and slows down the peering.

in this change, if this PG belongs to a replica pool, we just fast
forward it to `info.last_update` to avoid this overhead.

Signed-off-by: Mingxin Liu <mingxin@xsky.com>
2021-07-26 11:58:11 +08:00
Kefu Chai
4fc80fe094 tools/ceph_objectstore_tool: pass require_rollback to write_log_and_missing()
before this change, `require_rollback` is always true when it is passed
to write_log_and_missing(), which does not hurt. but this diverges from
how `PG::prepare_write()` is implemented. and probably, more important,
this is not correct, as replica pool does not `require_rollback`.

after this change, `require_rollback` is deduced from pgid. so we don't
always pass `true` to `write_log_and_missing()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 11:58:11 +08:00
Kefu Chai
da35cbe6bc
Merge pull request #42466 from adk3798/mgr-stop
mgr/cephadm: Don't allow stopping full mgr, mon or osd services

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-07-25 21:42:29 +08:00
Kefu Chai
c2ec6489fa vstart.sh: set objectstore to "cyanstore" if --cyanstore is specified
in e6ed65db8b, "cyanstore" is added to the
help message of vstart.sh, but we should also check for this option, and
set the ceph option accordingly.

in this change, the option is checked and "objectstore" is updated
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-25 14:28:02 +08:00
Kefu Chai
c4e1b4d7ef crimson/osd: print logging using logger() in main()
more consistent and easier to parse this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-25 11:07:52 +08:00
Kefu Chai
2e3f4689a1 kv/RocksDBStore: always initialize local variable
silences following warning from GCC:

 ../src/kv/RocksDBStore.cc:386:7: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   386 |       if (r < 0) {
       |       ^~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 21:31:15 +08:00
Kefu Chai
d6c931dc78 crimson/osd: cancel timer when stopping
so that reactor does not complain when being stopped, like:

ERROR 2021-07-24 20:50:23,711 [shard 0] seastar - Timer callback failed: seastar::gate_closed_exception (gate closed)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 21:30:54 +08:00
Kefu Chai
9816cb9940 crimson/osd: disable var-tracking-assignments
to disable following warning:

In file included from ../src/common/config.h:27,
                 from ../src/crimson/common/config_proxy.h:8,
                 from ../src/crimson/osd/main.cc:23:
../src/common/config_values.h: In copy constructor ‘ConfigValues::ConfigValues(const ConfigValues&)’:
../src/common/config_values.h:19:7: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
   19 | class ConfigValues {
      |       ^~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 21:30:54 +08:00
Kefu Chai
1c45d3ae02 crimson: use seastar::try_with_gate to guard background task
for less indent, and for more consistency with how tools/store_nbd
runs background task.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 21:30:54 +08:00
Kefu Chai
d686621fe7
Merge pull request #42467 from dsavineau/issue_51796
mgr/cephadm/grafana: check if dashboard is enabled

Reviewed-by: Adam King <adking@redhat.com>
2021-07-24 19:51:25 +08:00
Kefu Chai
7520a391fd
Merge pull request #42468 from dsavineau/refact_mgr_cephadm_iscsi_dashboard_check
mgr/cephadm/iscsi: simplify the dashboard check

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
2021-07-24 19:41:19 +08:00
Kefu Chai
6144acfb32
Merge pull request #41168 from aclamk/wip-bluefs-more-sync-metadata
os/bluestore/bluefs: Force metadata sync after file rename and file deletion

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2021-07-24 19:39:51 +08:00
Kefu Chai
1af0c6195c cmake: define OpenLDAP::OpenLDAP library
* define OpenLDAP::OpenLDAP, so this library can be consumed in a simpler way.
* use OpenLDAP::OpenLDAP instead of OpenLDAP_LIBRARIES when appropriate
* do not link against unused ${OpenLDAP_LIBRARIES}

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:56 +08:00
Kefu Chai
1c3c42fe52 cmake: s/OPENLDAP/OpenLDAP/
the upstream project of OpenLDAP is named "OpenLDAP", so rename OPENLDAP to
OpenLDAP for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:56 +08:00
Kefu Chai
99ba98cef0 cmake: use generator exp. for passing fmt::fmt's COMPILE_DEFINITIONS
more concise this way, without yet another redirection.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:56 +08:00
Kefu Chai
43d10b9e44 cmake: make rgw_common a static library
it was changed from a shared library to object library back in
3c0bba40b2, because we wanted to link both
rgw_a and rgw_admin_user against it.

but now that rgw_admin_user was dropped in
c89b59428e, let's compile rgw_common
as a static library. it'd be much easier for us to apply various compile
options and linkages to it this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:54 +08:00
Kefu Chai
2d955ebb5f cmake: build rgw_kmip_client_impl as part of radosgw
instead compiling rgw_kmip as an object library, just include
rgw_kmip_client_impl.cc in radosgw. simpler this way, as
rgw_kmip_client_impl.cc includes rgw/rgw_common.h. while rgw_common.cc
is indirectly included by ${rgw_libs} as a part of rgw_a, so
it would be simpler if we just compile rgw_kmip_client_impl.cc
as a part of radosgw, and let radosgw link against ${rgw_libs},
which are able to take care of the include directories of 3rd party
libraries, like spawn/spawn.hpp, which is pulled in by rgw/rgw_common.h.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:06 +08:00
Kefu Chai
832c10dfe8 rgw: drop unused rgw_admin_user.{h,cc}
their only consumer was dropped in c89b59428e

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:28:06 +08:00
Kefu Chai
65c65f7cf4
Merge pull request #42460 from MrFreezeex/fix-snaptrim-log
osd: log snaptrim message to dout

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-24 19:13:54 +08:00
Kefu Chai
aecb0bf250
Merge pull request #42448 from tchaikov/wip-rgw-lua-fs
rgw/rgw_lua: s/boost::filesystem/std::filesystem/

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
2021-07-24 19:12:26 +08:00
Kefu Chai
fa8f07561c
Merge pull request #42286 from dvanders/dvanders_selinux
ceph.spec: selinux scripts respect CEPH_AUTO_RESTART_ON_UPGRADE

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-24 13:57:19 +08:00
Kefu Chai
d25cd60776
Merge pull request #42451 from ideepika/wip-jaeger-buildthrift
cmake: fix failure due missing thrift build scripts if building with jaeger

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-24 13:54:40 +08:00
Kefu Chai
ab7e753ab5
Merge pull request #42452 from smithfarm/wip-first-arg-immutable
rpm: drop use of $FIRST_ARG in ceph-immutable-object-cache

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-24 13:53:33 +08:00
Kefu Chai
b278f07861
Merge pull request #42464 from tchaikov/wip-header-only-fmt
cmake, ceph.spec.in: build with header only fmt on RHEL

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-07-24 13:45:54 +08:00
Neha Ojha
c9ad86e9c5
Merge pull request #42438 from tchaikov/wip-qa-test_module_selftest
qa/tasks/mgr: clean crash reports before waiting for clean

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-07-23 15:39:14 -07:00
Adam King
6402c587d0 mgr/cephadm: Don't allow stopping full mgr, mon or osd services
I can't think of any case where we would want to allow this

Fixes: https://tracker.ceph.com/issues/51298

Signed-off-by: Adam King <adking@redhat.com>
2021-07-23 17:04:35 -04:00
Kefu Chai
5419131499 ceph.spec.in: build with header only fmt on RHEL
because fmt is packaged in EPEL, while librados is packaged
in RHEL, so we cannot have fmt as a runtime dependency of librados.
to address this issue, we should compile librados either with static library
or with header-only library of fmt. but because the fedora packaging
guideline does no encourage us to package static libraries, and it would
be complicated to package both static and dynamic library for fmt.

the simpler solution would be to compile Ceph with the header-only
version of fmt.

in this change, we compile ceph with the header-only version of fmt
on RHEL to address the runtime dependency issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 02:26:45 +08:00
Kefu Chai
d81478b569 cmake: add an option "WITH_FMT_HEADER_ONLY"
in this change:

* an interface library named "fmt-header-only" is introduced. it brings
  the support to the header only fmt library.
* fmt::fmt is renamed to fmt
* an option named "WITH_FMT_HEADER_ONLY" is introduced
* fmt::fmt is an alias of "fmt-header-only" if "WITH_FMT_HEADER_ONLY"
  is "ON", and an alias of "fmt" otherwise.

because fmt is packaged in EPEL, while librados is packaged
in RHEL, so we cannot have fmt as a runtime dependency of librados.
to address this issue an option "WITH_FMT_HEADER_ONLY" is introduced, so
that we can enable it when building Ceph with the header version of fmt.
and the built packages won't have runtime dependency of fmt.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 02:26:45 +08:00
Patrick Donnelly
a54e2a66c1
Merge PR #42256 into master
* refs/pull/42256/head:
	mds: update META_POP_READDIR/FETCH/STORE and cache_hit_rate for dump loads

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:18:13 -07:00
Patrick Donnelly
bab1d1fa0c
Merge PR #42173 into master
* refs/pull/42173/head:
	client: flush the mdlog in unsafe requests' relevant and auth MDSes only

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:17:03 -07:00
Patrick Donnelly
0e71ea4a13
Merge PR #42106 into master
* refs/pull/42106/head:
	mds: create file system with specific ID

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:15:33 -07:00
Kefu Chai
0e0d320525 osdc/Objecter: move LingerOp's ctor to .cc
so the linkage of fmt::fmt does not spill out to other compilation
units.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-24 01:01:57 +08:00
Josh Durgin
ce483a14ed
Merge pull request #42131 from tchaikov/wip-doc-dev-ci
doc/dev/continuous-integration: use ceph repos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-07-23 09:35:58 -07:00