Commit Graph

125322 Commits

Author SHA1 Message Date
Deepika Upadhyay
00f44782af cmake, ceph.spec, debian: use yaml-cpp >= 0.6
* since focal and centos both have yaml-cpp 0.6 available, which dropped
having boost as it's dependency, moving to 0.6 seems a good upgrade.

* cmake: delete Buildyaml, since distro suppilies v0.6 this is not needed

This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:58:08 +05:30
Deepika Upadhyay
9581902ac5 vstart.sh: use quay to pull jaeger instead of docker
In servers like the ones available in sepia labs, users might hit rate
limiting for docker pull, it is better to use quay image to avoid this
issue.

https://blog.container-solutions.com/dealing-with-docker-hub-rate-limiting

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:56:44 +05:30
Kefu Chai
bb129fc0ed doc/cephfs/journaler: document options using confval directive
better maintainablity this way. and drop unsupported options of

- journaler batch interval
- journaler batch max

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 11:16:56 +08:00
Kefu Chai
ffbc3164d4 cmake: add "mypy" back to tox envlist of "qa""
This reverts commit 286e46578d.

since 0017df2006 has been merged, let's
add mypy back.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 10:09:21 +08:00
Kefu Chai
0017df2006 qa/tasks/vstart_runner: add optional "sudo" param to _run_python()
to silence mypy warnings like:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 10:08:27 +08:00
Kefu Chai
975fd8d6f5
Merge pull request #42284 from orozery/remove-calc-target-vector-copies
osdc/Objecter: avoid vector copies in _calc_target

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-22 09:01:46 +08:00
Kefu Chai
b11d77a178
Merge pull request #42401 from tchaikov/wip-cmake-version
cmake: drop set(VERSION ...)

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
2021-07-22 08:58:10 +08:00
Kefu Chai
2bd034056a
Merge pull request #42408 from tchaikov/wip-mgr-always-on
mon/MgrMonitor: drop nautilus modules from always_on_modules

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-22 08:56:43 +08:00
Kefu Chai
8271916278
Merge pull request #42415 from sebastian-philipp/haproxy-user-root
cephadm: haproxy 2.4 defaults to a different container user.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
2021-07-22 08:55:46 +08:00
Neha Ojha
c9f8846b7f
Merge pull request #41907 from kamoltat/wip-ksirivad-progress-time-interval
pybind/mgr/progress: introduce 5 second sleep interval

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-07-21 16:53:38 -07:00
Patrick Donnelly
4b1e101571
Merge PR #42295 into master
* refs/pull/42295/head:
	doc/cephfs: recover file system after recovering

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-21 13:13:43 -07:00
Neha Ojha
d09a04617e
Merge pull request #42402 from ljflores/perf-counter-docs
doc/dev/perf_counters: update docs to include more context about perf counter usage

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-21 10:40:46 -07:00
Kefu Chai
98d8a9f0c7
Merge pull request #42333 from tchaikov/wip-doc-program
doc/man/8/ceph-volume: specify "program" for subcommands

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-22 01:35:34 +08:00
Casey Bodley
e3a6377099
Merge pull request #42196 from cbodley/wip-qa-rgw-rm-cephadm
qa/rgw: remove rgw_cephadm.yaml from rgw/singleton suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-07-21 13:14:35 -04:00
Kefu Chai
f54574d1ac doc/man/8/ceph-volume: specify "program" for subcommands
as per
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

> Like py:currentmodule, this directive produces no output. Instead, it
> serves to notify Sphinx that all following option directives document
> options for the program called name.
> ...
> The program name may contain spaces (in case you want to document
> subcommands like svn add and svn commit separately).

and to avoid the warnings like:

doc/man/8/ceph-volume.rst:424: WARNING: Duplicate explicit target name:
"cmdoption-ceph-volume-h".

we should specify different "program" for different set of options.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 01:13:54 +08:00
Casey Bodley
255293bd80
Merge pull request #42317 from cbodley/wip-39657
rgw multisite: metadata sync treats all errors as 'transient' for retry

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2021-07-21 13:13:54 -04:00
Casey Bodley
c896b05a6a
Merge pull request #41800 from BryceCao/wip-modfiy-XML-when-deleterole-take-error
rgw : modfiy error XML for deleterole

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-21 13:12:29 -04:00
Casey Bodley
4833b6cb1d
Merge pull request #41858 from BryceCao/wip-del-tenant-parse-in-createrole
rgw : add check for tenant provided in RGWCreateRole

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
2021-07-21 13:11:44 -04:00
Casey Bodley
1acee2ab76
Merge pull request #42361 from cbodley/wip-49747
qa/rgw: add failing tempest test to blocklist

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2021-07-21 13:10:30 -04:00
Casey Bodley
a59b6f2cf5
Merge pull request #35153 from BryceCao/wip-swift-create-bucket-constraint
rgw : skip bucket name detection for system request

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-21 13:09:10 -04:00
Casey Bodley
54235daf52
Merge pull request #35061 from joke-lee/rgw-multisite-forward-policy-accessblock
rgw: forward request in multisite for RGWDeleteBucketPolicy and RGWDeleteBucketPublicAccessBlock

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-21 13:07:34 -04:00
Casey Bodley
1ae13faf75
Merge pull request #42273 from dabukalam/object-lock-retain-date-iso8601
rgw: Dump Object Lock Retain Date as ISO 8601

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2021-07-21 13:06:47 -04:00
Kefu Chai
b4920f071e
Merge pull request #42425 from cbodley/wip-42788
test/rgw: fix use of poll() with timers in unittest_rgw_dmclock_scheduler

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-22 00:13:09 +08:00
Kefu Chai
ec8a40b08f qa/tasks/mgr: clean crash reports before waiting for clean
otherwise we have following warning in health report

{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]}

and it does not disappear after the test waits for 30 seconds.
and the tasks.mgr.test_module_selftest.TestModuleSelftest test
fails like:

2021-07-21T09:59:52.560 INFO:tasks.cephfs_test_runner:======================================================================
2021-07-21T09:59:52.561 INFO:tasks.cephfs_test_runner:ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
2021-07-21T09:59:52.561 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-07-21T09:59:52.561 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-07-21T09:59:52.562 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_6a5d5abc027f706687dec92f92ff6fc6f074d2ae/qa/tasks/mgr/test_module_selftest.py", line 201, in
test_mo
dule_commands
2021-07-21T09:59:52.562 INFO:tasks.cephfs_test_runner:    self.wait_for_health_clear(timeout=30)
2021-07-21T09:59:52.562 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_6a5d5abc027f706687dec92f92ff6fc6f074d2ae/qa/tasks/ceph_test_case.py", line 172, in
wait_for_health_c
lear
2021-07-21T09:59:52.563 INFO:tasks.cephfs_test_runner:    self.wait_until_true(is_clear, timeout)
2021-07-21T09:59:52.563 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_6a5d5abc027f706687dec92f92ff6fc6f074d2ae/qa/tasks/ceph_test_case.py", line 209, in
wait_until_true
2021-07-21T09:59:52.563 INFO:tasks.cephfs_test_runner:    raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-07-21T09:59:52.564 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries

in this change, the crash reports are nuked right after
we see the warning, so that we can have a clean health
report.

Fixes: https://tracker.ceph.com/issues/51743
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-21 22:46:18 +08:00
Neha Ojha
2c528248df
Merge pull request #42410 from ronen-fr/wip-ronenf-standalone-repair
qa/standalone: fixing the timings when waiting for deep-scrub to start

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
2021-07-21 06:57:41 -07:00
Radoslaw Zarzynski
82eb614ce5 crimson/os: let AlienStore adapt MemStore.
This is useful in perf testing and benchmarking as it allows,
through comparison with `CyanStore`, to judge to pentalty of
`AlienStore`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 13:11:16 +00:00
Radoslaw Zarzynski
e6ed65db8b crimson/os: generalize AlienStore beyond BlueStore.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 13:11:16 +00:00
Radoslaw Zarzynski
007133a6ac os: rework the factory of ObjectStore.
The goals here are:

1. make deprecation of `FileStore` easier as creational
   dependencies are segmented into a variant of `create()`
   that could be cut off altogether with `FileStore`.
2. Allow crimson adapt `create()` without burdening it with
   `FileStore`'s dependencies.
3. Simplify the implementation as a bunch of preprocessor
   directives accumulated there over the time.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 13:11:16 +00:00
Radoslaw Zarzynski
d0b8aeeea4 os: make MemStore suitable for AlienStore.
This boils down into 2 things:

1. building `MemStore` for crimson,
2. implementing the crimson-specific variant of `get_omap_values()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 13:11:16 +00:00
Sebastian Wagner
8e513c6c9b
Merge pull request #42430 from tchaikov/wip-cmake-qa-drop-mypy
cmake: drop "mypy" from tox envlist of "qa"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-21 08:58:18 +02:00
Kefu Chai
fa816c0586
Merge pull request #42418 from tchaikov/wip-crimson-nbd
crimson/tools/store_nbd: pass app.alien() down to FSDriver

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 14:49:02 +08:00
Kefu Chai
351e0df4a2
Merge pull request #42367 from tchaikov/wip-cmake-Og
cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

Reviewed-by:  Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2021-07-21 11:38:22 +08:00
Kefu Chai
c9707a989d
Merge pull request #42427 from tchaikov/wip-seastore-metric-cleanup
crimson/os/seastore: trade a map with a plain array

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-07-21 11:25:40 +08:00
Kefu Chai
056065c7e6
Merge pull request #42429 from neha-ojha/wip-51638-cleanup
qa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 11:24:40 +08:00
Kefu Chai
286e46578d cmake: drop "mypy" from tox envlist of "qa"
this change partially reverts 81305b0da9,
otherwise we have following errors:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-21 11:09:20 +08:00
Blaine Gardner
651b28f2e3
ceph-volume: use safer check for bluestore label
Using only the exit status of `ceph-bluestore-tool show-label` to
determine if a device is a bluestore OSD could report a false negative
if there is a system error when `ceph-bluestore-tool` opens the device.

A better check is to open the device and read the bluestore device
label (the first 22 bytes of the device) to look for the bluestore
device signature ("bluestore block device"). If ceph-volume fails to
open the device due to a system error, it is safest to assume the device
is BlueStore so that an existing OSD isn't overwritten.

Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>
2021-07-20 12:24:37 -06:00
Kefu Chai
9e4157b39f crimson/os/seastore: trade a map with a plain array
simpler this way, as we don't need a std::map<> at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:09:41 +08:00
Kefu Chai
eabe3c5cee
Merge pull request #42261 from cyx1231st/wip-seastore-metrics-2
crimson/os/seastore: implement metrics at cache level

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:06:51 +08:00
Kefu Chai
169101e5fe
Merge pull request #42386 from liu-chunmei/seastore-metrics
crimson/seastore: add seastore operation latency metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:05:34 +08:00
Casey Bodley
a8e3589a2c test/rgw: use spawn library for test_rgw_dmclock_scheduler
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2021-07-20 12:50:25 -04:00
Kefu Chai
dc1a8a8b0e
Merge pull request #41929 from sebastian-philipp/fix-qa-tox
qa: Various make check fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 00:36:59 +08:00
Mark Nelson
ac8cf275a6 common/options: Set osd_client_message_cap to 256.
This seems like a reasonable default value based on testing results here:
https://docs.google.com/spreadsheets/d/1dwKcxFKpAOWzDPekgojrJhfiCtPgiIf8CGGMG1rboRU/edit?usp=sharing

Eventually we may want to rethink how the throttles and even how flow control
works, but this at least gives us some basic limits now ( a little higher than
the old value of 100 that we used for many years).

Signed-off-by: Mark Nelson <mnelson@redhat.com>
2021-07-20 16:30:02 +00:00
Kefu Chai
50bff2d39d
Merge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries
doc/man/8/radosgw-admin: fix typo on max entries

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-20 23:40:13 +08:00
Sebastian Wagner
6be59759e3
Merge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom
cephadm: use pyfakefs during test_create_daemon_dirs_prometheus

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
2021-07-20 17:35:09 +02:00
Sebastian Wagner
2cd3fc4c5b
Merge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid
cephadm: infer fsid from ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-20 17:34:15 +02:00
Sebastian Wagner
cb553909d3
Merge pull request #41280 from sebastian-philipp/test_cephadm-stdin
qa/workunits/test_cephadm: Also test stdin

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-07-20 17:33:20 +02:00
Dimitri Savineau
99166e1044 doc/man/8/radosgw-admin: fix typo on max entries
This fixes a typo for the --max-entries option.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2021-07-20 10:49:41 -04:00
Kefu Chai
9f70239ef0 crimson/tools/store_nbd: mark padding variable with [[maybe_unused]]
also silences -Wunused-variable warning from GCC

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 22:48:37 +08:00
Kefu Chai
8ec97762c4 crimson/tools/store_nbd: pass app.alien() down to FSDriver
this was an oversight in 8fad523d41

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 22:48:27 +08:00
Sebastian Wagner
250064bdcb
cephadm: haproxy 2.4 defaults to a different container user.
Another alternative would be to investigage a different setup
leverageing `--sysctl net.ipv4.ip_unprivileged_port_start=0`,
but that would be a larger PR.

Fixes: https://tracker.ceph.com/issues/51355

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-20 16:10:36 +02:00