Commit Graph

115370 Commits

Author SHA1 Message Date
Kefu Chai
c90bbc0630 crimson/common/tri_mutex: add helpers for seastar::with_lock()
so we don't need to worry about unlocking the object context.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-19 13:30:51 +08:00
Kefu Chai
5fb318fd25 crimson/common: discard failure of finally() block
instead of assuming that the function passed to finally() returns an
erroratorized future, in this change:

* s/safe_then/then_wrapped/ to handle the exception thrown by
  the finally function.
* specialize for the case where the finally function does not return
  a future, and just call it. note, in seastar's implementation of
  finally, `finally_body` is used for specializing these two cases.
* rename "future" to "result", for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-19 12:40:50 +08:00
Patrick Donnelly
7eceaf45de
Merge PR #37202 into master
* refs/pull/37202/head:
	mon: allow overriding the initial mon_host

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-18 18:54:57 -07:00
Kefu Chai
a251ea097d
Merge pull request #37224 from tchaikov/wip-cmake-boost-MPL-list-size
cmake: introduce Boost::MPL interface library for increasing BOOST_MPL_LIMIT_LIST_SIZE

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-19 09:33:31 +08:00
Patrick Donnelly
0275e1c7af
Merge PR #37214 into master
* refs/pull/37214/head:
	mgr/volumes/nfs: Check if orchestrator spec service_id is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-18 17:29:27 -07:00
Patrick Donnelly
22b3d48fa0
Merge PR #37190 into master
* refs/pull/37190/head:
	mon/MonCap: check profile_grants too while checking caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-18 17:27:09 -07:00
Patrick Donnelly
d1a0a6eecc
Merge PR #37148 into master
* refs/pull/37148/head:
	mds/FSMap: do not set legacy_client_fscid after filtering

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-09-18 17:24:38 -07:00
Patrick Donnelly
b64c3784f0
Merge PR #37037 into master
* refs/pull/37037/head:
	mds: fix purge_queue's _calculate_ops is inaccurate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-18 17:23:32 -07:00
Patrick Donnelly
269667b3a2
Merge PR #37218 into master
* refs/pull/37218/head:
	qa: spawn MDS daemons before creating fs

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-18 16:25:59 -07:00
Neha Ojha
8ba0a61a51
Merge pull request #35906 from gregsfortytwo/wip-stretch-mode
Add a new stretch mode for 2-site Ceph clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-18 14:31:45 -07:00
Patrick Donnelly
2aae719653
Merge PR #36957 into master
* refs/pull/36957/head:
	mds: convert stringstream to CachedStackStringStream

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
2020-09-18 11:04:11 -07:00
Michael Fritch
a6c06a9ca7
Merge pull request #37245 from mgfritch/cephadm-extra-ceph-conf-test
mgr/cephadm: fixup expected extra ceph conf test result

Reviewed-by: Adam King <adking@redhat.com>
2020-09-18 11:50:35 -06:00
Michael Fritch
b3205210a1
mgr/cephadm: fixup expected extra ceph conf test result
fix test failure introduced by:
ff7e76348e

```
E           AssertionError: expected call not found.
E           Expected: _run_cephadm('test', 'mon.test', 'deploy', ['--name', 'mon.test', '--reconfig', '--config-json', '-'], stdin='{"config": "\\n\\n[mon]\\nk=v\\n", "keyring": ""}')
E           Actual: _run_cephadm('test', 'mon.test', 'deploy',
['--name', 'mon.test', '--reconfig', '--config-json', '-'],
stdin='{"config": "\\n\\n[mon]\\nk=v\\n", "keyring": ""}', image='')
```

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-09-18 08:54:57 -06:00
Lenz Grimmer
ec05d87432
Merge pull request #34545 from rhcs-dashboard/read_only
mgr/dashboard: Disabling the form inputs for the read_only modals

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2020-09-18 12:59:30 +02:00
Lenz Grimmer
0625509c7f
Merge pull request #37023 from p-se/grafana-many-to-many
mgr/dashboard: Fix many-to-many issue in host-details Grafana dashboard

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
2020-09-18 11:58:47 +02:00
Joshua Schmid
4057f30187
Merge pull request #37059 from guits/guits-cephadm-shell-multiple-mounts
cephadm: support multiple mounts when running interactive shell
2020-09-18 10:57:52 +02:00
Joshua Schmid
1fb9082b09
Merge pull request #36890 from sebastian-philipp/cephadm-extend-ceph.conf
mgr/cephadm: Add extra-ceph-conf
2020-09-18 10:56:24 +02:00
Joshua Schmid
e87a408995
Merge pull request #37135 from sebastian-philipp/cephadm-race-add-host-vs-apply
mgr/cephadm: Fix race between host_add and _apply_all_specs
2020-09-18 10:55:17 +02:00
Joshua Schmid
a0065d4a95
Merge pull request #36969 from votdev/issue_46666_container_spec
cephadm: Introduce 'container' specification to deploy custom containers
2020-09-18 10:50:30 +02:00
Rishabh Dave
e7a721f0d2 mon/MonCap: check profile_grants too while checking caps
When checking if a certain fs subcommand can and should be executed in
FSCommands.cc, check permissions in "profile_grants" too when the caps
for that entity contains a cap profile.

Fixes: https://tracker.ceph.com/issues/47423
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-18 13:23:35 +05:30
Kefu Chai
989af0b7cf
Merge pull request #37158 from tchaikov/wip-no-more-assertDictContainsSubset
mgr/dashboard: replace assertDictContainsSubset() with assertLessEqual()

Reviewed-by: Volker Theile <vtheile@suse.com>
2020-09-18 15:18:31 +08:00
Kefu Chai
12457bab47
Merge pull request #37170 from yaarith/add-smartctl-nvme-dependencies
ceph.spec, debian: add smartmontools, nvme-cli dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-18 13:42:44 +08:00
Kefu Chai
0cf2e89948 cmake: introduce Boost::MPL interface library
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-18 12:26:09 +08:00
Kefu Chai
822b91e4d1 src: Revert "Fix to raise BOOST_MPL_LIMIT_LIST_SIZE from 20 to 30"
This reverts commit 3f4e9a4526.

will fix the FTBFS by introducing a interface library in CMake script

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-18 12:26:09 +08:00
Kefu Chai
709c73ef4a cmake: extract admin/CMakeLists.txt
for better modularity

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-18 12:26:09 +08:00
Yaarit Hatuka
d5528a7e8e ceph.spec.in, debian/control: add smartmontools and nvme-cli dependencies
These packages are needed in order to scrape device health metrics from
devices used by OSD and MON daemons.

smartmontools' smartctl is what we use in order to scrape devices' SMART
attributes and general health metrics.
In addition, we use nvme-cli tool on NVMe devices, which fetches
vendor specific NVMe related health metrics.

Ceph rely on these tools for proper functioning of the underlying layers
of devicehealth mgr module, and other mgr modules which use devicehealth
functionality (such as diskprediction_local, telemetry, dashboard).

Essentially, most of devicehealth commands rely on proper functioning of
smartctl, otherwise they lack the device health metrics.

For example, in case smartctl is missing, the commands:
    ceph device scrape-daemon-health-metrics <who>
    ceph device scrape-health-metrics [<devid>]
will not be able to scrape health metrics, and the command:
    ceph device predict-life-expectancy <devid>
will not provide any meaningful output (since there are no metrics).

In short, when we scrape a device by its daemon (be it an OSD or a MON):
  ceph device scrape-daemon-health-metrics <who>
The devicehealth module command eventually invokes a
block_device_get_metrics() call in either osd/OSD.cc or mon/Monitor.cc,
which wraps calls to both
    block_device_run_smartctl()       (spawns smartctl)
    block_device_run_vendor_nvme()    (spawns nvme)
in common/blkdev.cc.

Minimum version requirements:
'smartmontools' is the package name, which contains two utility
programs: 'smartd' and 'smartctl'. Ceph uses the latter.

Version 6.7 of smartctl first introduced the --json option (beta), which
allows to output the metrics in a JSON format. Since then a few
adjustments were made and the feature officially launched in smartctl
version 7.0.
Since we rely on the JSON format to process the metrics, we must have
smartmontools' smartctl version >= 7.

That said, we choose not to specify smartmontools version here on
purpose, since there might be a scenario where:
We specified smartmontools version to be >= 7.
smartmontools 7 is not available yet in rhel 8 / centos 8.
A user installs via rpm ceph-osd, for example.
smartmontools will not be installed (since version >= 7 is not available
in this repo yet).
Then the user upgrades to 8.3 (which should have smartmontools >= 7),
but smartmontools will not get upgraded (since it's not installed).
In the scenario where we do not specify a version, smartmontools 6.6
will be installed, but it will be upgraded to >= 7 when a user upgrades
(and if it's a fresh installation - version >= 7 would be installed
anyway).

nvme-cli does not have a minimum version.

We use 'Recommends' for both rpm and deb packages since we do not want
the installation to fail in case of conflicts. 'Recommends' weakens the
dependency to be installed in case possible, but ignores it in cases of
conflicts with other dependencies.

It's worth mentioning that smartmontools and nvme-cli dependencies exist
in ceph-container builds.
We add them here for the cases of bare metal installations.

In the future we will add a separate package (with smartmontools and
nvme-cli dependencies) that can be installed on any node (running
rbd-mirror, rgw, mds, mgr, etc.), in order to be able to collect the
health metrics of its devices and offer their life expectancy
prediction.

Fixes: https://tracker.ceph.com/issues/47479
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
2020-09-18 03:25:56 +00:00
Patrick Donnelly
795cf4cecc
mds: convert stringstream to CachedStackStringStream
This is a simple performance refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-17 18:54:55 -07:00
Patrick Donnelly
ff16f5d67c
Merge PR #37163 into master
* refs/pull/37163/head:
	mds: silence warning ‘MDSRank::fs_name’ will be initialized after [-Wreorder]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-17 18:30:11 -07:00
Patrick Donnelly
5126bfca8d
Merge PR #37147 into master
* refs/pull/37147/head:
	mds/FSMap: check parse_role return before filtering

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2020-09-17 18:28:13 -07:00
David Zafman
3a95735010
Merge pull request #36989 from AmnonHanuhov/wip-ObjectStore_EIO_Handling
osd: Got rid of global flag eio_errors_to_process

Reviewed-by: David Zafman <dzafman@redhat.com>
2020-09-17 17:45:42 -07:00
David Zafman
a97a1cd858
Merge pull request #36397 from dzafman/wip-39012
distinguish unfound + impossible to find, vs start some down OSDs to get

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-17 17:30:43 -07:00
Jason Dillaman
251b940f4f
Merge pull request #37132 from lixiaoy1/dirty_cache_feature
librbd: add DIRTY_CACHE in IMPLICIT_ENABLE

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-09-17 19:04:29 -04:00
Jason Dillaman
c4fb0c1dfa
Merge pull request #36586 from MahatiC/wip-ssd-integration
librbd/cache: SSD cache integration framework

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-09-17 17:52:30 -04:00
Kefu Chai
5df5d8f8ff
Merge pull request #37141 from sebastian-philipp/cephadm-fix-rm-util.load_from_store
mgr/cephadm: fix RemoveUtil.load_from_store()

Reviewed-by: Joshua Schmid <jschmid@suse.de>
2020-09-18 00:32:22 +08:00
Patrick Donnelly
61db7a9c2e
qa: spawn MDS daemons before creating fs
This avoids unnecessary MDS_ALL_DOWN messages because the MDS daemons
have not yet been spawned.

Fixes: https://tracker.ceph.com/issues/47518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-17 09:26:38 -07:00
Patrick Donnelly
3428b020fd
Merge PR #37197 into master
* refs/pull/37197/head:
	doc: add "fs authorize" subcommand to ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-17 06:42:33 -07:00
Kefu Chai
f09fb275a7
Merge pull request #37120 from tchaikov/wip-rados-type-hintings
pybind/rados: add more type hintings

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-17 18:40:06 +08:00
Varsha Rao
db7353b4be mgr/volumes/nfs: Check if orchestrator spec service_id is valid
Fixes: https://tracker.ceph.com/issues/47512
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-09-17 16:09:52 +05:30
Kefu Chai
d814c5533b
Merge pull request #37143 from dvanders/dvanders_flush
ceph.in: ignore failures to flush stdout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-17 18:39:09 +08:00
Kefu Chai
8abfc5b7e2
Merge pull request #37100 from rhcs-dashboard/fix-47400-master
ceph: ignore BrokenPipeError when printing help

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-17 18:38:15 +08:00
Kefu Chai
499d0c7f2b
Merge pull request #37045 from tchaikov/wip-crimson-bt
common/BackTrace: extract demangle() out

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-09-17 18:16:06 +08:00
Volker Theile
7219377cfc cephadm: Introduce 'container' specification to deploy custom containers
Fixes: https://tracker.ceph.com/issues/46666

Signed-off-by: Volker Theile <vtheile@suse.com>
2020-09-17 10:04:11 +02:00
Kefu Chai
141f23480e common/BackTrace: let abi::__cxa_demangle() do the malloc
also use the returned length for constructing the string_view to be
appended.

we could reuse the buffer across multiple demangle() call for saving the
calls to malloc()/free(). but the upside of this change is that it's
simpler.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-17 15:24:37 +08:00
Kefu Chai
ea6abcf34c common/BackTrace: extract demangle() out
so it can be reused by crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-17 15:24:37 +08:00
Rishabh Dave
e461a763c8 doc: add "fs authorize" subcommand to ceph man page
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-17 10:06:21 +05:30
Kefu Chai
36d67cf59e
Merge pull request #37187 from kamoltat/patch-1
doc/dev/macos: remove nonexistence flag `--with-toolchain`

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-17 12:04:08 +08:00
Patrick Donnelly
ed3782e60a
mon: allow overriding the initial mon_host
This overrides what the CephContext believes to be the current quorum of
monitors (retrieved from other instances of the MonClient), introduced
by [1]. Tests need to be able to target a specific monitor for
exercising forwarding and other things.

[1] 731e2db9fb
Fixes: https://tracker.ceph.com/issues/47180
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-16 18:34:23 -07:00
Greg Farnum
a8963ccd32 test: remove a now-erroneous "mon add" CLI test from ceph_test_argparse.py
The "mon add" command now lets you pass in arbitrary numbers of strings,
so that you can include locations, so this test is invalid.

I considered updating it to only allow a single non-spaced string, but

datacenter=site1 rack=abc host=host1

is accepted elsewhere, so let's keep that consistent and just remove
this test instead.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2020-09-17 00:37:23 +00:00
Patrick Donnelly
23f9da70a7
Merge PR #37199 into master
* refs/pull/37199/head:
	qa/cephfs: add a note to test_nfs.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-16 16:44:38 -07:00
David Zafman
bb5058074d doc: Add doc and PendingRealeaseNotes
Add help about new list_unfound fields in troubleshooting-pg.rst
Add info in PendingReleaseNotes

Signed-off-by: David Zafman <dzafman@redhat.com>
2020-09-16 16:40:10 -07:00