Commit Graph

112597 Commits

Author SHA1 Message Date
Jason Dillaman
ae6dd86b22 librbd: add 'write_zeroes' public C/C++ API methods
Unlike the existing 'discard' option which is more of a hint to
attempt to release space, the new 'write_zeroes' APIs will ensure
that the entire provided extent is fully zeroed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-06-30 17:53:24 -04:00
Kefu Chai
1169dee259
Merge pull request #35787 from tchaikov/wip-46208
install-deps,cmake: use boost v1.73

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
2020-06-28 21:19:44 +08:00
Kefu Chai
0852df4e78
Merge pull request #35771 from sebastian-philipp/mypy-0.782
qa,src: update mypy to 0.782

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
2020-06-28 18:40:16 +08:00
Kefu Chai
a3c34dc572
Merge pull request #35763 from rzarzynski/wip-test-unused-var-in-test_ino_release_cb
test: fix unused variable warning in test_ino_release_cb.cc.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-28 18:39:10 +08:00
Kefu Chai
8c4e636c28
Merge pull request #35666 from inspur-wyq/wip-debian-cmake-version
debian: update cmake version to cmake_minimum_required(VERSION 3.10.2)

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-28 18:38:26 +08:00
Kefu Chai
5ccfc2d5d3
Merge pull request #35750 from rzarzynski/wip-common-config-dissect-boolcast
common, crimson: reuse the interpreter of Option::TYPE_BOOL

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-27 23:17:42 +08:00
Kefu Chai
d22536af5f
Merge pull request #35512 from ZhenLiu94/liuz02
src/mon/MonCommands.h: Remove the redundant right parenthesis in the help message

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-27 23:16:41 +08:00
Kefu Chai
2fa14e2daa
Merge pull request #35760 from tchaikov/wip-mgr-track-pending-module
mgr: avoid false alarm of MGR_MODULE_ERROR

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
2020-06-27 23:15:28 +08:00
Kefu Chai
66bf260267 json_spirit: avoid using bind placeholders in global namespace
to silence the warning from boost v1.73, like

json_spirit/json_spirit_reader.cpp:7:
/opt/ceph/include/boost/bind.hpp:36:1: note: ‘#pragma message: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> +
using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.’
   36 | BOOST_PRAGMA_MESSAGE(
      | ^~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 22:43:03 +08:00
Kefu Chai
eb5344a5d9 make-dist: redistribute boost 1.73
Fixes: https://tracker.ceph.com/issues/46208
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 22:43:03 +08:00
Sridhar Seshasayee
8b33a29f3a
Merge pull request #35490 from sseshasa/wip-grace-interval-timer-reset
mon/OSDMonitor: Reset grace period if failure interval exceeds a threshold.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-06-26 16:56:07 +05:30
Sebastian Wagner
78f3473f55 qa,src: update mypy to 0.782
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-26 12:27:32 +02:00
Kefu Chai
1676317572
Merge pull request #35791 from tchaikov/wip-mock-rados
script/gen_static_command_descriptions: add mock for rados python binding

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-26 18:09:59 +08:00
Kefu Chai
a0194ef92f script/gen_static_command_descriptions: add mock for rados python binding
to address issue like

Traceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/doc/scripts/../../src/script/gen_static_command_descriptions.py", line 168, in <module>
    print(json.dumps(gen_commands_dicts(), indent=2, sort_keys=True))
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/doc/scripts/../../src/script/gen_static_command_descriptions.py", line 163, in gen_commands_dicts
    comms = from_mon_commands_h() + from_mgr_modules()
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/doc/scripts/../../src/script/gen_static_command_descriptions.py", line 85, in from_mgr_modules
    comms = sum([list_mgr_module(name) for name in names], [])
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/doc/scripts/../../src/script/gen_static_command_descriptions.py", line 85, in <listcomp>
    comms = sum([list_mgr_module(name) for name in names], [])
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/doc/scripts/../../src/script/gen_static_command_descriptions.py", line 65, in list_mgr_module
    mgr_mod = __import__(m_name, globals(), locals(), [], 0)
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/src/pybind/mgr/volumes/__init__.py", line 2, in <module>
    from .module import Module
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/src/pybind/mgr/volumes/module.py", line 8, in <module>
    from .fs.nfs import NFSCluster, FSExport
  File "/home/jenkins-build/build/workspace/ceph-pr-docs/src/pybind/mgr/volumes/fs/nfs.py", line 6, in <module>
    from rados import TimedOut
ImportError: cannot import name 'TimedOut'

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 17:48:57 +08:00
Kefu Chai
7fe16eb0e6 cmake: download and build boost 1.73 if it is not installed
Fixes: https://tracker.ceph.com/issues/46208
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 17:36:54 +08:00
Kefu Chai
40f08b3b0e cmake: update FindBoost.cmake for 1.73
adapted from
https://github.com/Kitware/CMake/blob/master/Modules/FindBoost.cmake
commit b76b4dea1cd60e9899fd641602901ded9e0d7a7b

Fixes: https://tracker.ceph.com/issues/46208
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 17:36:54 +08:00
Kefu Chai
a0c91f940b install-deps.sh: use ceph-libboost1.73
Fixes: https://tracker.ceph.com/issues/46208
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-26 17:20:05 +08:00
Jan Fajerski
f199b9370d
Merge pull request #34744 from ceph/rhbz-1827349
ceph-volume: handle idempotency with batch and explicit scenarios
2020-06-26 11:15:59 +02:00
Sébastien Han
4fc128bb9c
Merge pull request #35468 from guits/guits-raw_dmcrypt
ceph-volume: add dmcrypt support in raw mode
2020-06-26 10:38:58 +02:00
Kefu Chai
16473d4124
Merge pull request #35770 from tchaikov/wip-install-deps
install-deps.sh: drop el7 bits and uninstall python-sphinx if sphinx-build is in python2

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-06-26 15:06:48 +08:00
Patrick Donnelly
8415e75682
Merge PR #31726 into master
* refs/pull/31726/head:
	mds: MDS-MDS message classes inherit from MMDSOp
	mds: MMDSOp implementation and detect unversioned messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-25 14:37:30 -07:00
Patrick Donnelly
0e645ce042
Merge PR #34878 into master
* refs/pull/34878/head:
	libcephfs: mark ceph_stat() family functions deprecated

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
2020-06-25 14:35:46 -07:00
Patrick Donnelly
dd56696230
Merge PR #35323 into master
* refs/pull/35323/head:
	tool: change coding style, indent with two spaces

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-25 14:34:53 -07:00
Patrick Donnelly
bba0cb878a
Merge PR #35420 into master
* refs/pull/35420/head:
	mgr/volumes: Fix pool removal on volume deletion

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
2020-06-25 14:33:12 -07:00
Patrick Donnelly
e26ad4e888
Merge PR #35624 into master
* refs/pull/35624/head:
	include: cleanup lru con/des

Reviewed-by: Jos Collin <jcollin@redhat.com>
2020-06-25 14:32:14 -07:00
Patrick Donnelly
4636a1bda8
Merge PR #35664 into master
* refs/pull/35664/head:
	qa: add omit_sudo=False for commands ran with sudo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2020-06-25 14:31:33 -07:00
Guillaume Abrioux
20e8286f5e ceph-volume: remove unused function
This function is never called in raw context, let's remove it.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-06-25 21:53:18 +02:00
Guillaume Abrioux
7bb811c3b6 ceph-volume: add raw testing coverage
This commit adds testing against `ceph-volume raw` subcommand.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-06-25 21:53:14 +02:00
Guillaume Abrioux
159c30d29f ceph-volume: add dmcrypt support in raw mode
This commit adds the dmcrypt support in `ceph-volume raw` mode.

Note about `ceph-volume raw list` change:
Given `lsblk -J` (json output) option isn't available on all OS, I came up with
adding '--inverse' option to the existing command which allows us to get the
mapper devices list in that command output. Not listing root devices containing
partitions shouldn't have side effect since we are in `ceph-volume raw`
context.

example:
running `lsblk --paths --nodeps --output=NAME --noheadings` doesn't allow to
get the mapper list because the output is like following :

$ lsblk --paths --nodeps --output=NAME --noheadings
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd

the dmcrypt mappers are hidden because of the `--nodeps` given they are
displayed as a dependency.

$ lsblk --paths --output=NAME --noheadings
/dev/sda
|-/dev/mapper/ceph-3b52c90d-6548-407d-bde1-efd31809702f-sda-block-dmcrypt
`-/dev/mapper/ceph-3b52c90d-6548-407d-bde1-efd31809702f-sda-db-dmcrypt
/dev/sdb
/dev/sdc
/dev/sdd

adding `--inverse` is a trick to get around this issue, the counterpart is that
we can't list root devices if they contain at least one partition but this
shouldn't be an issue in `ceph-volume raw` context given we only deal with
raw devices.

Fixes: https://tracker.ceph.com/issues/45959

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-06-25 21:50:10 +02:00
Patrick Donnelly
b2453e9c61
Merge PR #35757 into master
* refs/pull/35757/head:
	doc: update cephfs metadata pool recovery procedure

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
2020-06-25 11:47:41 -07:00
Sebastian Wagner
c5239d8ca3
Merge pull request #35537 from sebastian-philipp/cephadm-yaml-ordered-readable
mgr/orch: increase readability for yaml representation
2020-06-25 20:06:25 +02:00
Sebastian Wagner
37d1815f9a
Merge pull request #35274 from jmolmo/issue_45726
cephadm: error trying to get ceph auth entry for crash daemon

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-25 20:05:51 +02:00
Abhishek L
ea0a33719e
Merge pull request #35773 from theanalyst/master-rgw-cors-fixes
rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
2020-06-25 19:25:07 +02:00
Patrick Donnelly
0603aa681a
doc: update cephfs metadata pool recovery procedure
This is mostly clarifying a few things after personally experimenting
with this.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-06-25 09:40:27 -07:00
Sebastian Wagner
947d82f9f8
Merge pull request #35695 from sebastian-philipp/mypy-rm-global-ignore-imports
mypy.ini: Remove global `ignore_missing_imports = True`

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
2020-06-25 17:47:26 +02:00
Sebastian Wagner
47748de355
Merge pull request #35762 from sebastian-philipp/doc-dev-rm-confusing.txt
doc/dev: remove confusing.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-06-25 17:46:49 +02:00
Sebastian Wagner
21a9fb9ae4
Merge pull request #35503 from sebastian-philipp/rm-daemon-ok-to-stop
mgr/cephadm: move ok_to_stop to CephadmService 

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-06-25 17:28:19 +02:00
Sebastian Wagner
d2e7b0e83f
Merge pull request #35761 from sebastian-philipp/qa-cephad-upgrade-no-rgw
qa/cephadm/upgrade: Exclude RGW for now

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-06-25 17:25:50 +02:00
Matt Benjamin
a094e84158
Merge pull request #33228 from galsalomon66/s3select_integration_into_RGW_2
rgw: add s3select support
2020-06-25 11:05:22 -04:00
Sebastian Wagner
a27334c951
Merge pull request #35593 from p-se/wip-pse-cephadm-grafana-fix-version
mgr/cephadm: use fixed Grafana version

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-25 16:55:34 +02:00
Lenz Grimmer
584baf132f
Merge pull request #34606 from rhcs-dashboard/wip-45011-bucket-usage-feature
mgr/dashboard: Display users current bucket quota usage

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-06-25 14:49:01 +02:00
Casey Bodley
1524d3c0c5 rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader
the values in the <ExposeHeader> element are sent back to clients in a
Access-Control-Expose-Headers response header. if the values are allowed
to have newlines in them, they can be used to inject arbitrary response
headers

this issue only affects s3, which gets these values from an xml document

in swift, they're given in the request header
X-Container-Meta-Access-Control-Expose-Headers, so the value itself
cannot contain newlines

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Reported-by: Adam Mohammed <amohammed@linode.com>
2020-06-25 14:02:02 +02:00
Kefu Chai
2d8da82b23 install-deps.sh: always use python3-sphinx to build the docs
python-sphinx creates a symlink pointing ../share/sphinx/scripts/python2/sphinx-build
to /usr/bin/sphinx-build even if python3-sphinx is already installed.
in that case, if python-sphinx is installed after python3-sphinx,
sphinx-build is in python2. and it breaks the build of master, as we are
using python3 syntax in conf.py since
e9e17b9cef.

and we are still using python2 as well as "python-sphinx" when building nautilus,
so if a build slave happen to compile nautilus before it is scheduled to
build master or a wip- branch, the doc build fails.

in this change, python-sphinx is uninstalled, if /usr/bin/sphinx-build
is found to be written in python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-25 19:59:41 +08:00
Kefu Chai
c70415b897 install-deps.sh: always use dnf on RHEL/CentOS
as we don't build master on el7 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-25 19:59:41 +08:00
Kefu Chai
95f6f61534 install-deps.sh: drop bits enabling DTS repo
since GCC offered by RHEL8/CentOS8 is able to compile ceph, there is no
need to add repos providing devtoolset anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-25 19:59:41 +08:00
Lenz Grimmer
0e76c3a743
Merge pull request #35419 from votdev/issue_45897_add_host_labels
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2020-06-25 12:51:14 +02:00
Kefu Chai
0b8054166c
Merge pull request #35767 from tchaikov/wip-rpm-scikit-learn
ceph.spec.in: Requires python3-scikit-learn only on fedora and suse

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-06-25 18:36:41 +08:00
Sebastian Wagner
5150e95784 python-common: service_spec.from_json: Add docstring explaining histroy
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-25 12:33:19 +02:00
Sebastian Wagner
1650257015 qa/cephadm: Add test for --format=yaml
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-25 12:33:19 +02:00
Sebastian Wagner
87a150358e mgr/orchestrator: Add test for readable yaml
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-06-25 12:33:19 +02:00