Commit Graph

125206 Commits

Author SHA1 Message Date
Patrick Donnelly
a54e2a66c1
Merge PR #42256 into master
* refs/pull/42256/head:
	mds: update META_POP_READDIR/FETCH/STORE and cache_hit_rate for dump loads

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:18:13 -07:00
Patrick Donnelly
bab1d1fa0c
Merge PR #42173 into master
* refs/pull/42173/head:
	client: flush the mdlog in unsafe requests' relevant and auth MDSes only

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:17:03 -07:00
Patrick Donnelly
0e71ea4a13
Merge PR #42106 into master
* refs/pull/42106/head:
	mds: create file system with specific ID

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-23 11:15:33 -07:00
Josh Durgin
ce483a14ed
Merge pull request #42131 from tchaikov/wip-doc-dev-ci
doc/dev/continuous-integration: use ceph repos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-07-23 09:35:58 -07:00
Kefu Chai
9054b1a7f7
Merge pull request #42157 from markhpc/wip-osd_client_message_cap
common/options: Set osd_client_message_cap to 256.

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-23 20:58:53 +08:00
Kefu Chai
ade690b549
Merge pull request #42449 from kevinzs2048/wip-test-sanitize
qa/run-tox-mgr-dashboard: Do not write to /tmp/test_sanitize_password…

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-23 14:18:11 +08:00
Kefu Chai
042aea0ad2
Merge pull request #41794 from ceph/dependabot/pip/src/ceph-volume/plugin/zfs/pip-19.2
ceph-volume: bump pip in /src/ceph-volume/plugin/zfs

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-23 10:55:29 +08:00
Kefu Chai
8706aa15d0 ceph-volume: bump pip in /src/ceph-volume/plugin/zfs
Bumps [pip](https://github.com/pypa/pip) from 9.0.1 to 19.2.
- [Release notes](https://github.com/pypa/pip/releases)
- [Changelog](https://github.com/pypa/pip/blob/main/NEWS.rst)
- [Commits](pypa/pip@9.0.1...19.2)

---
updated-dependencies:
- dependency-name: pip
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-23 09:57:35 +08:00
Kefu Chai
af79875460
Merge pull request #42255 from tchaikov/wip-cmake-mgr-cleanup
cmake: install mgr module separately

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-23 09:51:52 +08:00
Kefu Chai
b82b5f963d
Merge pull request #42437 from rzarzynski/wip-crimson-generic-alienstore
crimson/os: generalize AlienStore beyond BlueStore.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
2021-07-23 09:47:53 +08:00
Casey Bodley
dc6e6aee83
Merge pull request #42455 from dang/wip-dang-zipper-get_obj_state
RGW - Zipper - Make default args match in get_obj_state

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-22 13:23:56 -04:00
Adam C. Emerson
9c00a343e0
Merge pull request #42247 from pritha-srivastava/wip-rgw-sts-session-policy-fixes
rgw/sts: createbucket op should take session_policies into account

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2021-07-22 11:59:29 -04:00
Casey Bodley
fb60094c97
Merge pull request #42249 from kalebskeithley/rgw-51599
rgw: cleanup virtual dtor decls and defns

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-07-22 11:52:02 -04:00
Daniel Gryniewicz
664fe75c2a RGW - Zipper - Make default args match in get_obj_state
The default arg for follow_olh was different between the base class and
the Rados class.  Fix it so they're the same.

Fixes: https://tracker.ceph.com/issues/51677
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2021-07-22 11:48:08 -04:00
Neha Ojha
54b41f71d9
Merge pull request #42365 from PracticalMetal/typo
doc: Fix typos and grammar in ecbackend.rst

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-22 08:46:33 -07:00
Patrick Donnelly
4ee63174e6
Merge PR #42431 into master
* refs/pull/42431/head:
	cmake: add "mypy" back to tox envlist of "qa""
	qa/tasks/vstart_runner: add optional "sudo" param to _run_python()

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-22 07:56:40 -07:00
Patrick Donnelly
e30663a18c
Merge PR #42444 into master
* refs/pull/42444/head:
	doc/cephfs/journaler: document options using confval directive

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-22 07:54:52 -07:00
Casey Bodley
c011af6903
Merge pull request #42380 from adamemerson/wip-51712
rgw: radosgw-admin errors if marker not specified on data/mdlog trim

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-22 10:19:17 -04:00
Milind Changire
6edebf8a1f
Merge pull request #42329 from vshankar/wip-cephfs-mirror-dir-remove-registery
cephfs-mirror: record directory path cancel in DirRegistry

Reviewed-by: Milind Changire <mchangir@redhat.com>
2021-07-22 18:20:59 +05:30
Kefu Chai
3117a6c3dd
Merge pull request #42450 from tchaikov/wip-debian-jaeger
debian/control: depend on libjaeger only if <pkg.ceph.jaeger>

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 20:25:46 +08:00
Kevin Zhao
d04ef800ab qa/run-tox-mgr-dashboard: Do not write to /tmp/test_sanitize_password.txt file
To allow running multiple instances of the same tests.

Fixes: https://tracker.ceph.com/issues/51792

Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
2021-07-22 13:20:29 +01:00
Sebastian Wagner
1f8560816e
Merge pull request #42433 from sebastian-philipp/fix-gen-keepalive-config
mgr/cephadm: Fix haproxy not being recognized as a proper daemon

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Asbjørn Sannes <asbjorn.sannes@interhost.no>
2021-07-22 12:04:54 +02:00
Kefu Chai
8e1edd746f debian/control: depend on libjaeger only if <pkg.ceph.jaeger>
otherwise we'd have following failure when trying to install ceph
packages if they are not built with pkg.ceph.jaeger profile:

ceph-common : Depends: libjaeger (= 17.0.0-6321-g62349ba4-1focal) but it is not installable

this change should address the regression introduced by
4b69d3a184

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 17:59:51 +08:00
Sebastian Wagner
a8f1cf2edb
mgr/cephadm: Fix haproxy not being recognized as a proper daemon
Turns out daemon types != service types:

    cephadm [WRN] Found unknown service type haproxy on host smithi019
    cephadm [WRN] Found unknown service type keepalived on host smithi019

leading to `self.mgr.cache.get_daemons_by_service(spec.service_name())`
not returning any daemons.

Fixes: https://tracker.ceph.com/issues/51311

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-22 10:55:54 +02:00
Kefu Chai
e3245d12a0 cmake: s/NPM/NPM_EXECUTABLE/
so the naming is more consistent with other executables found by
find_program().

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:35:57 +08:00
Kefu Chai
6d17e684ec cmake: use ${NPM} instead of npm
so it is more explicit that ${NPM} is used when NODEENV is not
specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:35:47 +08:00
Kefu Chai
e4f1de64d4 cmake: simplify the regex for matching npm command
npx command was added in 3fe39d4899,
but we don't use npx anymore, so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:35:40 +08:00
Kefu Chai
993e250682 cmake: push dashboard build logic down
it's simpler to build dashboard in dashboard/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:35:30 +08:00
Kefu Chai
7633a97a15 cmake: push frontend build logic down
it's simpler to build frontend in frontend/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:35:12 +08:00
Kefu Chai
73e983cfe8 pybind/mgr/dashboard: build frontend/dist in build
to enable us to build out of source, we should build the frontend
artifacts out of src. in this change:

* cmake:
  - install frontend/dist from build
  - drop rules to exclude unused frontend artifacts.
  - pass `--output-path` option to ng
  - copy `frontend/package.json` to build so that the frontend
    can find it
* pybind/mgr/dashboard/module.py: fall back to build directory
  if frontend/dist is not found.
* pybind/mgr/dashboard/__init__.py: use frontend/dist when performing
  unit test.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 16:33:33 +08:00
Pulkit Mittal
8f84f121e1 doc: fix typo and grammar
Signed-off-by: Pulkit Mittal <2pulkit2@gmail.com>
2021-07-22 13:48:07 +05:30
Kefu Chai
7048172cad
Merge pull request #38783 from ideepika/wip-osd-tracing
cmake/ninja: support ninja for jaegertracing

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-22 15:33:13 +08:00
Kefu Chai
a4d654b9c0 cmake: use PATTERN instead of REGEX to exclude installed files
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 13:45:04 +08:00
Kefu Chai
dfd3deb5b2 cmake: install mgr module separately
instead of installing the whole directory, install mgr modules
individually, so it's more clear that what EXCLUDE pattern is
used in which mgr module.

this improves the maintability and readability. in future, we
should only install the artifacts. with this change, we can
specify the include pattern only for dashboard mgr module so
that, for instance, only .js, .html and .css files are installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 13:44:34 +08:00
Deepika Upadhyay
5f5de4ccb5 cmake: fix jaeger linking to ceph libraries
updates:
* custom target jaeger_base
* target jaeger-base which encapsulates all jaeger libs
* include external libraries path needed linking libraries and including
  opentracing and jaegertracing headers files

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:09:09 +05:30
Deepika Upadhyay
a92a7aa236 cmake: change debian DESTDIR for installing jaeger deps
debian uses debian/tmp as destination dir for installing build files,
but since we are using common path($build_dir/external) available both
for rpm and debian based dependency installation, it becomes far more
complicated to maintain include/link path for these external projects.

elaborating on it:
path we are configuring for both rpm and debian installing including,
and linking of external librarires:
/build/ceph-17.0.0-5779-g928f9e55/obj-x86_64-linux-gnu/external/
debian appends DESTDIR to this path, and hence our predefined target
artificats cannot find correct path for external libs, I tried adding
ENV${DESTDIR} so that it could include correct external lib install
path, but it still cannot find them:

failed to link in case of:
-  install(DIRECTORY $ENV{DESTDIR}${CMAKE_BINARY_DIR}/external/include/jaegertracing
-                $ENV{DESTDIR}${CMAKE_BINARY_DIR}/external/include/opentracing
-          DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
-  include_directories(SYSTEM ${CMAKE_INSTALL_INCLUDEDIR}/jaegertracing)
-  include_directories(SYSTEM ${CMAKE_INSTALL_INCLUDEDIR}/opentracing)

-- Installing: /build/ceph-17.0.0-5790-g6bc03cbd/debian/tmp/build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/external/include/jaegertracing/Tracer.h

cd /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/src && /usr/bin/c++  -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/src/include -I/build/ceph-17.0.0-5790-g6bc03cbd/src -isystem /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/boost/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/obj-x86_64-linux-gnu/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/xxHash -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/rapidjson/include -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/include/jaegertracing -isystem /build/ceph-17.0.0-5790-g6bc03cbd/src/include/opentracing  -g -O2 -fdebug-prefix-map=/build/ceph-17.0.0-5790-g6bc03cbd=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -U_FORTIFY_SOURCE -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -fstack-protector-strong -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -std=c++17 -o CMakeFiles/common-objs.dir/mds/mdstypes.cc.o -c /build/ceph-17.0.0-5790-g6bc03cbd/src/mds/mdstypes.cc
In file included from /build/ceph-17.0.0-5790-g6bc03cbd/src/osd/OpRequest.h:21,
                 from /build/ceph-17.0.0-5790-g6bc03cbd/src/osd/OpRequest.cc:3:
/build/ceph-17.0.0-5790-g6bc03cbd/src/common/tracer.h:10:10: fatal error: jaegertracing/Tracer.h: No such file or directory

Since the install path is in our build environment for these librarires,
skipping DESTDIR looks to me hacky fix, but does the job.
with empty destdir:

-- Installing: /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/external/include/jaegertracing/Tracer.h

cd /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/src/mon && /usr/bin/c++  -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/src/include -I/build/ceph-17.0.0-5791-gb97b9640/src -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/boost/include -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/include -isystem /build/ceph-17.0.0-5791-gb97b9640/src/xxHash -isystem /build/ceph-17.0.0-5791-gb97b9640/src/rapidjson/include -isystem /build/ceph-17.0.0-5791-gb97b9640/obj-x86_64-linux-gnu/external/include -isystem /build/ceph-17.0.0-5791-gb97b9640/src/rocksdb/include  -g -O2 -fdebug-prefix-map=/build/ceph-17.0.0-5791-gb97b9640=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC   -U_FORTIFY_SOURCE -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -fstack-protector-strong -fdiagnostics-color=auto -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -std=c++17 -o CMakeFiles/mon.dir/MgrMonitor.cc.o -c /build/ceph-17.0.0-5791-gb97b9640/src/mon/MgrMonitor.cc

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:09:09 +05:30
Deepika Upadhyay
ffc3764606 cmake: adds BUILD_BYPRODUCT for external build jaegertracing libs
* adds BUILD_BYPRODUCT which tells ninja which library will be generated
after the build(needed for dependent build libs)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:09:09 +05:30
Deepika Upadhyay
1e2278e158 cmake: minor reorder of boost path in jaeger build external projet
* Boost is a dependency for jaeger, to use the right version, we pass
ceph build boost path to cmake jaeger build step

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:08:17 +05:30
Deepika Upadhyay
2eea50800e cmake: target name Jaeger >> jaegertracing
since jaegertracing is the original target that jaeger submodule uses in
it's cmake, cmake build complained if named otherwise

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:02:18 +05:30
Deepika Upadhyay
f51480b223 cmake: move ExternalProjectHelper & IncludeJaeger to BuildJaeger
set_library_properties_for_external_project assists with setting right target
properties for all jaeger dependencies.
IncludeJaeger would take care of linking and creating these targets
having them spread out when they are highly coupled seems not optimal.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:01:08 +05:30
Deepika Upadhyay
be43f6133c debian, install-deps.sh: use Build-Profiles to optionally build with jaeger
we will now be using BuildProfileSpec based optional pkg building,
removing comment from .install file is no longer needed If
pkg.ceph.jaeger is enabled debian/control, it shall work to install
jaeger and it's dependencies.

  https://wiki.debian.org/BuildProfileSpec.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:01:06 +05:30
Deepika Upadhyay
84ad544433 cmake: add find_package(thrift) support for thrift 0.13
adds Findthrift.cmake which is used to find thrift 0.13, as a dependency
for building jaegertracing

* bump up submodule version for jaeger-client-cpp for thrift compiler
removal

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:59:48 +05:30
Deepika Upadhyay
80e82686eb ceph.spec, cmake, debian: use thrift 0.13+ from distro pkg
the change to build and ship libthift was added when we didn't have 0.13.0
version shipped via distro pkgs, now that centos 8 and F34 supports req.
version, we do not need to build and ship it with jaeger library.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:59:45 +05:30
Kefu Chai
1a5bcbb335
Merge pull request #42342 from dsavineau/cephadm_template_jinja2_lint
mgr/cephadm/templates: add jinja2 lint

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-22 13:28:33 +08:00
Deepika Upadhyay
e92b4745ba ceph.spec: enable WITH_JAEGER=ON if --with-jaeger passed
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:58:12 +05:30
Deepika Upadhyay
00f44782af cmake, ceph.spec, debian: use yaml-cpp >= 0.6
* since focal and centos both have yaml-cpp 0.6 available, which dropped
having boost as it's dependency, moving to 0.6 seems a good upgrade.

* cmake: delete Buildyaml, since distro suppilies v0.6 this is not needed

This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:58:08 +05:30
Deepika Upadhyay
9581902ac5 vstart.sh: use quay to pull jaeger instead of docker
In servers like the ones available in sepia labs, users might hit rate
limiting for docker pull, it is better to use quay image to avoid this
issue.

https://blog.container-solutions.com/dealing-with-docker-hub-rate-limiting

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:56:44 +05:30
Kefu Chai
bb129fc0ed doc/cephfs/journaler: document options using confval directive
better maintainablity this way. and drop unsupported options of

- journaler batch interval
- journaler batch max

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 11:16:56 +08:00
Kefu Chai
ffbc3164d4 cmake: add "mypy" back to tox envlist of "qa""
This reverts commit 286e46578d.

since 0017df2006 has been merged, let's
add mypy back.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 10:09:21 +08:00
Kefu Chai
0017df2006 qa/tasks/vstart_runner: add optional "sudo" param to _run_python()
to silence mypy warnings like:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-22 10:08:27 +08:00