Commit Graph

1317 Commits

Author SHA1 Message Date
Kefu Chai
7048172cad
Merge pull request #38783 from ideepika/wip-osd-tracing
cmake/ninja: support ninja for jaegertracing

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-22 15:33:13 +08:00
Deepika Upadhyay
be43f6133c debian, install-deps.sh: use Build-Profiles to optionally build with jaeger
we will now be using BuildProfileSpec based optional pkg building,
removing comment from .install file is no longer needed If
pkg.ceph.jaeger is enabled debian/control, it shall work to install
jaeger and it's dependencies.

  https://wiki.debian.org/BuildProfileSpec.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 11:01:06 +05:30
Deepika Upadhyay
80e82686eb ceph.spec, cmake, debian: use thrift 0.13+ from distro pkg
the change to build and ship libthift was added when we didn't have 0.13.0
version shipped via distro pkgs, now that centos 8 and F34 supports req.
version, we do not need to build and ship it with jaeger library.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-22 10:59:45 +05:30
Deepika Upadhyay
7aa3c2c3e6 debian/control: update yaml-cpp version to 0.6
This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
```
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-19 21:43:17 +00:00
Deepika
52fc62520f debian, install-deps.sh: use Build-Profiles to optionally build debian libjaeger
library

* use pkg.ceph.jaeger for debian optional pkgs
remove mangling needing install-deps, instead use, buildProfileSpec feature
introduced for debian.  https://wiki.debian.org/BuildProfileSpec * check and

* set extraopts in debian/rules using pkg.ceph.jaeger
see: https://github.com/ceph/ceph/pull/38783#discussion_r662995612

* cleanup libjaeger.install mangling from CMakeLists

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-19 21:01:42 +00:00
Kefu Chai
4a3a74fd46 ceph.spec,debian: drop protobuf dependencies
since Seastar has dropped the protobuf dependencies, there is no
need to prepare them for building crimson anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-08 08:49:57 +08:00
Kefu Chai
2026a7f0d3
Merge pull request #41999 from tchaikov/wip-rpm-deb
rpm,debian: cleanups related to python3-setuptools dependencies

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2021-06-29 16:16:32 +08:00
Kevin Zhao
8c7729af08 ceph.spec.in, debian/rules: Set rbd-rwl-cache optional on arm64 and ppc64le
set rwl cache option on arm64 and ppc64le as PMDK is not well supported.
Currently, only 64-bit Linux* and Windows* on x86 are supported PMDK

Reference:
1. Experimental support on Arm64, but lacking of librpmem:
See: https://github.com/pmem/pmdk#experimental-support-for-64-bit-arm
2. No RPM for PMDK on Arm64:
See: https://bugzilla.redhat.com/show_bug.cgi?id=1340635
3. > Does PMDK support ARM64*?
   > Currently only 64-bit Linux* and Windows* on x86 are supported.
See: https://software.intel.com/content/www/us/en/develop/articles/persistent-memory-faq.html
4. Make check fail on Arm64
See: https://github.com/pmem/pmdk/issues/5255

Fixes: https://tracker.ceph.com/issues/51339
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
2021-06-25 11:53:18 +08:00
Kefu Chai
c4fbf39d2e cmake,deb,rpm: drop ceph-deploy manpage
ceph-deploy is not actively maintained anymore, and it was replaced by
ceph-volume and other high-level tools.

so there is no point to package its manpage anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-24 12:13:17 +08:00
Kefu Chai
128d6184eb debian/control: drop duplicated python3-setuptools from Build-Depends
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-24 12:13:17 +08:00
Kefu Chai
8867d7cd82 debian/control: add libfmt-dev for "make check"
so, on debian derivatives, we can use the libfmt-dev package for
building Ceph. this change is created in hope to reduce the compile
time.

>= 6.1.2 is specified, as it is the version packaged by ubuntu focal,
which is used for running "make check" and intergration tests.

find_package(fmt 6.0.0 QUIET)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-18 11:13:45 +08:00
Kefu Chai
9b1d524839 debian: mark "crimson" specific deps with "pkg.ceph.crimson"
see also https://wiki.debian.org/BuildProfileSpec

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-07 12:36:19 +08:00
Kefu Chai
c0fad448e9 debian: mark "make check" deps with "pkg.ceph.check" build
instead of inventing our way for defining "make check" dependencies, use
build profile for adding "make check" specific dependencies. see
https://wiki.debian.org/BuildProfileSpec

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-07 12:35:00 +08:00
Kefu Chai
814fb28892 debian/control: ceph-mgr-modules-core does not Recommend ceph-mgr-rook anymore
per https://www.debian.org/doc/debian-policy/ch-relationships.html

> Recommends
>   This declares a strong, but not absolute, dependency.
>
> The Recommends field should list packages that would be found together
> with this one in all but unusual installations.

ceph-mgr-modules-core provides a set of ceph-mgr modules which are
always enabeld. but the rook module enables ceph-mgr to install and
configure a Ceph cluster using Rook. this module is very useful but
it does not have such a strong connection with ceph-mgr-modules-core.
we can always install it separately for using better intergration with
Rook.

See-also: https://tracker.ceph.com/issues/45574
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-04 16:18:23 +08:00
Kefu Chai
8aa5ea0cf1 debian/ceph-common.postinst: fix indent and cleanups
* add editor variables for emacs
* replace tab with 8 spaces
* move "then" to previous line to be more consistent

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-05-08 23:15:07 +08:00
Kefu Chai
1c04ce6f3a doc/man: build and install cephfs-shell manpage
otherwise the manpage of cephfs-shell is not built and installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-05-05 14:41:50 +08:00
Patrick Donnelly
52faaf848e
Revert "Merge PR #41134 into master"
This reverts commit 93bca2d0e9, reversing
changes made to 7e70b15c43.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-05-04 14:46:47 -07:00
Kefu Chai
5191008aa4 doc/man: build and install cephfs-shell manpage
otherwise the manpage of cephfs-shell is not built and installed.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-05-04 15:11:22 +08:00
Varsha Rao
8ab4950171 spec: add nfs to spec file
Signed-off-by: Varsha Rao <varao@redhat.com>
2021-04-22 11:38:30 +05:30
Kefu Chai
5ddda38da4 common: extract options into yaml
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.

this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.

also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.

* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
  to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
  as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
  files, the subvars in it is then replaced with CMake variables,
  and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
  is not a CMake variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-04-13 09:13:47 +08:00
Ilya Dryomov
dc55f0bb43 packaging: require ceph-common for immutable object cache daemon
This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph.  "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.

Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-04-07 13:11:23 +02:00
Kefu Chai
2d3c6561b4 tools/ceph-dencoder: build dencoders as plugins
to reduce the memory footprint when linking ceph-dencoder.

* src/tools/ceph-dencoder:
  * build dencoders as shared libraries named with the prefix of
    "den-mod-". so ceph-dencoder can find them
  * install dencoders into $prefix/lib/ceph/denc, so ceph-dencoder
    can find them
  * only expose "register_dencoders()" function from plugins.
  * load plugins in specified directory
* ceph.spec.in: package plugins
* debian: package plugins

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-04-01 15:46:22 +08:00
Kefu Chai
f1eda0b994 script/run-make: enable WITH_SYSTEM_ZSTD on focal
to speed up the build for "make check"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-31 12:53:39 +08:00
Kefu Chai
406f2efafc
Merge pull request #40292 from tchaikov/wip-39191
debian,cmake,cephsqlite: hide non-public symbols

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-26 14:28:34 +08:00
Kefu Chai
9438c23acd debian/rules: add with_system_libs option
so the maintainer can choose to use the packages shipped by distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-24 18:52:42 +08:00
Kefu Chai
bcd7f3ac0f debian/control: remove cython from Build-Depends
as cython3 is enough. and we've dropped the python2 support.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-24 18:52:42 +08:00
Kefu Chai
561a34a685 debian/rules: exclude ceph_crypto from dh_shlibdep
as libceph_crypto* are plugins, and they are not self-contained. they
reference symbols offered by the executable loading them. dh_shlibdep
should not complain when checking them, so add them to the exclude list.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-24 18:52:42 +08:00
Kefu Chai
45b9da3c92 debian/control: remove libsqlite3-mod-ceph from dep of ceph-mgr
as the subvar of ${shlibs:Depends} is able to take care of it

Fixes: https://tracker.ceph.com/issues/49899
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-24 15:42:14 +08:00
Kefu Chai
4e2762fed7 debian/libsqlite3-mod-ceph: add .symbols file
for tracking the public symbols

see also dh_makeshlibs(1) and dpkg-gensymbols(1)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-23 11:36:41 +08:00
Patrick Donnelly
75980798f1
ceph.spec,debian: package libcephsqlite
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-03-19 08:52:55 -07:00
Kefu Chai
f381aa8bf0 test: run promtool test without docker on ubuntu/focal
before this change, we use docker for running promtools offered by
a docker image, but this is not efficient, and quite a few developers
do not want to use docker for running "make check". this change was
introduced by #39246, the reason was that, in Ceph's CI process, we
are using Ubuntu/Bionic for running "make check" jobs, but prometheus
packaged by Bionic does not offer the "test rules" command. so, to
address problem, we are using "dnanexus/promtool:2.9.2" docker image
for verifying monitoring/prometheus/alerts/test_alerts.yml.

after this change, we use prometheus packaged by debian derivatives
instead of pulling a docker image.

* debian/control: add prometheus as a "make check" dependency
* install-deps.sh: partially revert
  53a5816deda0874a3a37e131e9bc22d88bb2a588, as we don't need to
  pull docker or start docker service for using promtool anymore.
* cmake: check if promtool is capable of running "test rules"
  command, bail out if it is not.

see also: https://tracker.ceph.com/issues/49653

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-19 20:35:51 +08:00
Sage Weil
f64bbb1b5b Merge PR #39953 into master
* refs/pull/39953/head:
	debian/ceph-common.postinst: do not chown cephadm log dirs

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-03-10 09:59:46 -05:00
Sage Weil
b89ffdcae5 debian/ceph-common.postinst: do not chown cephadm log dirs
The container uid/gid is different than the debian uid/gid (because the
container is centos-based and we got a different uid/gid allocation there).

Fixes: https://tracker.ceph.com/issues/49677
Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-09 11:56:42 -06:00
Kefu Chai
b05ea0d6e4 do_cmake,run-make: switch to ninja
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-03-09 19:13:21 +08:00
Marcus Watts
8afd92425b rgw/kms/vault - need libicu to make canonical json for encryption contexts.
for encryption, aws s3 provides an "encryption" context to vary per-object
keys.  The encryption context is a base64 encoded json structure, which
must be converted to a determinstic form -- "canonical json".  This
requires converting all strings to a normalized canonical form: "utf-8 nfc",
it also requires thta keys in objects be sorted in a fixed order; so some
form of sorting based on nfc.

It turns out that libicu was the best way to produce utf-8 nfc (boost also
provides a mechanism, but it has many quirks).  So, here are the hooks
to pull the system libicu into the build.

Fixes: http://tracker.ceph.com/issues/48746
Signed-off-by: Marcus Watts <mwatts@redhat.com>
2021-03-05 17:15:24 -05:00
J. Eric Ivancich
8e03dbce42
Merge pull request #39176 from ivancich/wip-rgw-gap-list
rgw: tooling to locate rgw objects with missing rados components

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Kidd <linuxkidd@gmail.com>
2021-02-04 15:16:48 -05:00
J. Eric Ivancich
3c5154f010 rgw: add rgw-gap-list-comparator tool
The rgw-gap-list tool can produce a number of false positives when the
cluster is being used during its run. One technique to minimize the
number of false positives is to run the tool twice and look for the
objects that appear in both lists. The rgw-gap-list-comparator tool is
designed to do this comparison.

Signed-off-by: Michael Kidd <linuxkidd@gmail.com>
2021-02-03 16:57:50 -05:00
J. Eric Ivancich
07b42195fb rgw: add rgw-gap-list tool
Due to a prior bug (pr: 38228) tail rados objects of some RGW objects
could have been incorrectly deleted. This tool is designed to look for
such cases. It essentially does the opposite of rgw-orphan-list,
looking for rados objects that RGW expects to be there, but which are
not to be found.

IMPORTANT: This is very experimental at this point in time, and any
"results" produced should be verified by other means.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Michael Kidd <linuxkidd@gmail.com>
2021-02-03 16:57:50 -05:00
Sebastian Wagner
46188c4caa
Merge pull request #39007 from mgfritch/cephadm-podman-2.0
cephadm: require podman >= 2.0.0

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2021-01-29 10:26:03 +01:00
Michael Fritch
a899227895
spec, debian: recommend podman >= 2.0.2
Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-01-26 08:01:55 -07:00
Jason Dillaman
6aba1c7bba
Merge pull request #39049 from lixiaoy1/fix_spec
spec: enable RWL/SSD caches

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2021-01-26 09:46:44 -05:00
lixiaoy1
12ca24fc26 spec: enable RWL/SSD caches
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
2021-01-26 04:21:30 -05:00
Venky Shankar
b7acf7fc77 pybind/mgr/mirroring: interface to mirror CephFS directory snapshots
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-01-19 01:06:43 -05:00
Patrick Donnelly
78a7df1500
Merge PR #38846 into master
* refs/pull/38846/head:
	*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2021-01-14 08:10:19 -08:00
Patrick Donnelly
a3db265ad5
*: remove legacy ceph_volume_client.py library
This library is obsolete with the mgr volumes plugin since Nautilus.

The last remaining user of this library was Manila which will be using
the volumes plugin with Pacific and onwards.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-01-12 06:54:29 -08:00
Venky Shankar
d5062c60f6 spec, deb: package cephfs-top utility
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-01-11 06:15:53 -05:00
Ken Dreyer
ad85bdf8de rpm/deb/cmake: install mount.fuse.ceph man page
Install the mount.fuse.ceph man page and ship it in the ceph-fuse
packaging along with the corresponding mount.fuse.ceph binary.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2020-12-17 16:30:07 -07:00
Yuval Lifshitz
c1af849a0d
Merge pull request #37924 from yuvalif/allow_packages_in_lua
rgw/lua: support packages via luarocks
2020-12-17 19:53:00 +02:00
Kefu Chai
f55eb186f0
Merge pull request #38609 from ideepika/fix-yaml-failure
cmake: fix build failure due to yaml-cpp not available

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-12-17 08:05:44 +08:00
Deepika Upadhyay
a8213f07b6 debian/control: update yaml-cpp version if building with Jaeger
jaeger specifically requires yaml-dev +0.6 but since all supported
distro(bionic) still doesn't have this version, yaml-cpp will be build
from source if enabled.
This fixes the build failure:
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
  least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
```

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-12-16 18:13:03 +05:30