We can now use LTO when building ceph. The symver issue was fixed by
using the gcc __symver__ attribute. The systems that support it can now
re-enable LTO.
Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>
libcls_kvs was introduced back in
73d016fdb3, but we don't have an internal
user so far. to reduce the build time. let's disable the build of it by
default.
Signed-off-by: Kefu Chai <kchai@redhat.com>
since we've replaced "virtualenv" with "python3 -m venv", there is no
need to have it in the build deps list.
since, on ubuntu, venv modules is not available by default, we need to
install python3-venv.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change reverts 9132269421
back then, we were using rpm < 4.13, which does not support
the feature of "Debugsource and debuginfo sub-packages", but per
https://bugzilla.redhat.com/show_bug.cgi?id=185590. rpm >= 4.13
has this feature. see also http://rpm.org/wiki/Releases/4.13.0
in CentOS 8, RPM v4.14.3 is available. and by inspecting the log
when building ceph packages on CentOS 8, we have:
Wrote: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6524-g4e868f9a/rpm/el8/RPMS/x86_64/ceph-debugsource-17.0.0-6524.g4e868f9a.el8.x86_64.rpm
Wrote: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6524-g4e868f9a/rpm/el8/RPMS/x86_64/ceph-base-debuginfo-17.0.0-6524.g4e868f9a.el8.x86_64.rpm
Wrote: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6524-g4e868f9a/rpm/el8/RPMS/x86_64/ceph-common-debuginfo-17.0.0-6524.g4e868f9a.el8.x86_64.rpm
Wrote: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6524-g4e868f9a/rpm/el8/RPMS/x86_64/ceph-mds-debuginfo-17.0.0-6524.g4e868f9a.el8.x86_64.rpm
....
build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6524-g4e868f9a/rpm/el8/RPMS/x86_64/ceph-test-debuginfo-17.0.0-6524.g4e868f9a.el8.x86_64.rpm
so, rpmbuild does generate debuginfo package for each binary
package. this should make the life of valgrind a lot easier
when reading the dwz -- no need to read the debuginfo of all
the packages, only the .dwz of the related subpackage is read.
this change should help to decrease the size of debuginfo
rpm packages a little bit. see https://tracker.ceph.com/issues/19099#note-7
this change was inspired by Yuanming Chai <ychai@redhat.com>
See-also: https://tracker.ceph.com/issues/19099
Signed-off-by: Kefu Chai <kchai@redhat.com>
because fmt is packaged in EPEL, while librados is packaged
in RHEL, so we cannot have fmt as a runtime dependency of librados.
to address this issue, we should compile librados either with static library
or with header-only library of fmt. but because the fedora packaging
guideline does no encourage us to package static libraries, and it would
be complicated to package both static and dynamic library for fmt.
the simpler solution would be to compile Ceph with the header-only
version of fmt.
in this change, we compile ceph with the header-only version of fmt
on RHEL to address the runtime dependency issue.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The use of $FIRST_ARG was probably required because the SUSE-specific
%service_* rpm macros were playing tricks on the shell positional parameters.
This is bad practice and error-prone, so let's assume that no macros should do
that anymore and hence it's safe to assume that positional parameters remain
unchanged after any rpm macro call.
Thanks to Franck Bui for providing the original patch
926433f5d4 that this patch is modeled after.
NOTE: the use of FIRST_ARG had already been eliminated by
926433f5d4 but was re-introduced later by
9466d70985
Fixes: 9466d70985
Fixes: https://tracker.ceph.com/issues/51797
Signed-off-by: Nathan Cutler <ncutler@suse.com>
the change to build and ship libthift was added when we didn't have 0.13.0
version shipped via distro pkgs, now that centos 8 and F34 supports req.
version, we do not need to build and ship it with jaeger library.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
* since focal and centos both have yaml-cpp 0.6 available, which dropped
having boost as it's dependency, moving to 0.6 seems a good upgrade.
* cmake: delete Buildyaml, since distro suppilies v0.6 this is not needed
This fixes the build failure, as jaegertracing requires yaml-cpp v0.6+
```
Could NOT find yaml-cpp: Found unsuitable version "", but required is at
least "0.5.1" (found yaml-cpp_LIBRARY-NOTFOUND)
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
In /etc/sysconfig/ceph we allow operators to define if ceph daemons
should be restarted on upgrade: CEPH_AUTO_RESTART_ON_UPGRADE.
But the post selinux scripts will stop ceph.target regardless if this
is set to `no`, leading to operators adding various hacks to prevent
these unexpected or inconvenient daemon restarts. By now, if users
are using rpms directly, they are likely orchestrating their own
daemon restarts so should not rely on the rpm itself to do this.
Fixes: https://tracker.ceph.com/issues/21672
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
In RPM spec files, comment lines should not include macro invocations,
because RPM can and will expand them, with unpredictable results.
Fixes: https://tracker.ceph.com/issues/51622
Signed-off-by: Nathan Cutler <ncutler@suse.com>
since Seastar has dropped the protobuf dependencies, there is no
need to prepare them for building crimson anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
in the KVM instance offered by OBS, we have
[ 346s] + cat /proc/meminfo
[ 347s] MemTotal: 10167736 kB
[ 347s] MemFree: 4983964 kB
[ 347s] MemAvailable: 9826800 kB
[ 347s] Buffers: 85856 kB
[ 347s] Cached: 4615192 kB
[ 347s] SwapCached: 0 kB
...
[ 347s] SwapTotal: 2097148 kB
and its number of hardware threads is
[ 346s] ++ /usr/bin/getconf _NPROCESSORS_ONLN
[ 346s] + _threads=8
so ($MemTotal+$SwapTotal)/1024/2600 = 4.6, which is less
than the # of threads, so "4" was used for the number of jobs.
but per our recent observation in
38be14bc0f, some compiling jobs could
take up to 3GB. in the OOM failure in OBS, we had
[24915s] [24848.843594] Out of memory: Killed process 16894 (cc1plus) total-vm:4293756kB, anon-rss:2970012kB, file-rss:0kB, shmem-rss:0kB, UID:399 pgtables:8324kB oom_score_adj:0
where 4GiB memory was allocated, in which 3GiB was mapped into
memory. this matches with our findings.
in this change, the memory per core is bumped up to 3000MB
in hope to address the OOB. the downside of this change is
that it would take even longer to finish the build if the
building host is limited in memory.
Signed-off-by: Kefu Chai <kchai@redhat.com>
unlike rbd_rwl_cache, rbd_ssd_cache does not depend on pmdk (libpmem),
so let's enable it on all supported architecture and rpm based distros.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ceph-deploy is not actively maintained anymore, and it was replaced by
ceph-volume and other high-level tools.
so there is no point to package its manpage anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the ceph-volume tool is composed of the cli frontend and ceph_volume
python module. in 02bc369e05, its cli
frontend is moved from ceph-base package to ceph-osd. but the python
module was left in ceph-base.
since the only consumer of ceph_volume python package is ceph-volume,
better off moving this python module into ceph-osd. this also aligns
the rpm packaging with the deb packaging, where ceph-osd deb package
also include ceph_volume python module.
we could extract ceph-volumne into its own package, so it can be an
arch-independent package. let's leave it as a follow-up change.
Signed-off-by: Kefu Chai <kchai@redhat.com>
python3-setuptools was originally added to ceph-base as a dependency of
ceph-detect-init, see https://tracker.ceph.com/issues/14864. but since
ceph-disk and ceph-detect-init were replaced by ceph-volume, and were
removed from the debian packaging in
ee6bc23e89.
there is no need to have python3-setuptools in the ceph-base packages
anymore.
but since we are still using pkg_resources module provided by setuptools
in ceph-volume, we need to preserve this runtime dependency in ceph-osd.
as ceph-osd packages ceph-volume.
please note, pkg_resources module is also used by cephadm to poke around
ceph_iscsi python module installed in a container, so python-setuptools
should be installed along with ceph-iscsi if we need a better
interoperability between ceph-iscsi and cephadm. this is not in the
scope of this change.
Signed-off-by: Kefu Chai <kchai@redhat.com>
6.2.1 is the version packaged by EPEL8, in other words, this is the
version we've been testing. so to be more consistent with the
known-to-be-good version, let's bump up the required version.
Signed-off-by: Kefu Chai <kchai@redhat.com>
to lower the number of jobs, we are experiencing build failures on
a builder with 48c96t, 193 free mem. the failures were caused by
OOM killer which kills the c++ compiler
[498376.128969] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jenkins.service,task=cc1plus,pid=1387895,uid=1110
[498376.145288] Out of memory: Killed process 1387895 (cc1plus) total-vm:3323312kB, anon-rss:3164568kB, file-rss:0kB, shmem-rss:0kB, UID:1110
[498376.315185] oom_reaper: reaped process 1387895 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[498377.882072] cc1plus invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
before this change, we use the total memory to calculate the number
of jobs, and assume that each job takes at most 2.5GiB mem. in the
case above, the # of job is 96.
after this change, we use the free memory, and increse the mem per job
to 3.0GiB. in the case above, the # of job would be 85.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Otherwise fedora 33 complains there is no gcc-toolset-9-gcc-c++
when running "WITH_SEASTAR=true ./install_deps.sh"
Related to: 36759b5363
Signed-off-by: Misono Tomohiro <misono.tomohiro@jp.fujitsu.com>
we need to use libpmem 1.10 in #40493.
without enabling the module stream offering libpmem 1.9.2, we can only
have access to libpmem 1.6.1. and fedora 33 only has libpmem 1.9
packaged. the same applies to openSUSE Tumbleweed and openSUSE Leap. so
let's stop using libpmem packaged by distro by default, until these
distros include libpmem 1.10.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/40526/head:
spec: add nfs to spec file
mgr/nfs: Don't enable nfs module by default
mgr/nfs: check for invalid chars in cluster id
mgr/nfs: Use CLICommand wrapper
mgr/nfs: reorg nfs files
mgr/nfs: Check if transport or protocol are list instance
mgr/nfs: reorg cluster class and common helper methods
mgr/nfs: move common export helper methods to ExportMgr class
mgr/nfs: move validate methods into new ValidateExport class
mgr/nfs: add custom exception module
mgr/nfs: create new module for export utils
mgr/nfs: rename fs dir to export
mgr/volumes/nfs: Move nfs code out of volumes plugin
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.
this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.
also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.
* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
files, the subvars in it is then replaced with CMake variables,
and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
is not a CMake variable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph. "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.
Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
de6c8250a6 added an explicit %dir directive for
a new directory added to the ceph-common package, but -- due to a typo --
neglected to include the "%". As a result, RPM builds started to fail with:
Processing files: ceph-common-17.0.0-2787.gde6c8250.el8.x86_64
error: File must begin with "/": {_libdir}/ceph/denc/
RPM build errors:
File must begin with "/": {_libdir}/ceph/denc/
Fixes: de6c8250a6
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2d3c6561b4 introduced a new library directory
"%{_libdir}/ceph/denc/" in ceph-common but did not explicitly state that it
should be owned by the package. This caused OBS builds to fail as follows:
[ 5515s] ceph-common-17.0.0-2786.1.x86_64.rpm: directories not owned by a package:
[ 5515s] - /usr/lib64/ceph/denc
Fixes: 2d3c6561b4
Signed-off-by: Nathan Cutler <ncutler@suse.com>
to reduce the memory footprint when linking ceph-dencoder.
* src/tools/ceph-dencoder:
* build dencoders as shared libraries named with the prefix of
"den-mod-". so ceph-dencoder can find them
* install dencoders into $prefix/lib/ceph/denc, so ceph-dencoder
can find them
* only expose "register_dencoders()" function from plugins.
* load plugins in specified directory
* ceph.spec.in: package plugins
* debian: package plugins
Signed-off-by: Kefu Chai <kchai@redhat.com>
Commit 75980798f1 introduced a new package,
libcephsqlite, with a hard RPM dependency on a package "sqlite-libs" which
does not exist in openSUSE.
Since the runtime library dependencies of libcephsqlite are handled by RPM
transparently, this line is not needed.
Fixes: https://tracker.ceph.com/issues/50007
Signed-off-by: Nathan Cutler <ncutler@suse.com>
