to lower the number of jobs, we are experiencing build failures on
a builder with 48c96t, 193 free mem. the failures were caused by
OOM killer which kills the c++ compiler
[498376.128969] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jenkins.service,task=cc1plus,pid=1387895,uid=1110
[498376.145288] Out of memory: Killed process 1387895 (cc1plus) total-vm:3323312kB, anon-rss:3164568kB, file-rss:0kB, shmem-rss:0kB, UID:1110
[498376.315185] oom_reaper: reaped process 1387895 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[498377.882072] cc1plus invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
before this change, we use the total memory to calculate the number
of jobs, and assume that each job takes at most 2.5GiB mem. in the
case above, the # of job is 96.
after this change, we use the free memory, and increse the mem per job
to 3.0GiB. in the case above, the # of job would be 85.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Otherwise fedora 33 complains there is no gcc-toolset-9-gcc-c++
when running "WITH_SEASTAR=true ./install_deps.sh"
Related to: 36759b5363
Signed-off-by: Misono Tomohiro <misono.tomohiro@jp.fujitsu.com>
we need to use libpmem 1.10 in #40493.
without enabling the module stream offering libpmem 1.9.2, we can only
have access to libpmem 1.6.1. and fedora 33 only has libpmem 1.9
packaged. the same applies to openSUSE Tumbleweed and openSUSE Leap. so
let's stop using libpmem packaged by distro by default, until these
distros include libpmem 1.10.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/40526/head:
spec: add nfs to spec file
mgr/nfs: Don't enable nfs module by default
mgr/nfs: check for invalid chars in cluster id
mgr/nfs: Use CLICommand wrapper
mgr/nfs: reorg nfs files
mgr/nfs: Check if transport or protocol are list instance
mgr/nfs: reorg cluster class and common helper methods
mgr/nfs: move common export helper methods to ExportMgr class
mgr/nfs: move validate methods into new ValidateExport class
mgr/nfs: add custom exception module
mgr/nfs: create new module for export utils
mgr/nfs: rename fs dir to export
mgr/volumes/nfs: Move nfs code out of volumes plugin
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.
this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.
also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.
* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
files, the subvars in it is then replaced with CMake variables,
and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
is not a CMake variable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph. "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.
Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
de6c8250a6 added an explicit %dir directive for
a new directory added to the ceph-common package, but -- due to a typo --
neglected to include the "%". As a result, RPM builds started to fail with:
Processing files: ceph-common-17.0.0-2787.gde6c8250.el8.x86_64
error: File must begin with "/": {_libdir}/ceph/denc/
RPM build errors:
File must begin with "/": {_libdir}/ceph/denc/
Fixes: de6c8250a6
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2d3c6561b4 introduced a new library directory
"%{_libdir}/ceph/denc/" in ceph-common but did not explicitly state that it
should be owned by the package. This caused OBS builds to fail as follows:
[ 5515s] ceph-common-17.0.0-2786.1.x86_64.rpm: directories not owned by a package:
[ 5515s] - /usr/lib64/ceph/denc
Fixes: 2d3c6561b4
Signed-off-by: Nathan Cutler <ncutler@suse.com>
to reduce the memory footprint when linking ceph-dencoder.
* src/tools/ceph-dencoder:
* build dencoders as shared libraries named with the prefix of
"den-mod-". so ceph-dencoder can find them
* install dencoders into $prefix/lib/ceph/denc, so ceph-dencoder
can find them
* only expose "register_dencoders()" function from plugins.
* load plugins in specified directory
* ceph.spec.in: package plugins
* debian: package plugins
Signed-off-by: Kefu Chai <kchai@redhat.com>
Commit 75980798f1 introduced a new package,
libcephsqlite, with a hard RPM dependency on a package "sqlite-libs" which
does not exist in openSUSE.
Since the runtime library dependencies of libcephsqlite are handled by RPM
transparently, this line is not needed.
Fixes: https://tracker.ceph.com/issues/50007
Signed-off-by: Nathan Cutler <ncutler@suse.com>
The ceph-resource-agents package contains an architecture-independent
bash script and parent directories. There are no architecture-dependent
files here, so we can use a single noarch RPM across all host
architectures.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
And since the dependency is now distro-conditional, moving it down
to the 'distro-conditional make check dependencies' section.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
* build with WITH_SYSTEM_PMDK=ON on fedora, as f32 and f33 ship
libpmem1.8 and libpmem1.9 respectively. and we need libpmem v1.7
* build with WITH_SYSTEM_PMDK=ON on el8, as el8 and CentOS8 AppStream
ships libpmem v1.6,
quote from nvml.spec:
> By design, PMDK does not support any 32-bit architecture.
> Due to dependency on some inline assembly, PMDK can be compiled only
> on these architectures:
> - x86_64
> - ppc64le (experimental)
> - aarch64 (unmaintained, supporting hardware doesn't exist?)
so far, only x86_64 and ppc64le packages are built.
see also,
https://src.fedoraproject.org/rpms/nvml/blob/rawhide/f/nvml.spec
this change addresses a regression introduced by
a49d1dbb32
Signed-off-by: Kefu Chai <kchai@redhat.com>
to ease the build for developers using SUSE, Fedora, CentOS or RHEL.
so install-deps.sh can install ninja for them.
Fixes: https://tracker.ceph.com/issues/49694
Signed-off-by: Kefu Chai <kchai@redhat.com>
The recently merged commit 0e511973f7 replaced
make %{_smp_mflags}
with
%make_build
for the stated purpose of hiding the %_smp_mflags macro in a higher-level macro.
But, on SUSE, the higher-level macro (%make_build) expands to:
make -O %{_smp_mflags}
The addition of the -O flag makes the build considerably slower and increases
the memory requirement. The exact reason for this is unknown - possibly it's due
to a bug in make, although the same slowness was observed with ninja as well.
In any event, this is a deal-breaker when building in the OBS, because the build
infrastructure there is optimized for builds that do not require huge amounts of
memory and we would rather have a fast build with mixed up compiler messages
than a very slow one with synced compiler messages.
Fixes: 0e511973f7
Signed-off-by: Nathan Cutler <ncutler@suse.com>
for encryption, aws s3 provides an "encryption" context to vary per-object
keys. The encryption context is a base64 encoded json structure, which
must be converted to a determinstic form -- "canonical json". This
requires converting all strings to a normalized canonical form: "utf-8 nfc",
it also requires thta keys in objects be sorted in a fixed order; so some
form of sorting based on nfc.
It turns out that libicu was the best way to produce utf-8 nfc (boost also
provides a mechanism, but it has many quirks). So, here are the hooks
to pull the system libicu into the build.
Fixes: http://tracker.ceph.com/issues/48746
Signed-off-by: Marcus Watts <mwatts@redhat.com>
As of a49d1dbb32, when the rbd_rwl_cache and
rbd_ssd_cache bconds are enabled and WITH_SYSTEM_PMDK is disabled (as it is by
default), the RPM build attempts to
git clone https://github.com/ceph/pmdk.git
but of course that won't work in the OBS, where the build workers have no
Internet connectivity.
Fortunately, the openSUSE/SLE versions targeted by Ceph master and pacific ship
the necessary PMDK libraries as RPM packages.
Fixes: a49d1dbb32
Fixes: https://tracker.ceph.com/issues/49550
Signed-off-by: Nathan Cutler <ncutler@suse.com>
43b441f9a3 removed a bunch of code which the SUSE
builds were relying on to avoid OOM. This commit brings back that code in
a much-streamlined form: the SUSE-specific %limit_build macro.
This also has the advantage of not breaking the build on older RPMs which only
know about %_smp_mflags, and not the newer %_smp_build_ncpus etc. macros.
Fixes: 43b441f9a3
Fixes: https://tracker.ceph.com/issues/49556
Signed-off-by: Nathan Cutler <ncutler@suse.com>
This code causes the Ceph build in OBS to fail due to OOM, because the typical
setting of %_smp_build_ncpus in the OBS is 16, but available memory is
insufficient to sustain such a high degree of parallelism for the in-memory
compression operation.
Fixes: b50fc9e61c
Fixes: https://tracker.ceph.com/issues/49583
Signed-off-by: Nathan Cutler <ncutler@suse.com>
The luarocks conditional had gotten hard to read, and the openSUSE Leap 15.3
build needs lua53 as well.
Signed-off-by: Nathan Cutler <ncutler@suse.com>
This partially reverts 2b1e646f7a which
mistakenly changed a line inside an "%if 0%{?suse_version}" conditional.
Fixes: 2b1e646f7a
Signed-off-by: Nathan Cutler <ncutler@suse.com>
this change partially reverts da7030db79
which use %cmake rpm macro in the place of "cmake". but
%cmake sets BUILD_SHARED_LIBS=ON. so quite a few internal libraries
defined using add_library() are now compiled into shared libraries which
are not installed or packagesd. when we are installing the rpm packages
compiled with this option, rpm compiles because the linked libraries are
missing, for instance, `libgmock.so.1.10.0` was compiled as a static
library before da7030db79, and was
included by the test executables. but after that change it's compiled
as a shared library.
so we need to either package the linked shared libraries or just link
against them statically. at this moment, the latter approach is simpler,
albeit larger size of exectuable and dbg symbols.
Fixes: https://tracker.ceph.com/issues/49395
Signed-off-by: Kefu Chai <kchai@redhat.com>
