instead of building isa-l_crypto source files directly into the
ceph_crypto_isal plugin, use a cmake ExternalProject to build the
library with make and expose the static library as cmake target
ISAL::Crypto
Signed-off-by: Casey Bodley <cbodley@redhat.com>
This patch add UADK Acceleration for zlib compressor
[UADK: Userspace Acceleration Development Kit](https://github.com/Linaro/uadk)
1. Unity: one set of interfaces to support enc and comp acceleration
2. Efficiency: userspace zerocopy based on IOMMU & SVA(shared virtual address)
3. Security: the IOMMU limits the access rights and security boundaries of devices and processes
4. Compatibility: all acc devices can join this opensource project
Test on Kunpeng 920 platform, compression offload based on UadkAccel:
1. save almost 50% cpu usage compared with no-isal compression in RBD 4M workload
2. save almost 40% cpu usage compared with no-isal compression in RGW put op (4M) workload
3. lower cpu usage, better performance
Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
when enabled and found, the os_zfs_objs target would reference a missing
file filestore/ZFSFileStoreBackend.cc and cmake would fail to configure
i take this to mean that WITH_ZFS was accidentally left over from
filestore's removal. drop all references to WITH_ZFS, along with the
corresponding Findzfs.cmake script
Signed-off-by: Casey Bodley <cbodley@redhat.com>
If we are correctly configured for distributed mode, use the cluster's CPU count
instead of ours. If we are configured for sccache but without distributed mode,
inform the user but continue with normal job limits.
Signed-off-by: Zack Cerza <zack@redhat.com>
there are known leaks, which are tracked by qa/lsan.suppr, in Ceph.
so let's reuse it so we don't see them when running unit test with
ASan enabled.
see also https://clang.llvm.org/docs/AddressSanitizer.html#issue-suppression
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
turns out we have multiple copies of following symbol defined by rbd executable:
```
AddressSanitizer: odr-violation: global 'ceph::buffer::list::always_empty_bptr' at /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/buffer.cc:1267:34
```
before addressing it. let's disable this warning.
Refs https://tracker.ceph.com/issues/65098
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
if sanitizers are enabled, we have to populate the required link flags
to python extensions's building workflow. otherwise ld would fail to
link like:
```
/usr/bin/ld: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2: undefined reference to `__asan_stack_free_10'
```
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
if we set the CFLAGS globally, and the CFLAGS contains options
only acceptable by a certain C compiler, RocksDB could fail to
configure. for instance, if we set CXXFLAGS so it contains
`--config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg` and use clang++
as the CMAKE_CXX_COMPILER, while keep CMAKE_C_COMPILER unchanged.
RocksDB would fail to configure like:
```-- Check for working C compiler: /usr/bin/cc - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
The C compiler
"/usr/bin/cc"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV
Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_65e36/fast && gmake[3]: Entering directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV'
/usr/bin/gmake -f CMakeFiles/cmTC_65e36.dir/build.make CMakeFiles/cmTC_65e36.dir/build
gmake[4]: Entering directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV'
Building C object CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o
/usr/bin/cc -O2 -flto=thin -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_GLIBCXX_ASSERTIONS --config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIE -o CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o -c /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV/testCCompiler.c
cc: error: unrecognized command-line option ‘--config’; did you mean ‘-mpconfig’?
gmake[4]: *** [CMakeFiles/cmTC_65e36.dir/build.make:78: CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o] Error 1
```
where RocksDB tries to check C compiler -- /usr/bin/cc along with
the said CFLAGS, and fails to compile the test C program, because
GCC does not support this option.
so, in this change, let's pass the CMAKE_C_COMPILER as well.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
this allows us to use newer liburing features. Seastar is using
some of them which are not provided by liburing 0.7.
in this change, `--use-libc` is passed to configure. otherwise
it does not link against libc, and the symbles like memset()
won't be available when compiling liburing.so with -fPIC using
clang, which does not pull libc in that case.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
cmake/AddCephTest: bind crimson unittest to different cores
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
When running crimson unittest, the seastar framework always
use and only use cpu0, and with many parallel crimson unittest
jobs, all the jobs are running on cpu0, the other cpu cores
can't used, make the make check run very slow, even timeout
happens. Use set_property RESOURCE_GROUPS to specify cpu resources
to crimson unittest, and accelerate make check running.
Fixes: https://tracker.ceph.com/issues/64117
Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
With newer clang and gcc versions (observed on clang-17.0.6 as
well as gcc 12/13), asan is throwing stack-use-after-return
during OSD startup related to usage of seastar::async, which
relies on swapcontext internally.
seastar/src/core/thread.cc supports asan's hooks, but only if
SEASTAR_HAVE_ASAN_FIBER_SUPPORT is set. seastar's CMakeList.txt
sets it based on Sanitizers_FIBER_SUPPORT, which probably should
be set by the module at src/seastar/cmake/FindSanitizers.cmake,
but that module doesn't seem to be actually invoked anywhere.
Ceph's version of that module (cmake/modules/FindSanitizers.cmake)
does not set Sanitizers_FIBER_SUPPORT.
This commit adds that check as well as the related code snippet.
Fixes: https://tracker.ceph.com/issues/64512
Signed-off-by: Samuel Just <sjust@redhat.com>
tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
There are some older Arm server running pretty slow, the make
check jobs like `check-generated.sh` are killed as the job timeout.
Make CEPH_TEST_TIMEOUT more longer.
Signed-off-by: luo rixin <luorixin@huawei.com>
before this change, we link against libcap without finding it. this
works fine as long as libcap-devel or libcap-dev is installed in the
system. but if it is not, the source would fail to build due to missing
`sys/capability.h`. this is not a great developer experience.
in this change, a `Findcap.cmake` is added to find the capability
library. which would fail the build at the configure phase.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
boost has some different predefined build variants. they are quite
like CMake's CMAKE_BUILD_TYPE. in which, "debug" enables some
features related features. so it would be nice if we can have it
enabled for the Debug build, if the boost is built from source.
see also
https://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html
before this change, we always build the "release" variant. in this
change, "debug" variant is built if Ceph's is built with
CMAKE_BUILD_TYPE=Debug. please note, this change does not change
the way how boost is built when packaging Ceph, as our debian/rpm
receipts do not define CMAKE_BUILD_TYPE and respect the distros'
settings, in that case, the "release" variant is still built.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
FindQAT.cmake adds library targets QAT::qat and QAT::usdm for qatlib,
and FindQatDrv.cmake adds alias targets to the same
src/compressor and src/crypto/qat link against these common QAT targets
src/crypto/qat/qcccrypto.h uses new HAVE_QATDRV define to determine
which headers to include
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* build: add opentelemetry to cmake system
crimson targets that uses Message.cc/h are built before opentelemetry (o-tel), so we need to build o-tel eralier so we also add the library to the include path earlier
this shoud work for WITH_JAEGER flag both the ON/OFF cases, and for librados where the compilation flag is ignored
* msg/tracer: add o-tel trace to Messages with decode/encode function in tracer.h
some files that uses Message.cc/h just need the encode/decode functions and not all others functions.
some crimson targets does not link with ceph_context (common) which is required for tracer.cc file. so we just need to include that functions
* librados: Add opentelemtry trace param for aio_operate and operate methods
in order to propagate the trace info I added the otel-trace as an extra param.
in some places, there already was a blkin trace info, and since it is not used in other places we can safely change it to o-tel trace info.
this will be done in another commit, so the cleanup of blkin trace will be in a dedicated commit
* osd: use the o-tel trace of the msg as a parent span of the osd trace
if there is a valid span in the msg, we will add this op to the request
trace, otherwise it will start a new trace for the OSD op
* rgw: pass put obj trace info to librados
in order to make it possible, I saved the trace info inside the sal::Object, so we can use it later when writing the object to rados
it could be used also later for read ops.
note the trace field of req_state is initalized only in rgw_process, so it's also required in librgw request flow
* prevent breaking channges to kSize. make sure that changes between components built with
different versions of OTEL do not break message compatibility
Signed-off-by: Omri Zeneva <ozeneva@redhat.com>
with arrow's default BUILD_WARNING_LEVEL, -Werror is added to cflags and
debug builds fail on a warning about _FORTIFY_SOURCE
Fixes: https://tracker.ceph.com/issues/63130
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Use quotes to allow empty submodules variable during the cmake build
configuration. This is specially useful for lean build environments
where git is not available (which would result in an empty submodules
variable).
Further context is available at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056793#10.
Signed-off-by: Athos Ribeiro <athos.ribeiro@canonical.com>
CMake 3.28 has turned stricter when executing string(REPLACE …) and
expects four or more commands. In case of distro package builds from
tarball, it happens that git is not present. CTags.cmake tries to
catch that by veriying the exit status of the command, but as there
is in fact git | awk, awk returns 0 even when git does not exist.
Ensure that the variable submodules has been defined before trying
to replace substrings in this variable.
Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>
this change should silence the warning when building Ceph with Boost 1.82:
```
CMake Warning at cmake/modules/FindBoost.cmake:1384 (message):
New Boost version may have incorrect or missing dependencies and imported
targets
Call Stack (most recent call first):
cmake/modules/FindBoost.cmake:1507 (_Boost_COMPONENT_DEPENDENCIES)
cmake/modules/FindBoost.cmake:2125 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:687 (find_package)
```
despite that 244c5ebb added boost to `_Boost_KNOWN_VERSIONS`, it was
not enough. would be better to sync up the CMake upstream for an updated
version of this file like we did before.
in this change
* syncs this file with
897a149067/Modules/FindBoost.cmak
* re-apply the change of 06824bc1ec
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
cmake: adapt FindBoost.cmake to our needs
the vanilla FindBoost.cmake pulled from cmake has couple assumptions
which do not hold in our environment. so address them case by case.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
CMake allows us to customize `CMAKE_CXX_FLAGS` by setting CXXFLAGS
environmental variable. and Debian's debhelper also sets CXXFLAGS
when it builds cmake projects for customizing the building flags.
but we fail to populate this setting down when building external
projects. this is important when it comes to the projects which
is critical to the performance. RocksDB is one of them.
in this change, we pass the `CMAKE_CXX_FLAGS` down in
`BuildRocksDB.cmake` so that its `CMAKE_CXX_FLAGS` contains
the same set of `CMAKE_CXX_FLAGS` used by its parent project.
this should help with the performance in the bluestore, where
RocksDB is used.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Casey Bodley
Effi Ofer
Casey Bodley
Rongqi Sun
Yuri Weinstein
Adam Emerson
Aliaksei Makarau
J. Eric Ivancich
Zack Cerza
Samarah Uriarte
Kefu Chai
Samuel Just
luo rixin
Radoslaw Zarzynski
Omri Zeneva
Athos Ribeiro
Dominique Leuenberger