Commit Graph

646 Commits

Author SHA1 Message Date
Samarah Uriarte
979452a122 rgw/cache: This commit squashes commits related to boost::redis library.
RGW: Add boost_redis library
RGW: Add boost_redis to CMake
RGW: Add aedis methods to RedisDriver
RGW: Minor RedisDriver cleanup
build: Remove build/boost_redis
build: remove src/redis
build: Set src/boost_redis submodule to release version
RGW: Remove broken boost_redis
RGW: Add boost_redis
rgw: Get rgw_redis_driver to build
rgw/redis: Update `boost::redis` library
rgw/cache: fixing compilation error, by removing aedis header
file and src file includes from redis driver

Signed-off-by: Samarah <samarah.uriarte@ibm.com>
Co-authored-by: Adam Emerson <aemerson@redhat.com>
Co-authored-by: Pritha Srivastava <prsrivas@redhat.com>
2024-04-02 21:24:50 +05:30
Kefu Chai
02155b4ffc cmake: suppress LeakSanitizer reports of known leaks
there are known leaks, which are tracked by qa/lsan.suppr, in Ceph.
so let's reuse it so we don't see them when running unit test with
ASan enabled.

see also https://clang.llvm.org/docs/AddressSanitizer.html#issue-suppression

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-27 07:35:28 +08:00
Kefu Chai
a837f494b5 cmake: prevent ASAN_OPTIONS from detect_odr_violation
turns out we have multiple copies of following symbol defined by rbd executable:
```
AddressSanitizer: odr-violation: global 'ceph::buffer::list::always_empty_bptr' at /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/buffer.cc:1267:34
```
before addressing it. let's disable this warning.

Refs https://tracker.ceph.com/issues/65098

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-27 07:35:28 +08:00
Kefu Chai
d22734f6cb cmake: add ${CMAKE_SHARED_LINKER_FLAGS} to LDFLAGS when building python extensions
if sanitizers are enabled, we have to populate the required link flags
to python extensions's building workflow. otherwise ld would fail to
link like:
```
/usr/bin/ld: /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libceph-common.so.2: undefined reference to `__asan_stack_free_10'
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-27 07:35:28 +08:00
Kefu Chai
a13a61d63d cmake: error out on UBSan error
so we can be alerted if UBSan identify something wrong.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-27 07:35:28 +08:00
Kefu Chai
ce444d473c cmake: quote a list using quotes
otherwised we'd have

```
CMake Error at cmake/modules/FindSanitizers.cmake:17 (if):
  if given arguments:

    "address" "IN_LIST" "address" "thread" "undefined_behavior" "OR" "leak" "IN_LIST" "address" "thread" "undefined_behavior"

  Unknown arguments specified
```

when enabling TSan with WITH_TSAN=ON.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-27 07:35:28 +08:00
Kefu Chai
19622f7149
Merge pull request #55890 from tchaikov/wip-cmake-build-rocksdb-cc
cmake/modules/BuildRocksDB.cmake: inherit CMAKE_C_COMPILER from parent

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2024-03-05 11:35:36 +08:00
Kefu Chai
af6bb91505 cmake/modules/BuildRocksDB.cmake: inherit CMAKE_C_COMPILER from parent
if we set the CFLAGS globally, and the CFLAGS contains options
only acceptable by a certain C compiler, RocksDB could fail to
configure. for instance, if we set CXXFLAGS so it contains
`--config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg` and use clang++
as the CMAKE_CXX_COMPILER, while keep CMAKE_C_COMPILER unchanged.
RocksDB would fail to configure like:

```-- Check for working C compiler: /usr/bin/cc - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/usr/bin/cc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV

    Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_65e36/fast && gmake[3]: Entering directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV'
    /usr/bin/gmake  -f CMakeFiles/cmTC_65e36.dir/build.make CMakeFiles/cmTC_65e36.dir/build
    gmake[4]: Entering directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV'
    Building C object CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o
    /usr/bin/cc   -O2 -flto=thin -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security  -Wp,-D_GLIBCXX_ASSERTIONS --config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg -fstack-protector-strong -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -fPIE -o CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o -c /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1717-g0f726187/rpm/el9/BUILD/ceph-19.0.0-1717-g0f726187/redhat-linux-build/src/rocksdb/CMakeFiles/CMakeScratch/TryCompile-RU5UFV/testCCompiler.c
    cc: error: unrecognized command-line option ‘--config’; did you mean ‘-mpconfig’?
    gmake[4]: *** [CMakeFiles/cmTC_65e36.dir/build.make:78: CMakeFiles/cmTC_65e36.dir/testCCompiler.c.o] Error 1
```

where RocksDB tries to check C compiler -- /usr/bin/cc along with
the said CFLAGS, and fails to compile the test C program, because
GCC does not support this option.

so, in this change, let's pass the CMAKE_C_COMPILER as well.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-02 18:08:10 +08:00
Kefu Chai
95e03f8809 cmake: bump liburing from 0.7 to 2.5
this allows us to use newer liburing features. Seastar is using
some of them which are not provided by liburing 0.7.

in this change, `--use-libc` is passed to configure. otherwise
it does not link against libc, and the symbles like memset()
won't be available when compiling liburing.so with -fPIC using
clang, which does not pull libc in that case.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-03-02 12:16:14 +08:00
Kefu Chai
ac1d58ab3e
Merge pull request #55328 from rosinL/wip-fix-64117
cmake/AddCephTest: bind crimson unittest to different cores

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2024-02-22 10:47:29 +08:00
Samuel Just
ed736fab6e
Merge pull request #55684 from athanatos/sjust/wip-seastar-asan-fibers
crimson: fixes for stack-use-after-return on recent clang and gcc versions

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2024-02-21 10:17:56 -08:00
luo rixin
a1f9b59c1a cmake/AddCephTest: Specify resoureces to crimson unittest
When running crimson unittest, the seastar framework always
use and only use cpu0, and with many parallel crimson unittest
jobs, all the jobs are running on cpu0, the other cpu cores
can't used, make the make check run very slow, even timeout
happens. Use set_property RESOURCE_GROUPS to specify cpu resources
to crimson unittest, and accelerate make check running.

Fixes: https://tracker.ceph.com/issues/64117

Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
2024-02-21 10:36:21 +08:00
Samuel Just
f719449280 cmake/.../FindSanitizers: add check for Sanitizers_FIBER_SUPPPORT
With newer clang and gcc versions (observed on clang-17.0.6 as
well as gcc 12/13), asan is throwing stack-use-after-return
during OSD startup related to usage of seastar::async, which
relies on swapcontext internally.

seastar/src/core/thread.cc supports asan's hooks, but only if
SEASTAR_HAVE_ASAN_FIBER_SUPPORT is set.  seastar's CMakeList.txt
sets it based on Sanitizers_FIBER_SUPPORT, which probably should
be set by the module at src/seastar/cmake/FindSanitizers.cmake,
but that module doesn't seem to be actually invoked anywhere.

Ceph's version of that module (cmake/modules/FindSanitizers.cmake)
does not set Sanitizers_FIBER_SUPPORT.

This commit adds that check as well as the related code snippet.

Fixes: https://tracker.ceph.com/issues/64512
Signed-off-by: Samuel Just <sjust@redhat.com>
2024-02-20 16:20:23 -08:00
Radoslaw Zarzynski
b9f000f018
Merge pull request #52114 from yuvalif/wip-end2end-tracing
tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2024-02-20 16:57:39 +01:00
luo rixin
0f660886dd CMakeLists: Modify CEPH_TEST_TIMEOUT from 3600s to 7200s
There are some older Arm server running pretty slow, the make
check jobs like `check-generated.sh` are killed as the job timeout.
Make CEPH_TEST_TIMEOUT more longer.

Signed-off-by: luo rixin <luorixin@huawei.com>
2024-02-20 10:48:56 +08:00
Kefu Chai
6850bc28ae
Merge pull request #54963 from DimStar77/cmake328
cmake: Ensure git exists before executing it

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-02-17 23:18:23 +08:00
Kefu Chai
6c275abe19
Merge pull request #55529 from tchaikov/cmake-cap
cmake: find_package(cap) before linking against it

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2024-02-12 09:23:21 +08:00
Kefu Chai
0de5755531 cmake: find_package(cap) before linking against it
before this change, we link against libcap without finding it. this
works fine as long as libcap-devel or libcap-dev is installed in the
system. but if it is not, the source would fail to build due to missing
`sys/capability.h`. this is not a great developer experience.

in this change, a `Findcap.cmake` is added to find the capability
library. which would fail the build at the configure phase.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-02-11 16:57:38 +08:00
Kefu Chai
c7155c3db7 cmake: build boost debug variant when CMAKE_BUILD_TYPE is Debug
boost has some different predefined  build variants. they are quite
like CMake's CMAKE_BUILD_TYPE. in which, "debug" enables some
features related features. so it would be nice if we can have it
enabled for the Debug build, if the boost is built from source.
see also
https://www.boost.org/build/doc/html/bbv2/overview/builtins/features.html

before this change, we always build the "release" variant.  in this
change, "debug" variant is built if Ceph's is built with
CMAKE_BUILD_TYPE=Debug. please note, this change does not change
the way how boost is built when packaging Ceph, as our debian/rpm
receipts do not define CMAKE_BUILD_TYPE and respect the distros'
settings, in that case, the "release" variant is still built.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2024-02-11 15:48:54 +08:00
Casey Bodley
16acfe0213
Merge pull request #54624 from cbodley/wip-cmake-qat
cmake: add ExternalProjects to enable QAT by default

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2024-02-08 21:13:52 +00:00
Kefu Chai
8ac5645372
Merge pull request #55267 from athos-ribeiro/cmake-empty-submodules
cmake: quote variable to allow empty values

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2024-02-08 23:14:09 +08:00
Casey Bodley
84af8357d1 cmake: WITH_QATLIB/ZIP disabled on aarch64
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 11:13:21 -05:00
Casey Bodley
1df11fd17c cmake: build from qatzip submodule
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 11:13:20 -05:00
Casey Bodley
e4e7dfd472 cmake: build qatzip from source when WITH_SYSTEM_QATZIP=OFF
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 11:12:45 -05:00
Casey Bodley
74c437a0c8 cmake: build from qatlib submodule
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 11:10:41 -05:00
Casey Bodley
3068123667 cmake: build qat from source when WITH_SYSTEM_QATLIB=OFF
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 11:09:49 -05:00
Casey Bodley
127059e7c6 cmake: rename qatzip->QATzip and support pkg-config
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 10:37:47 -05:00
Casey Bodley
d6663c389b cmake: add WITH_QATLIB to build against qatlib
FindQAT.cmake adds library targets QAT::qat and QAT::usdm for qatlib,
and FindQatDrv.cmake adds alias targets to the same

src/compressor and src/crypto/qat link against these common QAT targets

src/crypto/qat/qcccrypto.h uses new HAVE_QATDRV define to determine
which headers to include

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-02-07 10:37:47 -05:00
Omri Zeneva
320a2179a3 tracer/osd/librados/build/rgw: rgw and osd end2end tracing using opentelemetry
* build: add opentelemetry to cmake system
crimson targets that uses Message.cc/h are built before opentelemetry (o-tel), so we need to build o-tel eralier so we also add the library to the include path earlier
this shoud work for WITH_JAEGER flag both the ON/OFF cases, and for librados where the compilation flag is ignored

* msg/tracer: add o-tel trace to Messages with decode/encode function in tracer.h
some files that uses Message.cc/h just need the encode/decode functions  and not all others functions.
some crimson targets does not link with ceph_context (common) which is required for tracer.cc file. so we just need to include that functions

* librados: Add opentelemtry trace param for aio_operate and operate methods
in order to propagate the trace info I added the otel-trace as an extra param.
in some places, there already was a blkin trace info, and since it is not used in other places we can safely change it to o-tel trace info.
this will be done in another commit, so the cleanup of blkin trace will be in a dedicated commit

* osd: use the o-tel trace of the msg as a parent span of the osd trace
if there is a valid span in the msg, we will add this op to the request
trace, otherwise it will start a new trace for the OSD op

* rgw: pass put obj trace info to librados
in order to make it possible, I saved the trace info inside the sal::Object, so we can use it later when writing the object to rados
it could be used also later for read ops.
note the trace field of req_state is initalized only in rgw_process, so it's also required in librgw request flow

* prevent breaking channges to kSize. make sure that changes between components built with
different versions of OTEL do not break message compatibility

Signed-off-by: Omri Zeneva <ozeneva@redhat.com>
2024-02-06 08:01:42 +00:00
Casey Bodley
5c1a57a1af cmake/arrow: don't treat warnings as errors
with arrow's default BUILD_WARNING_LEVEL, -Werror is added to cflags and
debug builds fail on a warning about _FORTIFY_SOURCE

Fixes: https://tracker.ceph.com/issues/63130

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2024-01-24 09:47:05 -05:00
Athos Ribeiro
657f59cced cmake: quote variable to allow empty values
Use quotes to allow empty submodules variable during the cmake build
configuration. This is specially useful for lean build environments
where git is not available (which would result in an empty submodules
variable).

Further context is available at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056793#10.

Signed-off-by: Athos Ribeiro <athos.ribeiro@canonical.com>
2024-01-22 15:06:01 -03:00
Kefu Chai
c6b0b2113b
Merge pull request #54918 from tchaikov/cmake-build-rockdb-with-cxxflags
cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-12-21 11:04:44 +08:00
Dominique Leuenberger
8615731637
cmake: Ensure git exists before executing it
CMake 3.28 has turned stricter when executing string(REPLACE …) and
expects four or more commands. In case of distro package builds from
tarball, it happens that git is not present. CTags.cmake tries to
catch that by veriying the exit status of the command, but as there
is in fact git | awk, awk returns 0 even when git does not exist.

Ensure that the variable submodules has been defined before trying
to replace substrings in this variable.

Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>
2023-12-19 15:27:45 +01:00
Kefu Chai
f61f651fb3 cmake: add 1.82 and 1.83 to known versions
this change should silence the warning when building Ceph with Boost 1.82:
```
CMake Warning at cmake/modules/FindBoost.cmake:1384 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  cmake/modules/FindBoost.cmake:1507 (_Boost_COMPONENT_DEPENDENCIES)
  cmake/modules/FindBoost.cmake:2125 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:687 (find_package)
```

despite that 244c5ebb added boost to `_Boost_KNOWN_VERSIONS`, it was
not enough. would be better to sync up the CMake upstream for an updated
version of this file like we did before.

in this change

* syncs this file with
  897a149067/Modules/FindBoost.cmak
* re-apply the change of 06824bc1ec

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

cmake: adapt FindBoost.cmake to our needs

the vanilla FindBoost.cmake pulled from cmake has couple assumptions
which do not hold in our environment. so address them case by case.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-12-15 19:49:11 +08:00
Kefu Chai
beb1a62492 cmake/modules/BuildRocksDB.cmake: inherit parent's CMAKE_CXX_FLAGS
CMake allows us to customize `CMAKE_CXX_FLAGS` by setting CXXFLAGS
environmental variable. and Debian's debhelper also sets CXXFLAGS
when it builds cmake projects for customizing the building flags.

but we fail to populate this setting down when building external
projects. this is important when it comes to the projects which
is critical to the performance. RocksDB is one of them.

in this change, we pass the `CMAKE_CXX_FLAGS` down in
`BuildRocksDB.cmake` so that its `CMAKE_CXX_FLAGS` contains
the same set of `CMAKE_CXX_FLAGS` used by its parent project.

this should help with the performance in the bluestore, where
RocksDB is used.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-12-15 19:11:54 +08:00
Kefu Chai
620b68a348 cmake/modules/BuildRocksDB.cmake: use string(APPEND ..) when appropriate
less repeatings this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-12-15 19:11:54 +08:00
Casey Bodley
353360581a cmake/rocksdb: make sure dependencies build before rocksdb
some of rocksdb's dependencies may not have built by the time its
ExternalProject starts, so it can fail with missing headers or
libraries. for example, `uring::uring` may itself be an ExternalProject,
and its include directory won't exist until it starts building:

```
[89/1345] Performing configure step for 'rocksdb_ext'
FAILED: src/rocksdb_ext-prefix/src/rocksdb_ext-stamp/rocksdb_ext-configure build/src/rocksdb_ext-prefix/src/rocksdb_ext-stamp/rocksdb_ext-configure
...
CMake Error in CMakeLists.txt:
  Imported target "uring::uring" includes non-existent path

    "build/src/liburing/src/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.
...
[91/1345] Performing download step (git clone) for 'liburing_ext'
Cloning into 'liburing'...
```

use `add_dependencies(rocksdb_ext)` to make sure all of its dependencies
are available before starting the build

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-11-23 00:19:29 -05:00
Ilya Dryomov
f77d49863b
Merge pull request #53346 from linuxbox2/wip-build-fio
cmake: BuildFIO.cmake should not introduce -std=gnu++17

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2023-11-01 18:18:40 +01:00
Casey Bodley
2560793827
Merge pull request #54216 from adamemerson/wip-fmt-felicity
build: fmt felicity

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-10-31 20:20:56 +00:00
Adam Emerson
686dd3d838 build: Fix fmt version check
Currently, when attempting to build ceph on a system with fmt
installed, we try to build against it whatever the version. This
constantly breaks people's builds, since newer versions of fmt often
change the API.

This change specifies that versions must be below 10 as well as at or
above 8.1.1, so that on systems with a new format, we fall back to
using the submodule.

It also removes the `Findfmt.cmake` module, as that does not check
the installed version. Instead, we use the cmake config file installed by
the system package of fmt and does support version checking.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
2023-10-30 10:38:36 -04:00
Casey Bodley
670118abfb cmake: fix Finduring.cmake
liburing-devel package provides `/usr/lib64/liburing.so`. without this
fix, `find_package(uring)` failed with this `CMAKE_FIND_DEBUG_MODE`
output:
```
  find_library considered the following locations:

    /home/cbodley/ceph/build/virtualenv/bin//(lib)liburing.a(\.so|\.a)
    /home/cbodley/ceph/build/virtualenv/bin/(lib)liburing.a(\.so|\.a)
    /usr/local/bin//(lib)liburing.a(\.so|\.a)
    /usr/local/bin/(lib)liburing.a(\.so|\.a)
    /usr/local/sbin//(lib)liburing.a(\.so|\.a)
    /usr/local/sbin/(lib)liburing.a(\.so|\.a)
    /usr/bin//(lib)liburing.a(\.so|\.a)
    /usr/bin/(lib)liburing.a(\.so|\.a)
    /usr/sbin//(lib)liburing.a(\.so|\.a)
    /usr/sbin/(lib)liburing.a(\.so|\.a)
    /usr/local/lib64//(lib)liburing.a(\.so|\.a)
    /usr/local/lib64/(lib)liburing.a(\.so|\.a)
    /usr/local/lib//(lib)liburing.a(\.so|\.a)
    /usr/local/lib/(lib)liburing.a(\.so|\.a)
    /usr/local//(lib)liburing.a(\.so|\.a)
    /usr/local/(lib)liburing.a(\.so|\.a)
    /usr/lib64//(lib)liburing.a(\.so|\.a)
    /usr/lib64/(lib)liburing.a(\.so|\.a)
    /usr/lib//(lib)liburing.a(\.so|\.a)
    /usr/lib/(lib)liburing.a(\.so|\.a)
    /usr//(lib)liburing.a(\.so|\.a)
    /usr/(lib)liburing.a(\.so|\.a)
    /lib64//(lib)liburing.a(\.so|\.a)
    /lib64/(lib)liburing.a(\.so|\.a)
    /lib//(lib)liburing.a(\.so|\.a)
    /lib/(lib)liburing.a(\.so|\.a)
    /opt//(lib)liburing.a(\.so|\.a)
    /opt/(lib)liburing.a(\.so|\.a)
    /usr/lib64/X11//(lib)liburing.a(\.so|\.a)
    /usr/lib64/X11/(lib)liburing.a(\.so|\.a)
```

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-10-20 09:18:30 -04:00
Patrick Donnelly
8addb35987
cmake: populate liburing include and library paths down to rocksdb external project
There are two problems:

1) The librocksdb does not use the built liburing if present.
2) You cannot turn off rocksdb's use of liburing with WITH_LIBURING=OFF.

This is to fix errors like:

	FAILED: bin/ceph_test_keyvaluedb_iterators
	: && /opt/rh/gcc-toolset-11/root/usr/bin/g++ -Og -g -rdynamic -pie src/test/ObjectMap/CMakeFiles/ceph_test_keyvaluedb_iterators.dir/test_keyvaluedb_iterators.cc.o src/test/ObjectMap/CMakeFiles/ceph_test_keyvaluedb_iterators.dir/KeyValueDBMemory.cc.o -o bin/ceph_test_keyvaluedb_iterators  -Wl,-rpath,/home/pdonnell/scratch/build/lib  lib/libos.a  lib/libgmock_maind.a  lib/libgmockd.	a  lib/libgtestd.a  -lpthread  -ldl  lib/libglobal.a  -ldl  /usr/lib64/librt.so  -lresolv  -ldl  lib/libblk.a  /lib64/libaio.so  src/liburing/src/liburing.a  lib/libkv.a  lib/libheap_profiler.a  /lib64/libtcmalloc.so  src/rocksdb/librocksdb.a  /lib64/libsnappy.so  /usr/lib64/liblz4.so  /usr/lib64/libz.so  /usr/lib64/libfuse.so  lib/libceph-common.so.2  src/opentelemetry-cpp/sdk/src/trace/libopentelemetry_trace.a  src/opentelemetry-cpp/sdk/src/resource/libopentelemetry_resources.a  src/opentelemetry-cpp/sdk/src/common/libopentelemetry_common.a  src/opentelemetry-cpp/exporters/jaeger/libopentelemetry_exporter_jaeger_trace.a  src/opentelemetry-cpp/ext/src/http/client/curl/libopentelemetry_http_client_curl.a  /usr/lib64/libcurl.so  /usr/lib64/libthrift.so  lib/libjson_spirit.a  lib/libcommon_utf8.a  lib/liberasure_code.a  lib/libextblkdev.a  -lcap  boost/lib/libboost_thread.a  boost/lib/libboost_chrono.a  boost/lib/libboost_atomic.a  boost/lib/libboost_system.a  boost/lib/libboost_random.a  boost/lib/libboost_program_options.a  boost/lib/libboost_date_time.a  boost/lib/libboost_iostreams.a  boost/lib/libboost_regex.a  lib/libfmtd.a  /usr/lib64/libblkid.so  -lpthread  /usr/lib64/libcrypto.so  /usr/lib64/libudev.so  /usr/lib64/libz.so  -ldl  -lresolv   -Wl,--as-needed -latomic && :
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `io_uring_wait_cqe_nr':
	/home/pdonnell/scratch/build/src/liburing/src/include/liburing.h:494: undefined reference to `__io_uring_get_cqe'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
	/home/pdonnell/ceph/src/rocksdb/env/fs_posix.cc:1125: undefined reference to `io_uring_get_sqe'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/fs_posix.cc:1134: undefined reference to `io_uring_submit'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `rocksdb::CreateIOUring()':
	/home/pdonnell/ceph/src/rocksdb/env/io_posix.h:272: undefined reference to `io_uring_queue_init'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `io_uring_wait_cqe_nr':
	/home/pdonnell/scratch/build/src/liburing/src/include/liburing.h:494: undefined reference to `__io_uring_get_cqe'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
	/home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:674: undefined reference to `io_uring_get_sqe'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:684: undefined reference to `io_uring_submit_and_wait'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `rocksdb::PosixRandomAccessFile::ReadAsync(rocksdb::FSReadRequest&, rocksdb::IOOptions const&, std::function<void (rocksdb::FSReadRequest const&, void*)>, void*, void**, std::function<void (void*)>*, rocksdb::IODebugContext*)':
	/home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:901: undefined reference to `io_uring_get_sqe'
	/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:910: undefined reference to `io_uring_submit'
	collect2: error: ld returned 1 exit status

Fixes: https://tracker.ceph.com/issues/63218
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-10-17 13:44:49 -04:00
Daniel Gryniewicz
5258bcbd73 RGW - Add POSIX Driver
This is the MVP for a driver for RGW that operates on top of a POSIX
filesystem.  It supports get, put, list, copy, multipart, external
access via the filesystem itself, and ordered bucket listings via an
LRU-based cache.

Note that this is currently a Filter, indended to run on top of dbstore.
This is because it currently doesn't have any User implementation, so it
depends on dbstore's User.  Everything else is implemented in
POSIXDriver.  Once there is a User implementation, this will become a
Store, instead of a Filter.

Commit messages from bucket listing cache:

  rgw/posixdriver: recycle lmdb database handles as required

    While LMDB workflows often do not close/return database handles,
    ours continually reuses them.  This requires us to close each
    handle (atomically) when a cache entry is recycled.

  rgw/posixdriver: don't instantiate bucket cache entries from notify events

  rgw/posixdriver: incorporate lmdb-safe for now

    The current inclusion is based on https://github.com/Martchus/lmdb-safe,
    which is actively maintained but currently has some packaging issues the
    author has agreed to accept fixes for.

    For now, skip the submodule to save time and remove an external dependency.

  rgw/posixdriver: fix listing of cached, empty bucket

    * check lmdb enumeration result in all cases and w/better style
    * add unit test for enumeration of an empty cached directory

  rgw/posixdriver: nest lmdbs in a directory under the dbroot path to avoid cleanup issues

  rgw/posixdriver: refactor for posix integration

    * Derive BucketCache types as templates on a SAL driver and SAL
      bucket pair.

    * Integrate cache fills as callbacks into SAL layer (or mock, for
      tests)

    * Renaming and cleanups

  rgw/posixdriver: add bucket cache implementation and tests

    Adds free-standing cache of buckets and object names, with
    bucket names (and listing attributes, upcoming) managed in
    a hashed set of lmdb databases, which provides ordering and
    a high-performance listing cache.

    An framework for notification on new object creation (e.g.,
    outside S3 workflow) is provided, and a Linux implementation
    using inotify.

    FindLMDB.cmake taken with attribution and license.

  rgw/posixdriver: add zpp_bits serialization (FAST)

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2023-09-14 12:09:40 -04:00
Matt Benjamin
e7779fefc6 cmake: BuildFIO.cmake should not introduce -std=gnu++17
Not correct in general, and a build bug because fio-objectstore
includes c++20 headers.

Fixes: https://tracker.ceph.com/issues/62778

Add CXX_EXTENSIONS ON, and cleanup INTERFACE_COMPILE_OPTIONS per Kefu review

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2023-09-10 11:41:38 -04:00
Kefu Chai
27c05edc8e cmake: update FindPython3 module
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
  - dropping the `cmake_policy()` calls which set the policy not supported
    by 3.16.
  - `include (FindPackageHandleStandardArgs)` without specifying the
    relative path.
  - dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
    this option was introduced by CMake v3.19, see
    https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
    but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.

the new FindPython3 module from CMake:

* enables us to find the recent Python intepreter and development files up to
  CPython 3.13.
* finds Python intepreter with the new `Python_FIND_STRATEGY`. the old and
  default strategy always finds the most recent version with all specified name
  and in all locations. so, if /usr/bin/python exists, it would accept, even
  if it is a symlink to python3.9 and what we want is python3.6. the
  new policy stops at the one which satisfies the constraints. this
  helps us to address the problem of issue we have when recreating
  the building system after the CMakeLists.txt is updated.

Fixes: https://tracker.ceph.com/issues/62428
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-08-17 10:24:16 +08:00
Kefu Chai
fb93d6e032 cmake: update FindPython3 module
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
  - dropping the `cmake_policy()` calls which set the policy not supported
    by 3.16.
  - `include (FindPackageHandleStandardArgs)` without specifying the
    relative path.
  - dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
    this option was introduced by CMake v3.19, see
    https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
    but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.

the new FindPython3 module from CMake:

* enables us to find the recent Python intepreter and development files up to
  CPython 3.13.
* finds intepreter with the new `Python_FIND_STRATEGY`. the old and default
  strategy always finds the most recent version with all specified name
  and in all locations. so, if /usr/bin/python exists, it would accept, even
  if it is  a symlink to python3.9 and what we want is python3.6. while
  the new policy stops at the one which satisfies the constraints.
  simpler this way and less error prone.

Fixes: https://tracker.ceph.com/issues/62428
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-08-17 10:16:45 +08:00
Adam C. Emerson
244c5ebbd4 build: Bump boost to 1.82
Needed to fix coroutine detection under Clang

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2023-07-05 15:45:07 -04:00
Dongsheng Yang
54853a5ae7 tools/rbd: add support of new device type for ubbd
ubbd (Userspace Backend Block Device) is a method to build block device
and handle IO by process started in userspace. That means we can provide
a generic block device to user and handle IO requests by librbd
in linux.

This way, we can allow user to use rbd image as a linux block device
supporting full image features, especial the journaling feature, which
is not supported by krbd.

For more information: https://github.com/DataTravelGuide/ubbd

Signed-off-by: Dongsheng Yang <dongsheng.yang.linux@gmail.com>
2023-05-18 12:35:39 +00:00
Casey Bodley
83bac8d01e cmake/boost: WITH_ASAN adds BOOST_USE_ASAN to context
also adds 'context-impl=ucontext' to the b2 command line and the
BOOST_USE_UCONTEXT define as described in:
https://www.boost.org/doc/libs/1_79_0/libs/context/doc/html/context/stack/sanitizers.html

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-03-07 13:33:57 -05:00
Casey Bodley
3753469d1b cmake: update FindSanitizers for new libasan versions
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-03-07 13:33:57 -05:00