this change should silence the warning when building Ceph with Boost 1.82:
```
CMake Warning at cmake/modules/FindBoost.cmake:1384 (message):
New Boost version may have incorrect or missing dependencies and imported
targets
Call Stack (most recent call first):
cmake/modules/FindBoost.cmake:1507 (_Boost_COMPONENT_DEPENDENCIES)
cmake/modules/FindBoost.cmake:2125 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:687 (find_package)
```
despite that 244c5ebb added boost to `_Boost_KNOWN_VERSIONS`, it was
not enough. would be better to sync up the CMake upstream for an updated
version of this file like we did before.
in this change
* syncs this file with
897a149067/Modules/FindBoost.cmak
* re-apply the change of 06824bc1ec
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
cmake: adapt FindBoost.cmake to our needs
the vanilla FindBoost.cmake pulled from cmake has couple assumptions
which do not hold in our environment. so address them case by case.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
CMake allows us to customize `CMAKE_CXX_FLAGS` by setting CXXFLAGS
environmental variable. and Debian's debhelper also sets CXXFLAGS
when it builds cmake projects for customizing the building flags.
but we fail to populate this setting down when building external
projects. this is important when it comes to the projects which
is critical to the performance. RocksDB is one of them.
in this change, we pass the `CMAKE_CXX_FLAGS` down in
`BuildRocksDB.cmake` so that its `CMAKE_CXX_FLAGS` contains
the same set of `CMAKE_CXX_FLAGS` used by its parent project.
this should help with the performance in the bluestore, where
RocksDB is used.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
some of rocksdb's dependencies may not have built by the time its
ExternalProject starts, so it can fail with missing headers or
libraries. for example, `uring::uring` may itself be an ExternalProject,
and its include directory won't exist until it starts building:
```
[89/1345] Performing configure step for 'rocksdb_ext'
FAILED: src/rocksdb_ext-prefix/src/rocksdb_ext-stamp/rocksdb_ext-configure build/src/rocksdb_ext-prefix/src/rocksdb_ext-stamp/rocksdb_ext-configure
...
CMake Error in CMakeLists.txt:
Imported target "uring::uring" includes non-existent path
"build/src/liburing/src/include"
in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include:
* The path was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and references files it does not
provide.
...
[91/1345] Performing download step (git clone) for 'liburing_ext'
Cloning into 'liburing'...
```
use `add_dependencies(rocksdb_ext)` to make sure all of its dependencies
are available before starting the build
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Currently, when attempting to build ceph on a system with fmt
installed, we try to build against it whatever the version. This
constantly breaks people's builds, since newer versions of fmt often
change the API.
This change specifies that versions must be below 10 as well as at or
above 8.1.1, so that on systems with a new format, we fall back to
using the submodule.
It also removes the `Findfmt.cmake` module, as that does not check
the installed version. Instead, we use the cmake config file installed by
the system package of fmt and does support version checking.
Signed-off-by: Adam Emerson <aemerson@redhat.com>
There are two problems:
1) The librocksdb does not use the built liburing if present.
2) You cannot turn off rocksdb's use of liburing with WITH_LIBURING=OFF.
This is to fix errors like:
FAILED: bin/ceph_test_keyvaluedb_iterators
: && /opt/rh/gcc-toolset-11/root/usr/bin/g++ -Og -g -rdynamic -pie src/test/ObjectMap/CMakeFiles/ceph_test_keyvaluedb_iterators.dir/test_keyvaluedb_iterators.cc.o src/test/ObjectMap/CMakeFiles/ceph_test_keyvaluedb_iterators.dir/KeyValueDBMemory.cc.o -o bin/ceph_test_keyvaluedb_iterators -Wl,-rpath,/home/pdonnell/scratch/build/lib lib/libos.a lib/libgmock_maind.a lib/libgmockd. a lib/libgtestd.a -lpthread -ldl lib/libglobal.a -ldl /usr/lib64/librt.so -lresolv -ldl lib/libblk.a /lib64/libaio.so src/liburing/src/liburing.a lib/libkv.a lib/libheap_profiler.a /lib64/libtcmalloc.so src/rocksdb/librocksdb.a /lib64/libsnappy.so /usr/lib64/liblz4.so /usr/lib64/libz.so /usr/lib64/libfuse.so lib/libceph-common.so.2 src/opentelemetry-cpp/sdk/src/trace/libopentelemetry_trace.a src/opentelemetry-cpp/sdk/src/resource/libopentelemetry_resources.a src/opentelemetry-cpp/sdk/src/common/libopentelemetry_common.a src/opentelemetry-cpp/exporters/jaeger/libopentelemetry_exporter_jaeger_trace.a src/opentelemetry-cpp/ext/src/http/client/curl/libopentelemetry_http_client_curl.a /usr/lib64/libcurl.so /usr/lib64/libthrift.so lib/libjson_spirit.a lib/libcommon_utf8.a lib/liberasure_code.a lib/libextblkdev.a -lcap boost/lib/libboost_thread.a boost/lib/libboost_chrono.a boost/lib/libboost_atomic.a boost/lib/libboost_system.a boost/lib/libboost_random.a boost/lib/libboost_program_options.a boost/lib/libboost_date_time.a boost/lib/libboost_iostreams.a boost/lib/libboost_regex.a lib/libfmtd.a /usr/lib64/libblkid.so -lpthread /usr/lib64/libcrypto.so /usr/lib64/libudev.so /usr/lib64/libz.so -ldl -lresolv -Wl,--as-needed -latomic && :
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `io_uring_wait_cqe_nr':
/home/pdonnell/scratch/build/src/liburing/src/include/liburing.h:494: undefined reference to `__io_uring_get_cqe'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
/home/pdonnell/ceph/src/rocksdb/env/fs_posix.cc:1125: undefined reference to `io_uring_get_sqe'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/fs_posix.cc:1134: undefined reference to `io_uring_submit'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(fs_posix.cc.o): in function `rocksdb::CreateIOUring()':
/home/pdonnell/ceph/src/rocksdb/env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `io_uring_wait_cqe_nr':
/home/pdonnell/scratch/build/src/liburing/src/include/liburing.h:494: undefined reference to `__io_uring_get_cqe'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:674: undefined reference to `io_uring_get_sqe'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:684: undefined reference to `io_uring_submit_and_wait'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: src/rocksdb/librocksdb.a(io_posix.cc.o): in function `rocksdb::PosixRandomAccessFile::ReadAsync(rocksdb::FSReadRequest&, rocksdb::IOOptions const&, std::function<void (rocksdb::FSReadRequest const&, void*)>, void*, void**, std::function<void (void*)>*, rocksdb::IODebugContext*)':
/home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:901: undefined reference to `io_uring_get_sqe'
/opt/rh/gcc-toolset-11/root/usr/bin/ld: /home/pdonnell/ceph/src/rocksdb/env/io_posix.cc:910: undefined reference to `io_uring_submit'
collect2: error: ld returned 1 exit status
Fixes: https://tracker.ceph.com/issues/63218
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
This is the MVP for a driver for RGW that operates on top of a POSIX
filesystem. It supports get, put, list, copy, multipart, external
access via the filesystem itself, and ordered bucket listings via an
LRU-based cache.
Note that this is currently a Filter, indended to run on top of dbstore.
This is because it currently doesn't have any User implementation, so it
depends on dbstore's User. Everything else is implemented in
POSIXDriver. Once there is a User implementation, this will become a
Store, instead of a Filter.
Commit messages from bucket listing cache:
rgw/posixdriver: recycle lmdb database handles as required
While LMDB workflows often do not close/return database handles,
ours continually reuses them. This requires us to close each
handle (atomically) when a cache entry is recycled.
rgw/posixdriver: don't instantiate bucket cache entries from notify events
rgw/posixdriver: incorporate lmdb-safe for now
The current inclusion is based on https://github.com/Martchus/lmdb-safe,
which is actively maintained but currently has some packaging issues the
author has agreed to accept fixes for.
For now, skip the submodule to save time and remove an external dependency.
rgw/posixdriver: fix listing of cached, empty bucket
* check lmdb enumeration result in all cases and w/better style
* add unit test for enumeration of an empty cached directory
rgw/posixdriver: nest lmdbs in a directory under the dbroot path to avoid cleanup issues
rgw/posixdriver: refactor for posix integration
* Derive BucketCache types as templates on a SAL driver and SAL
bucket pair.
* Integrate cache fills as callbacks into SAL layer (or mock, for
tests)
* Renaming and cleanups
rgw/posixdriver: add bucket cache implementation and tests
Adds free-standing cache of buckets and object names, with
bucket names (and listing attributes, upcoming) managed in
a hashed set of lmdb databases, which provides ordering and
a high-performance listing cache.
An framework for notification on new object creation (e.g.,
outside S3 workflow) is provided, and a Linux implementation
using inotify.
FindLMDB.cmake taken with attribution and license.
rgw/posixdriver: add zpp_bits serialization (FAST)
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Not correct in general, and a build bug because fio-objectstore
includes c++20 headers.
Fixes: https://tracker.ceph.com/issues/62778
Add CXX_EXTENSIONS ON, and cleanup INTERFACE_COMPILE_OPTIONS per Kefu review
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
- dropping the `cmake_policy()` calls which set the policy not supported
by 3.16.
- `include (FindPackageHandleStandardArgs)` without specifying the
relative path.
- dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
this option was introduced by CMake v3.19, see
https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.
the new FindPython3 module from CMake:
* enables us to find the recent Python intepreter and development files up to
CPython 3.13.
* finds Python intepreter with the new `Python_FIND_STRATEGY`. the old and
default strategy always finds the most recent version with all specified name
and in all locations. so, if /usr/bin/python exists, it would accept, even
if it is a symlink to python3.9 and what we want is python3.6. the
new policy stops at the one which satisfies the constraints. this
helps us to address the problem of issue we have when recreating
the building system after the CMakeLists.txt is updated.
Fixes: https://tracker.ceph.com/issues/62428
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
- dropping the `cmake_policy()` calls which set the policy not supported
by 3.16.
- `include (FindPackageHandleStandardArgs)` without specifying the
relative path.
- dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
this option was introduced by CMake v3.19, see
https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.
the new FindPython3 module from CMake:
* enables us to find the recent Python intepreter and development files up to
CPython 3.13.
* finds intepreter with the new `Python_FIND_STRATEGY`. the old and default
strategy always finds the most recent version with all specified name
and in all locations. so, if /usr/bin/python exists, it would accept, even
if it is a symlink to python3.9 and what we want is python3.6. while
the new policy stops at the one which satisfies the constraints.
simpler this way and less error prone.
Fixes: https://tracker.ceph.com/issues/62428
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
ubbd (Userspace Backend Block Device) is a method to build block device
and handle IO by process started in userspace. That means we can provide
a generic block device to user and handle IO requests by librbd
in linux.
This way, we can allow user to use rbd image as a linux block device
supporting full image features, especial the journaling feature, which
is not supported by krbd.
For more information: https://github.com/DataTravelGuide/ubbd
Signed-off-by: Dongsheng Yang <dongsheng.yang.linux@gmail.com>
Arrow Flight integration is triggered by defining
WITH_RADOSGW_ARROW_FLIGHT=ON with the cmake invocation.
For now this assumes that grpc-plugins is installed on the system and
won't be built internally.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
the vanilla FindBoost.cmake pulled from cmake has couple assumptions
which do not hold in our environment. so address them case by case.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
we are not interested in the compiling warning when building
Boost, and even worse we don't want to fail the build because of
the warnings which default to error. and "-w" does not disable
them for us. see also the discussion at
https://reviews.llvm.org/D131307.
to fully disable the warnings, let's add -Wno-everything to the
cflags.
Signed-off-by: Kefu Chai <kchai@redhat.com>
it is "b2 install" which builds the target. "stage" defines "install"
rule by copying the target to a single location. so we need to pass
them to the "install" rule. now that we already have "user-config.jam"
defining the used toolset like gcc, we can just use it for passing
the cflags and cxxflags.
in this change, the cflags set with BOOST_CXXFLAGS are persisted in
"user-config.jam". so they won't get lost when building Boost using
"b2 install".
also, strictly speaking the flags of "-fPIC -w" are not
CXXFLAGS, they are compile flags. so including them in "compileflags"
is more correct.
Signed-off-by: Kefu Chai <kchai@redhat.com>
to address the build failure like:
-- Checking for one of the modules 'libcares'
CMake Error at cmake/modules/BuildBoost.cmake:287 (_add_library):
_add_library cannot create ALIAS target "c-ares::c-ares" because target
"c-ares::cares" is imported but not globally visible.
Call Stack (most recent call first):
cmake/modules/Findc-ares.cmake:31 (add_library)
src/CMakeLists.txt:344 (find_package)
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
to be aligned with the c-ares upstream, which provides
c-ares-config.cmake in c-ares v1.17 and up. this cmake config file
is also packaged by fedora and CentOS stream in c-ares-devel-1.17.1 and
up.
this change prepares us for bumping up Seastar submodule, which also
has a similar change: aea45c98f2
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
To build with DAOS backend, use -DWITH_RADOSGW_DAOS=YES cmake
option. `daos-devel` rpm should be installed beforehand.
To connect to DAOS pool, add the following configuration
parameters to ceph.conf:
```
[client]
...
rgw backend store = daos
daos pool = tank
```
A pool could be created using the following command:
```
dmg pool create --size=<size> <pool_name>
```
To install `daos-devel` do:
```
sudo wget -O /etc/yum.repos.d/daos-packages.repo https://packages.daos.io/v2.0/EL8/packages/x86_64/daos_packages.repo
sudo rpm --import https://packages.daos.io/RPM-GPG-KEY
sudo yum install -y epel-release daos-server daos-client daos-devel
```
Co-authored-by: Walter Warniaha <walter.warniaha@seagate.com>
Signed-off-by: Zuhair AlSader <zuhair.alsader@seagate.com>
In 4a3afcf, the $PATH is set for the test, but we cannot set multiple
properties with a single `set_property()` cmake command. We fix that by
adding the installation path of jsonnet-bundler
(CMAKE_CURRENT_BINARY_DIR) to the $PATH used for every tox test.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Co-Authored-By: Kefu Chai <tchaikov@gmail.com>
cmake should fail when generating the building system, if the necessary
development libraries or the header files are missing.
in this change, Findmotr.cmake is added to detect the motr library
and header so that we can use it to find the motr package before using
it.
See-also: https://tracker.ceph.com/issues/57079
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
we are facing following FTBFS on jammy + GCC-11.2 + Cython 0.29 +
CMake 3.22:
creating /home/jenkins-build/build/workspace/ceph-api/build/lib/cython_modules/temp.linux-x86_64-3.10/home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs
compile options: '-I/usr/include/python3.10 -I/usr/include/python3.10 -c'
extra options: '-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -iquote/home/jenkins-build/build/workspace/ceph-api/src/include -w -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2'
cc: /home/jenkins-build/build/workspace/ceph-api/build/src/pybind/cephfs/cephfs.c
cc: warning: ##: linker input file unused because linking not done
cc: error: ##: linker input file not found: No such file or directory
cc: warning: 0: linker input file unused because linking not done
cc: error: 0: linker input file not found: No such file or directory
it seems cython is not able to escape the space in the "extra options"
anymore, so the "##" and "0" are considered as object files passed to
compiler in addition to cephfs.c.
in this change the spaces are removed to help cython to make the right
decision.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Prior to this change, Fedora and RHEL 9 on s390x would incorrectly pass
the HAVE_CXX11_ATOMIC check. As a consequence, the rest of Ceph would
fail to link because of missing atomic methods "__atomic_load_16",
"__atomic_store_16", "__atomic_compare_exchange_16".
Mark this method so that it is not optimized out, even with LTO. With
this change, s390x properly fails HAVE_CXX11_ATOMIC and falls back to
HAVE_LIBATOMIC.
Move the "BuildRequires: libatomic" RPM line out from the seastar+fedora
conditional and into the main fedora+rhel conditional so that the
HAVE_LIBATOMIC check will pass.
Fixes: https://tracker.ceph.com/issues/54514
Fixes: https://tracker.ceph.com/issues/56492
Co-authored-by: Kaleb S. Keithley <kkeithle@redhat.com>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
In order to enable the pmem character device, add ndctl=y to the parameter
of compiling the pmdk library when WITH_BLUESTORE_PMEM is ON. Because
after find_ndctl and find_daxctl are added, the dependency packages
required by WITH_BLUESTORE_PMEM and WITH_RBD_RWL become more different.
So separated these two. libpmem has no version required. libpmemobj
required version >=1.8.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
In order to support the character device of pmem usage in bluestore via
libpmem built by Ceph itself, we need to enable daxctl and ndctl
dependency. add the installation of ndctl and find it. the version of
ndctl and daxctl library requires >63. "apt-get install" meet the version
under ubuntu focal.
the installation of ndctl-devel in ceph.spec.in has not been verified.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
At first, libpmem was the only library. Later, pmem related libraries
such as libpmemobj and libpmem2 were gradually added. These libraries
were also integrated into one named pmdk. So rename to pmdk.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
We were using git fetch and an unofficial remote to fetch
opentelemetry-cpp, as it needed additional patch to include boost
headers, with the merge of
open-telemetry/opentelemetry-cpp#1100open-telemetry/opentelemetry-cpp#1020
we do not require to maintain our own patched version. Also, we were
using git fetch as a temporary change, now that tracing is always on, it
should be right time to adapt to using opentelemetry-cpp as a submodule.
This removes compile time fetch for opentelemetry-cpp instead use
official opentelemetry-cpp lib as a submodule.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
It's hitting pacific with a nuisance -Werror=array-parameter= const
char * vs const char[37] mismatch. Follow commit 91a616b26e ("cmake:
pass RTE_DEVEL_BUILD=n when building dpdk") and just disable -Werror.
Fixes: https://tracker.ceph.com/issues/55977
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
