- ubuntu focal ships libcryptsetup-dev (2:2.2.2),
- centos 8 app stream comes with cryptsetup-devel-2.3.3.
- openSUSE Leap 15.3 packages libcryptsetup-devel-2.3.4
- openSUSE Leap 15.2 packages libcryptsetup-devel-2.0.5
so we can drop the support for libcryptsetup < 2.0.5
see also ea3c1bfb9e
Signed-off-by: Kefu Chai <kchai@redhat.com>
When running dbstore as RGW backend, use the logging mechanism
and config values provided by default by RGW.
For other tests, set logfile and loglevel to the input values provided.
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
As part of Zipper, adding support for DB backend (using SQLite) for Stackable RGW.
The base class methods implemented are generic which could be extended to any backend database, not just SQLite.
More details on design/implementation can be found at -
https://docs.google.com/document/d/1xCoHT5DCujqbe1pnEfYpSaSqiBcW8z87CgQCC5LivOo/edit#
Current status:
[Done]
- User related APIs
- Bucket APIs
- Testcases (using Gtest) to test the DBStore APIs
[Not handled in the first pass]
- Quota
- Usage Stats
- Swift Users
- Multiple Zones/Zonegroups
[WIP]
- Object APIs
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
the upstream project of OpenLDAP is named "OpenLDAP", so rename OPENLDAP to
OpenLDAP for better readability.
Signed-off-by: Kefu Chai <kchai@redhat.com>
add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.
this change partially reverts 5fb90bb796.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
since we dropped the support of bionic, we now have the luxury of using
newer CMake! and by using CMake 3.16, we now have access to
* add_compile_definitions(), which was introduced in cmake 3.12
also, drop the comment of
> remove cmake/modules/FindPython* once 3.12 is required
because we need to support python 3.10 in future, better off keeping
the FindPython files around, so we can backport cmake changes from
upstream when necessary.
we are allowed to bump up the CMake's required to 3.16, because:
- ubuntu focal ships with cmake 3.16.3
- RHEL/CentOS8's AppStream has cmake 3.18
Signed-off-by: Kefu Chai <kchai@redhat.com>
ceph::allocator was originally introduced to address
https://tracker.ceph.com/issues/39703, so that if tcmalloc < 2.6.2
detected, the homebrew allocator is used. otherwise std::allocator would
use the aligned_alloc() function which was not implemented by tcmalloc <
2.6.2. but the fix in tcmalloc implementing aligned_alloc() was
introduced in https://github.com/gperftools/gperftools/commit/d406f228.
that fix was included since gperftools 2.6.2.
- CentOS/RHEL8 includes gperftools 2.7,
- fedora 33 includes gperftools 2.8,
- ubuntu/focal comes with libgoogle-perftools-dev 2.7
- ubuntu/bionic ships libgoogle-perftools-dev 2.5. but since we
detect libtcmalloc 2.6.2, when building on bionic, JeMalloc
or libc allocator is used as fallback.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The Windows build is hitting linking errors after
bumping the Boost version to 1.75. The issue is that Boost
is now setting the zlib dependecy using INTERFACE_LINK_LIBRARIES,
which means that it's no longer located using the standard
"find_package" mechanism.
In order for the linker to locate zlib, we'll add it to the
linker search path.
[1] https://github.com/boostorg/boost_install/issues/47
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
it does not depend on WITH_MGR_DASHBOARD_FRONTEND, which is disabled by
default and is used for enable/disable the inclusion of dashboard
support. while root client is used by orchestrator. so it should depend
on WITH_MGR not WITH_MGR_DASHBOARD_FRONTEND.
this change addresses the regression introduced by
1003f1ffee
Signed-off-by: Kefu Chai <kchai@redhat.com>
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.
this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.
also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.
* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
files, the subvars in it is then replaced with CMake variables,
and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
is not a CMake variable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
as FreeBSD also has msghdr but it does not have systemd, or
flags like MFD_ALLOW_SEALING, O_TMPFILE or F_SEAL_GROW. so
use WITH_SYSTEMD for enabling journald backend of logging system.
also move the option of "WITH_SYSTEMD" up so that the src/CMakeLists.txt
is able to see the variable of WITH_SYSTEMD defined by it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
because ssl::host_name_verification was introduced in Asio 1.16.1 / Boost 1.73, and we
are using this class in src/librbd/migration/HttpClient.cc
This is an known error with Boost:
https://cplusplus.github.io/LWG/issue3171
without this change, on a host with boost 1.72
it will complain (on FreeBSD/Clang):
```
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:699:37: error: no member named 'host_name_verification' in namespace 'boost::asio::ssl'
[host, next=boost::asio::ssl::host_name_verification(host),
~~~~~~~~~~~~~~~~~~^
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:714:16: error: no matching function for call to 'next'
return next(preverified, ctx);
^~~~
```
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
to limit the parallel level of compile jobs which consumes lots of memory
to compile and/or link.
compile following targets with heavy_compile_pool.
* unittest_librbd
* crimson*
* ceph-dencoder
some of the .cc files in the target above use up to 6 GiB memory to
compile.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This is a complete rewrite on top of SimpleRADOSStriper as well as the
API. The VFS is now a loadable extension as well.
Fixes: https://tracker.ceph.com/issues/40609
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
This library provides a SQLite front-end to the RADOS objects.
This effort will help alleviate the restriction on number of key-value pairs
that can be stored in an object.
This interface is a generic one without any constraint on the database
schema either. Library clients can enforce any schema and use SQLite API
to store data in the database backed by RADOS Objects.
Signed-off-by: Milind Changire <mchangir@redhat.com>
this option is available only if CMAKE_BUILD_TYPE is Debug.
this change helps us to unify the checks for WITH_CEPH_DEBUG_MUTEX,
without this change, we always have to check both WITH_CEPH_DEBUG_MUTEX
*and* CMAKE_BUILD_TYPE.
after this change, we only respect WITH_CEPH_DEBUG_MUTEX.
Signed-off-by: Kefu Chai <kchai@redhat.com>
In order to expose ceph filesystems to Windows hosts, we propose
including ceph-dokan[1][2] in the Ceph tree, while updating it to
work with the latest CephFS and Dokany APIs.
Dokany is a well maintained project (fork of the original Dokan
project), allowing filesystems to be implemented in userspace,
even providing a Fuse compatibility layer.
One reason for not using the FUSE compatibility layer is that it's
only covering the high level API while Ceph is using the low level
FUSE API, which among other things is inode centric.
Changes made by this patch compared to the upstream ceph-dokan:
* support latest stable Dokany API. The upstream version relies on
the legacy unmaintained Dokan API
* return proper error codes, converting standard errno.h values to
NTSTATUS
* minor changes to support latest cephfs API
* drop duplicated ceph code, no longer needed if we're to include it
in tree. This makes it much easier to maintain.
* drop redundant permission checks, leaving it up to libcephfs
* use ceph argparse helpers
* use ceph logging and daemon initialization
* fixed unicode handling
* switched to ceph coding style
* made ceph.conf param optional, using the default path if available
* enabled setting file timestamps
* append support
* configurable timeouts set once per mount
* ensure that the error code is always logged
* various cleanups (removed unused entry points, checks that have
been moved to dokany, simplified conditional statements,
unnecessary conversions in the hot path, etc).
[1] https://github.com/ketor/ceph-dokan
[2] https://github.com/ceph/ceph-dokan
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Downstream Red Hat products do not support the older QCOW format. This
will allow the support for the legacy QCOW format to be disabled for the
new RBD import-only migration support.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
to be more consistent with other find_program() calls.
and more correct this way, as ${CCACHE_FOUND} is not a boolean, it
is a path pointing to the found cache executable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the WITH_SYSTEM_BOOST binaries are not built with BOOST_USE_VALGRIND, so
it probably isn't safe to define for the headers only
this flag is needed for teuthology testing, and the shaman builds use
WITH_SYSTEM_BOOST=OFF. so the better fix is to enable WITH_BOOST_VALGRIND
so BuildBoost.cmake will build the libraries with valgrind support and add
-DBOOST_USE_VALGRIND to the necessary targets
this change was merged in https://github.com/ceph/ceph-build/pull/1736
Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:
* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased
In order to do a dynamic build, we'll:
* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
executable entry points
* by default, cmake generates import libs for executables. The issue
is that for rados.exe, it generates librados.dll.a, thus overriding
the librados.dll import library, which breaks the build process.
We'll configure cmake to skip generating import libs for executables.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
We're porting to Windows some of the unit tests, mostly related to
rados and rbd.
This change filters out the tests that aren't supported on Windows.
At the same time, we're defining some missing dependencies since
unlike ELFs SOs, all DLL symbols have to be resolved at compile time.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
* refs/pull/32776/head:
win32*.sh: fetch WNBD dependency
win32*.sh: bump win32 boost dependency
cmake: skip osd, Lua and neorados on Windows
win32*.sh: improve lib handling
win32*.sh: update openssl location
win32*.sh: cleanup dependency build dir when targetting Windows
win32*.sh: Strip binaries individually
win32*.sh: Cleanup zip archive creation
cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings
win32*.sh: Use nproc for CPU count when targeting win32
test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling
win32*.sh: improve win32 dependency check
win32*.sh: isolate network operations when building win32 dependencies
cmake: drop Python dependency when targeting Windows
win32*.sh: cleanup Windows build script
win32*.sh: fix patch command
win32*.sh: update Windows build scripts and readme
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
* This commit introduces Jaegertracing library as package libjaeger,
pickwhich would be consumed by other ceph pacakges such as ceph-common0
* adds the following dependencies, which would be build from source
using ExternalProjectHelper.cmake +IncludeJaeger.cmake +
Build<package>.cmake scripts:
jaegertracing: v0.6.0 [added as a submodule]
opentracing: v1.6.0 [added as a submodule]
thrift: 0.13.0 [added as a submodule]
yaml-cpp: 0.6.0
json(optional)
* updates Boost to be installed instead of being build only, because
jaegertracing them during their build process.
* ceph.spec.in: introduces a default enabled jaeger packaging option,
which could be disabled using --without-jaeger flag during rpmbuild
* note: libjaeger package if enabled will be a dependency on ceph-common, ceph-mon, rgw_common and transitively will be a dependency for modules that have them as a dependency.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
The Python bindings aren't supported on Windows for the time being,
for which reason we're dropping the Python dependency.
When cross-compiling, Cmake sometimes complains about not finding
the Python files instead of just using the host files.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
src,cmake: define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT for Boost.Asio users and cleanups
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
This commit introduces internal (not yet part of the api) librbd functions for:
1. formating an RBD image in LUKS format
2. parsing an RBD image in LUKS format
The actual implementation of the LUKS format is done via libcryptsetup, which is added as a new dependency.
Signed-off-by: Or Ozeri <oro@il.ibm.com>
to search prefixes specified by the <PackageName>_ROOT CMake
variable and the <PackageName>_ROOT environment variable. find_package()
in CMake 3.12 started to use these prefixes if it is enabled.
Signed-off-by: Kefu Chai <kchai@redhat.com>
`acconfig.h` is generated using
configure_file(
${CMAKE_SOURCE_DIR}/src/include/config-h.in.cmake
${CMAKE_BINARY_DIR}/include/acconfig.h
)
in `config-h.in.cmake`, the cmake variable of `HAVE_LIBZBD` is checked.
so we need to ensure that this variable is visible from this
`configure_file()` statement.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Adds build option and implements init functionality for SSD cache
Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
errno.h values aren't the same across platforms, for which reason
we'll have to convert remote errors.
We'll use the cmake "add_definitions" function instead of
"add_compile_options", which generates incorrect compiler params
while "add_compile_definitions" isn't available at all when
using CMake 3.10, which is by default available on Ubuntu 18.04.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Libzbc maintainers recommend switching to libzbd, which is lighter and supports
both ZNS SSDs and HM-SMR HDDs.
Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>
* cmake/modules/FindSanitizers.cmake: do not pollute CMAKE_REQUIRED_FLAGS
* cmake/modules/FindSanitizers.cmake: expose Sanitizers_COMPILE_OPTIONS
as a list
* CMakeLists.txt: append Sanitizers_COMPILE_OPTIONS to
*_LINKER_FLAGS after replacing ";" with " " in it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Boost::iostreams links against zlib. and FindBoost.cmake
does not add this linkage to Boost::iostreams target, let's
do this after `find_package(Boost...)`. in theory, it'd be
better to have this change in FindBoost.cmake, but it's
error-prone, and increases the risk of regression when
we sync our own copy of FindBoost.cmake with CMake upstream.
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0xf7): undefined reference to `crc32'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x165): undefined reference to
`deflateReset'../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x185): undefined reference to `inflateEnd'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x1a1): undefined reference to
`inflateReset'../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x1b3): undefined reference to `deflateEnd'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x38d): undefined reference to `inflateInit2_'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x3d5): undefined reference to `deflateInit2_'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x134): undefined reference to `deflate'
../../build.deps/mingw/boost/lib/libboost_iostreams.a(zlib.o):zlib.cpp:(.text+0x144): undefined reference to `inflate'
collect2: error: ld returned 1 exit status
src/CMakeFiles/ceph-common.dir/build.make:441: recipe for target 'bin/libceph-common.dll' failed
Signed-off-by: Kefu Chai <kchai@redhat.com>
For things like cephadm, where there is a lot of "from X import Y",
the import tags become cumbersome. .tox dirs and
python-common/build are just repeats of source files found elsewhere
so result in duplicate tags
Signed-off-by: Dan Mick <dmick@redhat.com>
WITH_PYTHON2 option was deprecated in
5fc657b40d, that commit was included by
Ceph v15.1.0 and up, assuming all downstream packagings have removed
WITH_PYTHON2 option, we are now removing it completely.
Signed-off-by: Kefu Chai <kchai@redhat.com>
OFED_PREFIX was added to compile with xio messenger for supporting RDMA
using the Accelio and/or MLNX_OFED package. but xio messenger was removed in
in cc9a9142fd
let remove the leftover in CMakeLists
Signed-off-by: Kefu Chai <kchai@redhat.com>
instead of appending compile flags to CMAKE_C_FLAGS, use
add_compile_options(), as COMPILE_OPTIONS is a list, it'd simpler to
append options to it and to access it in a structured way.
Signed-off-by: Kefu Chai <kchai@redhat.com>
so systemd/CMakeLists.txt can have access to the variables defined by
it. quote from https://cmake.org/cmake/help/latest/command/include.html.
> Variable reads and writes access the scope of the caller (dynamic
> scoping).
Signed-off-by: Kefu Chai <kchai@redhat.com>
Windows provides socket and other related structures that are mostly
following the Posix standards, but using different headers.
Rather than adding lots of platform checks and require future commits
to do so as well, we're adding headers with the same names, which
in turn include the required Windows headers.
In a few cases, some functions declared by unistd.h are actually
defined by different Windows headers, so we'll need additional includes.
One thing to note here is that boost requires us to include aio.hpp
before the Windows socket headers.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
an option named "mgr_disabled_modules" is added in this change to
prevent mgr from loading modules listed in this option. because mgr
loads *all* modules found in the configured path, and per
https://tracker.ceph.com/issues/45147, python subinterpreter could hang
when loading numpy, so this behavior practically creates a deadlock
in mgr.
this issue is found when mgr uses python3.8 runtime. in development
environment, it'd be inconvenient to disable the offending mgr module
without changing the source code, even if we can choose to not install
them, for instance, the enduser can workaround this issue by
uninstalling `ceph-mgr-diskprediction-local`.
an option would be useful in this case, so we can add the module to the
list before mgr tries to load it.
as this issue is found with python3.8 + diskprediction_local (numpy), so
this mgr module is disabled by default if mgr is compiled with python3.8
runtime.
Fixes: https://tracker.ceph.com/issues/45147
Signed-off-by: Kefu Chai <kchai@redhat.com>
1. WITH_LIBURING is used to set HAVE_LIBURING to decide
use liburing in KernelDevice or not.
2. WITH_SYSTEM_LIBURING is to choose use system installed
liburing or build the liburing from source code.
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
To build Boost.Context (and other libraries) with support to allow
them to be valground usefully, and to include the define to link
against them.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
We'll update the cmake files in order to be able to build ceph
components for Windows targets.
Cross compiling using MINGW is the easiest approach for now.
Subsequently, we'll add support for Clang and MSVC.
This patch provides the following changes:
* include winsock2.h, which provides ntohl on Windows
* avoid unsupported compiler flags when using msvc
* add a custom toolchain file for mingw
* update install command for ceph-common in order to work with mingw
* avoid running test sample when cross compiling
* link against the ws_32 lib
* set argeted Windows version
* skip yasm checks when targeting Windows
* allow multiple redefinitions when using mingw, picking the first one.
this is a workaround for a mingw TLS bug:
https://sourceforge.net/p/mingw-w64/bugs/816/
Signed-off-by: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com>
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
python version for python binding and ceph-mgr embedded python
interpreter anymore. as before switching to python3-only build,
we can build python3 and python2 python bindings, and ceph-mgr
can use either of them. but after switching to python3-only
build, ceph-mgr has to use whatever python version used to
build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
parameter anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
