Fixing build conflicts with latest sal multipart/writer changes.
Also enable WITH_RADOSGW_DBSTORE option by default.
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
libcls_kvs was introduced back in
73d016fdb3, but we don't have an internal
user so far. to reduce the build time. let's disable the build of it by
default.
Signed-off-by: Kefu Chai <kchai@redhat.com>
- ubuntu focal ships libcryptsetup-dev (2:2.2.2),
- centos 8 app stream comes with cryptsetup-devel-2.3.3.
- openSUSE Leap 15.3 packages libcryptsetup-devel-2.3.4
- openSUSE Leap 15.2 packages libcryptsetup-devel-2.0.5
so we can drop the support for libcryptsetup < 2.0.5
see also ea3c1bfb9e
Signed-off-by: Kefu Chai <kchai@redhat.com>
When running dbstore as RGW backend, use the logging mechanism
and config values provided by default by RGW.
For other tests, set logfile and loglevel to the input values provided.
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
As part of Zipper, adding support for DB backend (using SQLite) for Stackable RGW.
The base class methods implemented are generic which could be extended to any backend database, not just SQLite.
More details on design/implementation can be found at -
https://docs.google.com/document/d/1xCoHT5DCujqbe1pnEfYpSaSqiBcW8z87CgQCC5LivOo/edit#
Current status:
[Done]
- User related APIs
- Bucket APIs
- Testcases (using Gtest) to test the DBStore APIs
[Not handled in the first pass]
- Quota
- Usage Stats
- Swift Users
- Multiple Zones/Zonegroups
[WIP]
- Object APIs
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
the upstream project of OpenLDAP is named "OpenLDAP", so rename OPENLDAP to
OpenLDAP for better readability.
Signed-off-by: Kefu Chai <kchai@redhat.com>
add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.
this change partially reverts 5fb90bb796.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
since we dropped the support of bionic, we now have the luxury of using
newer CMake! and by using CMake 3.16, we now have access to
* add_compile_definitions(), which was introduced in cmake 3.12
also, drop the comment of
> remove cmake/modules/FindPython* once 3.12 is required
because we need to support python 3.10 in future, better off keeping
the FindPython files around, so we can backport cmake changes from
upstream when necessary.
we are allowed to bump up the CMake's required to 3.16, because:
- ubuntu focal ships with cmake 3.16.3
- RHEL/CentOS8's AppStream has cmake 3.18
Signed-off-by: Kefu Chai <kchai@redhat.com>
ceph::allocator was originally introduced to address
https://tracker.ceph.com/issues/39703, so that if tcmalloc < 2.6.2
detected, the homebrew allocator is used. otherwise std::allocator would
use the aligned_alloc() function which was not implemented by tcmalloc <
2.6.2. but the fix in tcmalloc implementing aligned_alloc() was
introduced in https://github.com/gperftools/gperftools/commit/d406f228.
that fix was included since gperftools 2.6.2.
- CentOS/RHEL8 includes gperftools 2.7,
- fedora 33 includes gperftools 2.8,
- ubuntu/focal comes with libgoogle-perftools-dev 2.7
- ubuntu/bionic ships libgoogle-perftools-dev 2.5. but since we
detect libtcmalloc 2.6.2, when building on bionic, JeMalloc
or libc allocator is used as fallback.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The Windows build is hitting linking errors after
bumping the Boost version to 1.75. The issue is that Boost
is now setting the zlib dependecy using INTERFACE_LINK_LIBRARIES,
which means that it's no longer located using the standard
"find_package" mechanism.
In order for the linker to locate zlib, we'll add it to the
linker search path.
[1] https://github.com/boostorg/boost_install/issues/47
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
it does not depend on WITH_MGR_DASHBOARD_FRONTEND, which is disabled by
default and is used for enable/disable the inclusion of dashboard
support. while root client is used by orchestrator. so it should depend
on WITH_MGR not WITH_MGR_DASHBOARD_FRONTEND.
this change addresses the regression introduced by
1003f1ffee
Signed-off-by: Kefu Chai <kchai@redhat.com>
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.
this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.
also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.
* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
files, the subvars in it is then replaced with CMake variables,
and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
is not a CMake variable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
as FreeBSD also has msghdr but it does not have systemd, or
flags like MFD_ALLOW_SEALING, O_TMPFILE or F_SEAL_GROW. so
use WITH_SYSTEMD for enabling journald backend of logging system.
also move the option of "WITH_SYSTEMD" up so that the src/CMakeLists.txt
is able to see the variable of WITH_SYSTEMD defined by it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
because ssl::host_name_verification was introduced in Asio 1.16.1 / Boost 1.73, and we
are using this class in src/librbd/migration/HttpClient.cc
This is an known error with Boost:
https://cplusplus.github.io/LWG/issue3171
without this change, on a host with boost 1.72
it will complain (on FreeBSD/Clang):
```
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:699:37: error: no member named 'host_name_verification' in namespace 'boost::asio::ssl'
[host, next=boost::asio::ssl::host_name_verification(host),
~~~~~~~~~~~~~~~~~~^
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:714:16: error: no matching function for call to 'next'
return next(preverified, ctx);
^~~~
```
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
to limit the parallel level of compile jobs which consumes lots of memory
to compile and/or link.
compile following targets with heavy_compile_pool.
* unittest_librbd
* crimson*
* ceph-dencoder
some of the .cc files in the target above use up to 6 GiB memory to
compile.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This is a complete rewrite on top of SimpleRADOSStriper as well as the
API. The VFS is now a loadable extension as well.
Fixes: https://tracker.ceph.com/issues/40609
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
This library provides a SQLite front-end to the RADOS objects.
This effort will help alleviate the restriction on number of key-value pairs
that can be stored in an object.
This interface is a generic one without any constraint on the database
schema either. Library clients can enforce any schema and use SQLite API
to store data in the database backed by RADOS Objects.
Signed-off-by: Milind Changire <mchangir@redhat.com>
this option is available only if CMAKE_BUILD_TYPE is Debug.
this change helps us to unify the checks for WITH_CEPH_DEBUG_MUTEX,
without this change, we always have to check both WITH_CEPH_DEBUG_MUTEX
*and* CMAKE_BUILD_TYPE.
after this change, we only respect WITH_CEPH_DEBUG_MUTEX.
Signed-off-by: Kefu Chai <kchai@redhat.com>
In order to expose ceph filesystems to Windows hosts, we propose
including ceph-dokan[1][2] in the Ceph tree, while updating it to
work with the latest CephFS and Dokany APIs.
Dokany is a well maintained project (fork of the original Dokan
project), allowing filesystems to be implemented in userspace,
even providing a Fuse compatibility layer.
One reason for not using the FUSE compatibility layer is that it's
only covering the high level API while Ceph is using the low level
FUSE API, which among other things is inode centric.
Changes made by this patch compared to the upstream ceph-dokan:
* support latest stable Dokany API. The upstream version relies on
the legacy unmaintained Dokan API
* return proper error codes, converting standard errno.h values to
NTSTATUS
* minor changes to support latest cephfs API
* drop duplicated ceph code, no longer needed if we're to include it
in tree. This makes it much easier to maintain.
* drop redundant permission checks, leaving it up to libcephfs
* use ceph argparse helpers
* use ceph logging and daemon initialization
* fixed unicode handling
* switched to ceph coding style
* made ceph.conf param optional, using the default path if available
* enabled setting file timestamps
* append support
* configurable timeouts set once per mount
* ensure that the error code is always logged
* various cleanups (removed unused entry points, checks that have
been moved to dokany, simplified conditional statements,
unnecessary conversions in the hot path, etc).
[1] https://github.com/ketor/ceph-dokan
[2] https://github.com/ceph/ceph-dokan
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Downstream Red Hat products do not support the older QCOW format. This
will allow the support for the legacy QCOW format to be disabled for the
new RBD import-only migration support.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
to be more consistent with other find_program() calls.
and more correct this way, as ${CCACHE_FOUND} is not a boolean, it
is a path pointing to the found cache executable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the WITH_SYSTEM_BOOST binaries are not built with BOOST_USE_VALGRIND, so
it probably isn't safe to define for the headers only
this flag is needed for teuthology testing, and the shaman builds use
WITH_SYSTEM_BOOST=OFF. so the better fix is to enable WITH_BOOST_VALGRIND
so BuildBoost.cmake will build the libraries with valgrind support and add
-DBOOST_USE_VALGRIND to the necessary targets
this change was merged in https://github.com/ceph/ceph-build/pull/1736
Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:
* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased
In order to do a dynamic build, we'll:
* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
executable entry points
* by default, cmake generates import libs for executables. The issue
is that for rados.exe, it generates librados.dll.a, thus overriding
the librados.dll import library, which breaks the build process.
We'll configure cmake to skip generating import libs for executables.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
We're porting to Windows some of the unit tests, mostly related to
rados and rbd.
This change filters out the tests that aren't supported on Windows.
At the same time, we're defining some missing dependencies since
unlike ELFs SOs, all DLL symbols have to be resolved at compile time.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
* refs/pull/32776/head:
win32*.sh: fetch WNBD dependency
win32*.sh: bump win32 boost dependency
cmake: skip osd, Lua and neorados on Windows
win32*.sh: improve lib handling
win32*.sh: update openssl location
win32*.sh: cleanup dependency build dir when targetting Windows
win32*.sh: Strip binaries individually
win32*.sh: Cleanup zip archive creation
cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings
win32*.sh: Use nproc for CPU count when targeting win32
test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling
win32*.sh: improve win32 dependency check
win32*.sh: isolate network operations when building win32 dependencies
cmake: drop Python dependency when targeting Windows
win32*.sh: cleanup Windows build script
win32*.sh: fix patch command
win32*.sh: update Windows build scripts and readme
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
* This commit introduces Jaegertracing library as package libjaeger,
pickwhich would be consumed by other ceph pacakges such as ceph-common0
* adds the following dependencies, which would be build from source
using ExternalProjectHelper.cmake +IncludeJaeger.cmake +
Build<package>.cmake scripts:
jaegertracing: v0.6.0 [added as a submodule]
opentracing: v1.6.0 [added as a submodule]
thrift: 0.13.0 [added as a submodule]
yaml-cpp: 0.6.0
json(optional)
* updates Boost to be installed instead of being build only, because
jaegertracing them during their build process.
* ceph.spec.in: introduces a default enabled jaeger packaging option,
which could be disabled using --without-jaeger flag during rpmbuild
* note: libjaeger package if enabled will be a dependency on ceph-common, ceph-mon, rgw_common and transitively will be a dependency for modules that have them as a dependency.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
The Python bindings aren't supported on Windows for the time being,
for which reason we're dropping the Python dependency.
When cross-compiling, Cmake sometimes complains about not finding
the Python files instead of just using the host files.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
