ceph::allocator was originally introduced to address
https://tracker.ceph.com/issues/39703, so that if tcmalloc < 2.6.2
detected, the homebrew allocator is used. otherwise std::allocator would
use the aligned_alloc() function which was not implemented by tcmalloc <
2.6.2. but the fix in tcmalloc implementing aligned_alloc() was
introduced in https://github.com/gperftools/gperftools/commit/d406f228.
that fix was included since gperftools 2.6.2.
- CentOS/RHEL8 includes gperftools 2.7,
- fedora 33 includes gperftools 2.8,
- ubuntu/focal comes with libgoogle-perftools-dev 2.7
- ubuntu/bionic ships libgoogle-perftools-dev 2.5. but since we
detect libtcmalloc 2.6.2, when building on bionic, JeMalloc
or libc allocator is used as fallback.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The Windows build is hitting linking errors after
bumping the Boost version to 1.75. The issue is that Boost
is now setting the zlib dependecy using INTERFACE_LINK_LIBRARIES,
which means that it's no longer located using the standard
"find_package" mechanism.
In order for the linker to locate zlib, we'll add it to the
linker search path.
[1] https://github.com/boostorg/boost_install/issues/47
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
it does not depend on WITH_MGR_DASHBOARD_FRONTEND, which is disabled by
default and is used for enable/disable the inclusion of dashboard
support. while root client is used by orchestrator. so it should depend
on WITH_MGR not WITH_MGR_DASHBOARD_FRONTEND.
this change addresses the regression introduced by
1003f1ffee
Signed-off-by: Kefu Chai <kchai@redhat.com>
extract the options in common/options.cc into separate .yaml.in
files, and preprocess them using CMake before translating them into .cc
files using a python script.
this change paves the road to render the options using sphinx, and
will allow us to further annotate the options to include more metadata.
also, a this YAML file can be consumed by applications like dashboard
and Sphinx to consume these metadata in a simpler way.
* use @variable-name@ for substituting the variables in .yaml.in file
* use cmake variable of `mgr_disabled_modules` instead of C macro
to define `mgr_disabled_modules` in global.yaml.in
* debian/control, ceph.spec.in, win32_deps_build.sh: add python3-yaml
as build dep
* add y2c.py (short for YAML to C++) to translate .yaml to .cc file
* common/options/*.yaml.in: extract and split options into .yaml.in
files, the subvars in it is then replaced with CMake variables,
and copied to the corresponding .yaml files
* include/config-h.in.cmake: remove MGR_DISABLED_MODULES, as it
is not a CMake variable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
as FreeBSD also has msghdr but it does not have systemd, or
flags like MFD_ALLOW_SEALING, O_TMPFILE or F_SEAL_GROW. so
use WITH_SYSTEMD for enabling journald backend of logging system.
also move the option of "WITH_SYSTEMD" up so that the src/CMakeLists.txt
is able to see the variable of WITH_SYSTEMD defined by it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
because ssl::host_name_verification was introduced in Asio 1.16.1 / Boost 1.73, and we
are using this class in src/librbd/migration/HttpClient.cc
This is an known error with Boost:
https://cplusplus.github.io/LWG/issue3171
without this change, on a host with boost 1.72
it will complain (on FreeBSD/Clang):
```
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:699:37: error: no member named 'host_name_verification' in namespace 'boost::asio::ssl'
[host, next=boost::asio::ssl::host_name_verification(host),
~~~~~~~~~~~~~~~~~~^
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:714:16: error: no matching function for call to 'next'
return next(preverified, ctx);
^~~~
```
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
to limit the parallel level of compile jobs which consumes lots of memory
to compile and/or link.
compile following targets with heavy_compile_pool.
* unittest_librbd
* crimson*
* ceph-dencoder
some of the .cc files in the target above use up to 6 GiB memory to
compile.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This is a complete rewrite on top of SimpleRADOSStriper as well as the
API. The VFS is now a loadable extension as well.
Fixes: https://tracker.ceph.com/issues/40609
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
This library provides a SQLite front-end to the RADOS objects.
This effort will help alleviate the restriction on number of key-value pairs
that can be stored in an object.
This interface is a generic one without any constraint on the database
schema either. Library clients can enforce any schema and use SQLite API
to store data in the database backed by RADOS Objects.
Signed-off-by: Milind Changire <mchangir@redhat.com>
this option is available only if CMAKE_BUILD_TYPE is Debug.
this change helps us to unify the checks for WITH_CEPH_DEBUG_MUTEX,
without this change, we always have to check both WITH_CEPH_DEBUG_MUTEX
*and* CMAKE_BUILD_TYPE.
after this change, we only respect WITH_CEPH_DEBUG_MUTEX.
Signed-off-by: Kefu Chai <kchai@redhat.com>
In order to expose ceph filesystems to Windows hosts, we propose
including ceph-dokan[1][2] in the Ceph tree, while updating it to
work with the latest CephFS and Dokany APIs.
Dokany is a well maintained project (fork of the original Dokan
project), allowing filesystems to be implemented in userspace,
even providing a Fuse compatibility layer.
One reason for not using the FUSE compatibility layer is that it's
only covering the high level API while Ceph is using the low level
FUSE API, which among other things is inode centric.
Changes made by this patch compared to the upstream ceph-dokan:
* support latest stable Dokany API. The upstream version relies on
the legacy unmaintained Dokan API
* return proper error codes, converting standard errno.h values to
NTSTATUS
* minor changes to support latest cephfs API
* drop duplicated ceph code, no longer needed if we're to include it
in tree. This makes it much easier to maintain.
* drop redundant permission checks, leaving it up to libcephfs
* use ceph argparse helpers
* use ceph logging and daemon initialization
* fixed unicode handling
* switched to ceph coding style
* made ceph.conf param optional, using the default path if available
* enabled setting file timestamps
* append support
* configurable timeouts set once per mount
* ensure that the error code is always logged
* various cleanups (removed unused entry points, checks that have
been moved to dokany, simplified conditional statements,
unnecessary conversions in the hot path, etc).
[1] https://github.com/ketor/ceph-dokan
[2] https://github.com/ceph/ceph-dokan
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Downstream Red Hat products do not support the older QCOW format. This
will allow the support for the legacy QCOW format to be disabled for the
new RBD import-only migration support.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
to be more consistent with other find_program() calls.
and more correct this way, as ${CCACHE_FOUND} is not a boolean, it
is a path pointing to the found cache executable.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the WITH_SYSTEM_BOOST binaries are not built with BOOST_USE_VALGRIND, so
it probably isn't safe to define for the headers only
this flag is needed for teuthology testing, and the shaman builds use
WITH_SYSTEM_BOOST=OFF. so the better fix is to enable WITH_BOOST_VALGRIND
so BuildBoost.cmake will build the libraries with valgrind support and add
-DBOOST_USE_VALGRIND to the necessary targets
this change was merged in https://github.com/ceph/ceph-build/pull/1736
Fixes: https://tracker.ceph.com/issues/48963
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* refs/pull/39039/head:
src/test/cli/monmaptool: adjust for new monmap features
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
qa/suites/rados/upgrade: include pacific-x
qa/suites/upgrade/pacific-x/parallel
qa/suites: some clean up for quincy
mon: updates for quincy
mon: update for quincy ondisk features
script: add pacific
doc/dev/release-checklist: we tagged v17.0.0
ceph-volume: change to quincy
include/ceph_features: retire MON_SINGLE_PAXOS
include/ceph_features: define FEATURE_SERVER_QUINCY
mon/MgrMonitor: add always_on_modules for quincy
add feature/release name quincy
kickoff v17 quincy
doc/dev/release-checklists: uncheck everything!
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:
* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased
In order to do a dynamic build, we'll:
* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
executable entry points
* by default, cmake generates import libs for executables. The issue
is that for rados.exe, it generates librados.dll.a, thus overriding
the librados.dll import library, which breaks the build process.
We'll configure cmake to skip generating import libs for executables.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
We're porting to Windows some of the unit tests, mostly related to
rados and rbd.
This change filters out the tests that aren't supported on Windows.
At the same time, we're defining some missing dependencies since
unlike ELFs SOs, all DLL symbols have to be resolved at compile time.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
* refs/pull/32776/head:
win32*.sh: fetch WNBD dependency
win32*.sh: bump win32 boost dependency
cmake: skip osd, Lua and neorados on Windows
win32*.sh: improve lib handling
win32*.sh: update openssl location
win32*.sh: cleanup dependency build dir when targetting Windows
win32*.sh: Strip binaries individually
win32*.sh: Cleanup zip archive creation
cmake,win32*.sh: Add OS=(ubuntu|suse) flag, for distro settings
win32*.sh: Use nproc for CPU count when targeting win32
test: Include compat.h to find uid_t/gid_t/gidlist when cross compiling
win32*.sh: improve win32 dependency check
win32*.sh: isolate network operations when building win32 dependencies
cmake: drop Python dependency when targeting Windows
win32*.sh: cleanup Windows build script
win32*.sh: fix patch command
win32*.sh: update Windows build scripts and readme
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
* This commit introduces Jaegertracing library as package libjaeger,
pickwhich would be consumed by other ceph pacakges such as ceph-common0
* adds the following dependencies, which would be build from source
using ExternalProjectHelper.cmake +IncludeJaeger.cmake +
Build<package>.cmake scripts:
jaegertracing: v0.6.0 [added as a submodule]
opentracing: v1.6.0 [added as a submodule]
thrift: 0.13.0 [added as a submodule]
yaml-cpp: 0.6.0
json(optional)
* updates Boost to be installed instead of being build only, because
jaegertracing them during their build process.
* ceph.spec.in: introduces a default enabled jaeger packaging option,
which could be disabled using --without-jaeger flag during rpmbuild
* note: libjaeger package if enabled will be a dependency on ceph-common, ceph-mon, rgw_common and transitively will be a dependency for modules that have them as a dependency.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
The Python bindings aren't supported on Windows for the time being,
for which reason we're dropping the Python dependency.
When cross-compiling, Cmake sometimes complains about not finding
the Python files instead of just using the host files.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
src,cmake: define BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT for Boost.Asio users and cleanups
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
This commit introduces internal (not yet part of the api) librbd functions for:
1. formating an RBD image in LUKS format
2. parsing an RBD image in LUKS format
The actual implementation of the LUKS format is done via libcryptsetup, which is added as a new dependency.
Signed-off-by: Or Ozeri <oro@il.ibm.com>
to search prefixes specified by the <PackageName>_ROOT CMake
variable and the <PackageName>_ROOT environment variable. find_package()
in CMake 3.12 started to use these prefixes if it is enabled.
Signed-off-by: Kefu Chai <kchai@redhat.com>
`acconfig.h` is generated using
configure_file(
${CMAKE_SOURCE_DIR}/src/include/config-h.in.cmake
${CMAKE_BINARY_DIR}/include/acconfig.h
)
in `config-h.in.cmake`, the cmake variable of `HAVE_LIBZBD` is checked.
so we need to ensure that this variable is visible from this
`configure_file()` statement.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Adds build option and implements init functionality for SSD cache
Signed-off-by: Lisa Li <xiaoyan.li@intel.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
errno.h values aren't the same across platforms, for which reason
we'll have to convert remote errors.
We'll use the cmake "add_definitions" function instead of
"add_compile_options", which generates incorrect compiler params
while "add_compile_definitions" isn't available at all when
using CMake 3.10, which is by default available on Ubuntu 18.04.
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
