the commit d406f228 in gperf implements a c11 feature used by a
recent change in rocksdb: 16e03882, which uses aligned_alloc().
and 16e03882 in rocksdb was merged after v5.7 was tagged, while
16e03882 in gperf was merged after v2.6.1 was tagged.
because aligned_alloc() is not implemented by tcmalloc until the
not-yet-released 2.6.2, if we call aligned_alloc() in an application
linked against tcmalloc, what gets called will be the glibc's
aligned_alloc(). but if we free() the memory chunk allocated by
aligned_alloc(), the tcmalloc's implementation kicks in, then
InvalidFree() is called, because the memory chunk being freed was
allocated by tcmalloc. in short, "mixing allocators", quote from
Dan Mick.
in rocksdb, aligned_alloc() is used if _ISOC11_SOURCE is defined, this
makes sense, because aligned_alloc() is a C11 function. we could avoid
using it by not defining _ISOC11_SOURCE. but as long as _GNU_SOURCE is
defined, glibc defines _ISOC11_SOURCE. and libstdc++ requires
_GNU_SOURCE, because it uses a fair amount of GNU extensions.
Fixes: http://tracker.ceph.com/issues/21422
Signed-off-by: Kefu Chai <kchai@redhat.com>
Skip nonexistent 'expected' Python packaging files, and add bin/ as
a possible file for symlinking during build (for Python projects with
slightly different forms, currently ceph-volume)
Signed-off-by: Dan Mick <dan.mick@redhat.com>
The ppc64 support requires a couple of changes:
- adding the ppc64 support to cmake
- changing optimized crc32 code to compile on ppc64le only
- moving ifdef condition before crc32_align to avoid defined but not
used warning
Signed-off-by: Boris Ranto <branto@redhat.com>
* do not REQUIRE libzfs if it is enabled, this follows the way how we
handle `WITH_XFS` option.
* also refactor the cmake script related to libxfs backend support a
little bit.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Scope SIMD CPU flag detection base on target architecture,
resolving build failures in gf-complete on i386; this is
inline with the previous autotools based feature detection.
Only compile crypto/isa-l if yasm 64 bit complier is found,
effectively limiting scope to x86_64.
Signed-off-by: James Page <james.page@ubuntu.com>
The cmake package names of libraries are case-sensitive. The name used in
FIND_PACKAGE_HANDLE_STANDARD_ARGS - e.g. "cryptopp" - must match the name used
in the module name - "Findcryptopp.cmake" - as well as the name used in
find_package() call - "find_package(cryptopp REQUIRED)"
Note that the term "package" here refers to a cmake abstraction, *not* an
actual RPM or Debian package.
Fixes: http://tracker.ceph.com/issues/19853
Signed-off-by: Nathan Cutler <ncutler@suse.com>
so instead of excluding all submodules, one can just exclude boost and
rocksdb using:
cmake -DCTAG_EXCLUDES="boost;rocksdb" ..
Signed-off-by: Kefu Chai <kchai@redhat.com>
Test not only for -march support, but also the actual
presence of the intrinsic routines. Not sure why, but gcc
4.8.5 passes the first but not the second.
Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>
ARMv8 defines PMULL crypto instruction.
This patch optimizes crc32c calculate with the instruction when
available rather than original linear crc instructions.
ceph crc32c performance unit test shows that the optimization get
~ x3.90 speedups on ThunderX ARM Core@2.0GHz (Cavium)
~ x1.45 speedups on ARM Cortex-A57@2.1GHz (Huaiwei)
~ x1.16 speedups on ARM Cortex-A57@2.0GHz (Softiron)
Jira: ENTLLT-358
Change-Id: I657422cd20c9ca78237cd060210a5383f4122575
Signed-off-by: wei xiao <wei.xiao@linaro.org>
This makes it more portable
Remove /usr/include as it is searched by find_path anyway.
Remove unnecessary include that weren't used.
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
simplified the package to find jemalloc. this one matches
the one in src/rocksdb/cmake/modules. The existing one
was struggling to find jemalloc on some hosts.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
libcephd is a library that contains ceph daemon code
that can be statically linked in other applications.
Added MergeStaticLibraries.cmake that can merge static libraries
to form a bigger one. This approach avoids the need to mess with
STATIC libraries all over the code base.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
The used compiler may or may not be recent enough to recognize the
crc32 extended cpu type. However, it does not really have to know about
them either, since all we do is pass inline assembly instructions to
the assembler.
This patch moves the crc cpu extension detection from compiler based
to assembler based, so that we can build optimized code even when the
compiler does not know about the cpu type yet.
Fixes: http://tracker.ceph.com/issues/17516
Signed-off-by: Alexander Graf <agraf@suse.de>
Resurrected jerasure_generic, jerasure_sse3, jerasure_sse4, jerasure_neon,
shec_generic, shec_sse3, shec_sse4 and shec_neon. These all are exact
copies of the new jerasure and shec plugins that support SIMD detection.
Moved EC preload code in ceph-mon and ceph-osd to a central location, added
warning when preloading legacy plugins.
OSMonitor::get_erasure_code and OSDMonitor:normalize_profile will now check
if legacy EC plugins are used and log a warning.
Added tests to check that warnings make it to the log.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
By switching to a new gf-complete with SIMD runtime detection, we can now remove all the different flavors of jerasure and shec. This simplifies deployment and configuration of erasure coding, enables hetergenous OSDs, and enables us to take advantage of new performance improvements in jerasure without config/build changes.
This commit removes flavors from cmake, removes ErasureCodePluginSelect___, and fixes unit tests. There is now a single plugin for jerasure and a single plugin for shec.
SIMDExt.cmake was changed so that its a little more generic, and is not polluted with gf-complete specific CFLAG defines. The #define for SIMD instruction were based on gf-complete.
I also added a small init helper for jerasure that has code that was common between jerasure and shec.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
* renamed target to fio_ceph_objectstore
* moved into src/test/fio subdirectory
* added to cmake build
* added support for DDIR_READ
* added required fio option 'conf' to load a ceph configuration file
* added multiple collections per job to simulate parallelism from pgs
* creates objects of the appropriate size on setup
* added support for multiple jobs that share an ObjectStore instance
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* in addition to the -march=foobar, we also need to pass -DFOOBAR to
gf-complete sources' cflags, so that the SIMD instructions can be
generated as expected.
* also extract the compiler support for instruction extensions detections
into SIMDExt.cmake.
* and only check the compiler support if CMAKE_SYSTEM_PROCESSOR matches
with the instruction-set/arch to be checked.
Signed-off-by: tone.zhang <tone.zhang@linaro.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>
find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Include Python 3 bindings into the cmake build and make packages for them
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
* remove Findtcmalloc.cmake, use Findgperftools.cmake instead.
* tcmalloc is packaged in gperftools-dev, so we can check them in a single
place.
* fix the check of heap-profiler.h, we should enable the check as long
as tcmalloc is enabled or profiler is enabled.
* only check gperftools headers in "include/gperftools", and do not
"include/google" anymore. as we only support the distros which
shipping recent gperftools-dev package with "include/gperftools".
and "google/*.h" are deprecated.
* set ALLOC_LIBS with GPERFTOOLS_TCMALLOC_LIBRARY, so we can link
against tcmalloc with full path.
Signed-off-by: Kefu Chai <kchai@redhat.com>
distutils' install command installs all stuff under $build-base, which
includes the temporary files. and rpm packaging does not like::
Installed (but unpackaged) file(s)
and to support both py2 and py3, we should encode the python version
into the path, otherwise the built files will be overwritten.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the empty .so file does not compile at all, so we need to bypass the
sanity check, if we are building docs. and what sphinx needs is just the
docstrings.
Fixes: http://tracker.ceph.com/issues/16940
Signed-off-by: Kefu Chai <kchai@redhat.com>
* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
arguments.
* also detect osxfuse on MacOS.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we used to check the header file of "aio.h" which is part of glibc,
the glibc async i/o, and the corresponding library is librt. but what
we are using is libaio.
* reference libaio using AIO_LIBRARIES for consistency.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we don't need the BABELTRACE_PATH_HINT setting, as packaging script
requires the the babeltrace pacakges explicitly.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
find_package() modules.
Signed-off-by: Kefu Chai <kchai@redhat.com>
libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.
Signed-off-by: Kefu Chai <kchai@redhat.com>
FindPackageHandleStandardArgs() takes care of the find_package()
boilderplate stuff. so no need to repeat them. and remove the checkings
in env variables.
Signed-off-by: Kefu Chai <kchai@redhat.com>
please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.
* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
"ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"
Signed-off-by: Kefu Chai <kchai@redhat.com>
it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
--cython-include-dirs to "build_ext" command
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like
+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Distutils.cmake:
set --prefix=${CMAKE_INSTALL_PREFIX} for python packages installed using
setuptools. it was set to --prefix=/user only when $DESTDIR is set. so
if user installs ceph using -DCMAKE_INSTALL_PREFIX, these python
packages still go to /usr, which is unexpected.
* ceph-disk/CMakeLists.txt:
install script into ${CMAKE_INSTALL_SBINDIR} instead of /usr/sbin
Signed-off-by: Kefu Chai <kchai@redhat.com>
Git now can have a history for a short branch in the logs, breaking the
rev parsing. Update to a newer version that fixes this.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
* add keyword "INSTALL_SCRIPT" to distutils_install_module(), so we can
override the install path of ceph-disk script.
* refactor the Distutils.cmake module a little bit, the ${option}
variable out lives the CODE snipplet. so we need to reset it
at the beginning.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
with existing targets like "rbd" or "rados". so we can not reuse the
name in ${name}.pyx. and instead, we should specify the target name
explicitly.
* add distutils_install_cython_module() function to build and install
cython modules.
* we can split build and install of cython module, but the install phase
always tries to build the module. so keep it this way. will look at it
later on.
* move the variables initializations into the Distutils.cmake module.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Findlttng-ust.cmake: detect lttng-gen-tp, and make it a required var
* Findlttng-ust.cmake: fix the detection of lttng
* src/tracing/CMakeLists.txt:
- do not put the generated header files into ${CMAKE_SOURCE_DIR}/src/tracing,
instead we should put the generated files into ${PROJECT_BINARY_DIR}/include.
- do not compile the tracing library using the .c files in the repo,
instead, we should generate them at compile time using lttng-gen-tp,
and compile the genererated .c files.
* CMakeLists.txt: make the lttng-use package REQUIRED if WITH_LTTNG=ON
* src/CMakeLists.txt: let librados depend on librados-tp if LTTNG is
enabled: librados includes tracing/librados.h. the same applies to
librbd, libos and libosd. cmake could deduce the dependencies if we
add the lttng header files under include/tracing into the
corresponding target, as the lttng header files are already listed
as OUTPUT and GENERATED files, but this requires us to put all the
involved targets in the same CMakeLists.txt. in this case, three
CMakeLists.txts are impacted. so we create a new target for each
lttng header, and let the impacted target depend on it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
