Scope SIMD CPU flag detection base on target architecture,
resolving build failures in gf-complete on i386; this is
inline with the previous autotools based feature detection.
Only compile crypto/isa-l if yasm 64 bit complier is found,
effectively limiting scope to x86_64.
Signed-off-by: James Page <james.page@ubuntu.com>
The cmake package names of libraries are case-sensitive. The name used in
FIND_PACKAGE_HANDLE_STANDARD_ARGS - e.g. "cryptopp" - must match the name used
in the module name - "Findcryptopp.cmake" - as well as the name used in
find_package() call - "find_package(cryptopp REQUIRED)"
Note that the term "package" here refers to a cmake abstraction, *not* an
actual RPM or Debian package.
Fixes: http://tracker.ceph.com/issues/19853
Signed-off-by: Nathan Cutler <ncutler@suse.com>
so instead of excluding all submodules, one can just exclude boost and
rocksdb using:
cmake -DCTAG_EXCLUDES="boost;rocksdb" ..
Signed-off-by: Kefu Chai <kchai@redhat.com>
Test not only for -march support, but also the actual
presence of the intrinsic routines. Not sure why, but gcc
4.8.5 passes the first but not the second.
Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>
ARMv8 defines PMULL crypto instruction.
This patch optimizes crc32c calculate with the instruction when
available rather than original linear crc instructions.
ceph crc32c performance unit test shows that the optimization get
~ x3.90 speedups on ThunderX ARM Core@2.0GHz (Cavium)
~ x1.45 speedups on ARM Cortex-A57@2.1GHz (Huaiwei)
~ x1.16 speedups on ARM Cortex-A57@2.0GHz (Softiron)
Jira: ENTLLT-358
Change-Id: I657422cd20c9ca78237cd060210a5383f4122575
Signed-off-by: wei xiao <wei.xiao@linaro.org>
This makes it more portable
Remove /usr/include as it is searched by find_path anyway.
Remove unnecessary include that weren't used.
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
simplified the package to find jemalloc. this one matches
the one in src/rocksdb/cmake/modules. The existing one
was struggling to find jemalloc on some hosts.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
libcephd is a library that contains ceph daemon code
that can be statically linked in other applications.
Added MergeStaticLibraries.cmake that can merge static libraries
to form a bigger one. This approach avoids the need to mess with
STATIC libraries all over the code base.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
The used compiler may or may not be recent enough to recognize the
crc32 extended cpu type. However, it does not really have to know about
them either, since all we do is pass inline assembly instructions to
the assembler.
This patch moves the crc cpu extension detection from compiler based
to assembler based, so that we can build optimized code even when the
compiler does not know about the cpu type yet.
Fixes: http://tracker.ceph.com/issues/17516
Signed-off-by: Alexander Graf <agraf@suse.de>
Resurrected jerasure_generic, jerasure_sse3, jerasure_sse4, jerasure_neon,
shec_generic, shec_sse3, shec_sse4 and shec_neon. These all are exact
copies of the new jerasure and shec plugins that support SIMD detection.
Moved EC preload code in ceph-mon and ceph-osd to a central location, added
warning when preloading legacy plugins.
OSMonitor::get_erasure_code and OSDMonitor:normalize_profile will now check
if legacy EC plugins are used and log a warning.
Added tests to check that warnings make it to the log.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
By switching to a new gf-complete with SIMD runtime detection, we can now remove all the different flavors of jerasure and shec. This simplifies deployment and configuration of erasure coding, enables hetergenous OSDs, and enables us to take advantage of new performance improvements in jerasure without config/build changes.
This commit removes flavors from cmake, removes ErasureCodePluginSelect___, and fixes unit tests. There is now a single plugin for jerasure and a single plugin for shec.
SIMDExt.cmake was changed so that its a little more generic, and is not polluted with gf-complete specific CFLAG defines. The #define for SIMD instruction were based on gf-complete.
I also added a small init helper for jerasure that has code that was common between jerasure and shec.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
* renamed target to fio_ceph_objectstore
* moved into src/test/fio subdirectory
* added to cmake build
* added support for DDIR_READ
* added required fio option 'conf' to load a ceph configuration file
* added multiple collections per job to simulate parallelism from pgs
* creates objects of the appropriate size on setup
* added support for multiple jobs that share an ObjectStore instance
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* in addition to the -march=foobar, we also need to pass -DFOOBAR to
gf-complete sources' cflags, so that the SIMD instructions can be
generated as expected.
* also extract the compiler support for instruction extensions detections
into SIMDExt.cmake.
* and only check the compiler support if CMAKE_SYSTEM_PROCESSOR matches
with the instruction-set/arch to be checked.
Signed-off-by: tone.zhang <tone.zhang@linaro.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>
find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Include Python 3 bindings into the cmake build and make packages for them
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
* remove Findtcmalloc.cmake, use Findgperftools.cmake instead.
* tcmalloc is packaged in gperftools-dev, so we can check them in a single
place.
* fix the check of heap-profiler.h, we should enable the check as long
as tcmalloc is enabled or profiler is enabled.
* only check gperftools headers in "include/gperftools", and do not
"include/google" anymore. as we only support the distros which
shipping recent gperftools-dev package with "include/gperftools".
and "google/*.h" are deprecated.
* set ALLOC_LIBS with GPERFTOOLS_TCMALLOC_LIBRARY, so we can link
against tcmalloc with full path.
Signed-off-by: Kefu Chai <kchai@redhat.com>
distutils' install command installs all stuff under $build-base, which
includes the temporary files. and rpm packaging does not like::
Installed (but unpackaged) file(s)
and to support both py2 and py3, we should encode the python version
into the path, otherwise the built files will be overwritten.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the empty .so file does not compile at all, so we need to bypass the
sanity check, if we are building docs. and what sphinx needs is just the
docstrings.
Fixes: http://tracker.ceph.com/issues/16940
Signed-off-by: Kefu Chai <kchai@redhat.com>
* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
arguments.
* also detect osxfuse on MacOS.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we used to check the header file of "aio.h" which is part of glibc,
the glibc async i/o, and the corresponding library is librt. but what
we are using is libaio.
* reference libaio using AIO_LIBRARIES for consistency.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we don't need the BABELTRACE_PATH_HINT setting, as packaging script
requires the the babeltrace pacakges explicitly.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
find_package() modules.
Signed-off-by: Kefu Chai <kchai@redhat.com>
libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.
Signed-off-by: Kefu Chai <kchai@redhat.com>
FindPackageHandleStandardArgs() takes care of the find_package()
boilderplate stuff. so no need to repeat them. and remove the checkings
in env variables.
Signed-off-by: Kefu Chai <kchai@redhat.com>
please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.
* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
"ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"
Signed-off-by: Kefu Chai <kchai@redhat.com>
it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
--cython-include-dirs to "build_ext" command
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like
+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Distutils.cmake:
set --prefix=${CMAKE_INSTALL_PREFIX} for python packages installed using
setuptools. it was set to --prefix=/user only when $DESTDIR is set. so
if user installs ceph using -DCMAKE_INSTALL_PREFIX, these python
packages still go to /usr, which is unexpected.
* ceph-disk/CMakeLists.txt:
install script into ${CMAKE_INSTALL_SBINDIR} instead of /usr/sbin
Signed-off-by: Kefu Chai <kchai@redhat.com>
Git now can have a history for a short branch in the logs, breaking the
rev parsing. Update to a newer version that fixes this.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
* add keyword "INSTALL_SCRIPT" to distutils_install_module(), so we can
override the install path of ceph-disk script.
* refactor the Distutils.cmake module a little bit, the ${option}
variable out lives the CODE snipplet. so we need to reset it
at the beginning.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
with existing targets like "rbd" or "rados". so we can not reuse the
name in ${name}.pyx. and instead, we should specify the target name
explicitly.
* add distutils_install_cython_module() function to build and install
cython modules.
* we can split build and install of cython module, but the install phase
always tries to build the module. so keep it this way. will look at it
later on.
* move the variables initializations into the Distutils.cmake module.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Findlttng-ust.cmake: detect lttng-gen-tp, and make it a required var
* Findlttng-ust.cmake: fix the detection of lttng
* src/tracing/CMakeLists.txt:
- do not put the generated header files into ${CMAKE_SOURCE_DIR}/src/tracing,
instead we should put the generated files into ${PROJECT_BINARY_DIR}/include.
- do not compile the tracing library using the .c files in the repo,
instead, we should generate them at compile time using lttng-gen-tp,
and compile the genererated .c files.
* CMakeLists.txt: make the lttng-use package REQUIRED if WITH_LTTNG=ON
* src/CMakeLists.txt: let librados depend on librados-tp if LTTNG is
enabled: librados includes tracing/librados.h. the same applies to
librbd, libos and libosd. cmake could deduce the dependencies if we
add the lttng header files under include/tracing into the
corresponding target, as the lttng header files are already listed
as OUTPUT and GENERATED files, but this requires us to put all the
involved targets in the same CMakeLists.txt. in this case, three
CMakeLists.txts are impacted. so we create a new target for each
lttng header, and let the impacted target depend on it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Replaced relative paths in shell scripts in
test/ceph_objectstore_tool.py and init-ceph.in to
work with CEPH_FOO environment variables set in cmake.
Also added CEPH_BUILD_DIR environment variable set
to CMAKE_BINARY_DIR. It is used in init-ceph and
ceph_objectstore_tool.py.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
make check targets now use add_ceph_test, or
add_ceph_unittest to be added to CTest. Renamed and
reordered targets in tests/CMakeLists.txt
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Permit building without LDAP support--support is enabled by default.
Tested with CMake and autotools standalone builds.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Implement a new external authenticator based on LDAP and the
new external token format.
External LDAP auth now works, at least with openldap/X.500 style
naming and ldaps:// (SSL).
The latter is AD-friendly, but since AD uses dnattr=cn (IIRC)
everywhere, AD will need testing.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
* some distros install libsnappy into /usr/lib/x86_64-linux-gnu
which is completely resonable. but this directory is not listed
in the Findsnappy.cmake.
* also add SNAPPY_ROOT_DIR, in case user install snappy in some
other directory.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Option WITH_XIO now OFF but required by default,
Findxio.cmake now using
FIND_PACKAGE_HANDLE_STANDARD_ARGS.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Set LTTNG option to ON by default, added
Findlttng.cmake, added CMakeLists.txt in
src/tracing that generates .h files using
lttng-gen-tp, added WITH_LTTNG to
config-h.in.cmake.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Added a FindJeMalloc.cmake file, changed
allocator checking logic (tcmalloc is default),
replaced TCMALLOC_LIBS to ALLOC_LIBS, added
HAVE_LIB(ALLOCATOR) to config-h.in.cmake.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
client/fuse_ll.cc is now including <fuse.h> and <fuse_lowlevel.h>
instead of <fuse/fuse.h> and <fuse/fuse_lowlevel.h>, so we need to add
the fuse directory to the FUSE_INCLUDE_DIRS variable
using find_path() with just fuse.h was finding a /usr/include/fuse.h
instead of the one in /usr/include/fuse/. looking for fuse_common.h and
fuse_lowlevel.h first causes it to generate the correct
FUSE_INCLUDE_DIRS=/usr/include/fuse
Fixes: #12909
Signed-off-by: Casey Bodley <cbodley@redhat.com>
The nss lib needs the nspr library to build. Add file to
get the nspr lib/includes from the pc file.
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
CMake Ceph Build System (Firefly)
CMake. Add tests.
Respace src/CMakeLists.txt.
CMake. Spacing cleanups.
CMake for Firefly is Triumphant
CMake for Giant
Adapt to Giant.
Fix installation for scripts and man pages
Fix CEPH_LIBDIR and CEPH_PKGLIBDIR defines
Add erasure-code libraries
uses try_compile() to detect support for -msse flags
Fix rados object classes
Propagate Casey's cls library change to src/test.
Fix CMake build for Hammer.
Try-add rados and common to librbd link.
Fix name and linkage of libec_lrc.
Rename arch/neon.c arm.c
Fix libcommon.a dependencies (some unit tests).
Authors:
Ali Maredia <ali@cohortfs.com>
Casey Bodley <casey@cohortfs.com>
Adam Emerson <aemerson@cohortfs.com>
Marcus Watts <mdw@cohortfs.com>
Matt Benjamin <matt@cohortfs.com>
Signed-off-by: Matt Benjamin <matt@cohortfs.com>
