Resurrected jerasure_generic, jerasure_sse3, jerasure_sse4, jerasure_neon,
shec_generic, shec_sse3, shec_sse4 and shec_neon. These all are exact
copies of the new jerasure and shec plugins that support SIMD detection.
Moved EC preload code in ceph-mon and ceph-osd to a central location, added
warning when preloading legacy plugins.
OSMonitor::get_erasure_code and OSDMonitor:normalize_profile will now check
if legacy EC plugins are used and log a warning.
Added tests to check that warnings make it to the log.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
By switching to a new gf-complete with SIMD runtime detection, we can now remove all the different flavors of jerasure and shec. This simplifies deployment and configuration of erasure coding, enables hetergenous OSDs, and enables us to take advantage of new performance improvements in jerasure without config/build changes.
This commit removes flavors from cmake, removes ErasureCodePluginSelect___, and fixes unit tests. There is now a single plugin for jerasure and a single plugin for shec.
SIMDExt.cmake was changed so that its a little more generic, and is not polluted with gf-complete specific CFLAG defines. The #define for SIMD instruction were based on gf-complete.
I also added a small init helper for jerasure that has code that was common between jerasure and shec.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
* renamed target to fio_ceph_objectstore
* moved into src/test/fio subdirectory
* added to cmake build
* added support for DDIR_READ
* added required fio option 'conf' to load a ceph configuration file
* added multiple collections per job to simulate parallelism from pgs
* creates objects of the appropriate size on setup
* added support for multiple jobs that share an ObjectStore instance
Signed-off-by: Casey Bodley <cbodley@redhat.com>
* in addition to the -march=foobar, we also need to pass -DFOOBAR to
gf-complete sources' cflags, so that the SIMD instructions can be
generated as expected.
* also extract the compiler support for instruction extensions detections
into SIMDExt.cmake.
* and only check the compiler support if CMAKE_SYSTEM_PROCESSOR matches
with the instruction-set/arch to be checked.
Signed-off-by: tone.zhang <tone.zhang@linaro.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>
find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Include Python 3 bindings into the cmake build and make packages for them
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
* remove Findtcmalloc.cmake, use Findgperftools.cmake instead.
* tcmalloc is packaged in gperftools-dev, so we can check them in a single
place.
* fix the check of heap-profiler.h, we should enable the check as long
as tcmalloc is enabled or profiler is enabled.
* only check gperftools headers in "include/gperftools", and do not
"include/google" anymore. as we only support the distros which
shipping recent gperftools-dev package with "include/gperftools".
and "google/*.h" are deprecated.
* set ALLOC_LIBS with GPERFTOOLS_TCMALLOC_LIBRARY, so we can link
against tcmalloc with full path.
Signed-off-by: Kefu Chai <kchai@redhat.com>
distutils' install command installs all stuff under $build-base, which
includes the temporary files. and rpm packaging does not like::
Installed (but unpackaged) file(s)
and to support both py2 and py3, we should encode the python version
into the path, otherwise the built files will be overwritten.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the empty .so file does not compile at all, so we need to bypass the
sanity check, if we are building docs. and what sphinx needs is just the
docstrings.
Fixes: http://tracker.ceph.com/issues/16940
Signed-off-by: Kefu Chai <kchai@redhat.com>
* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
arguments.
* also detect osxfuse on MacOS.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we used to check the header file of "aio.h" which is part of glibc,
the glibc async i/o, and the corresponding library is librt. but what
we are using is libaio.
* reference libaio using AIO_LIBRARIES for consistency.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* we don't need the BABELTRACE_PATH_HINT setting, as packaging script
requires the the babeltrace pacakges explicitly.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
find_package() modules.
Signed-off-by: Kefu Chai <kchai@redhat.com>
libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.
Signed-off-by: Kefu Chai <kchai@redhat.com>
FindPackageHandleStandardArgs() takes care of the find_package()
boilderplate stuff. so no need to repeat them. and remove the checkings
in env variables.
Signed-off-by: Kefu Chai <kchai@redhat.com>
please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.
* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
"ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"
Signed-off-by: Kefu Chai <kchai@redhat.com>
it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
--cython-include-dirs to "build_ext" command
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like
+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Distutils.cmake:
set --prefix=${CMAKE_INSTALL_PREFIX} for python packages installed using
setuptools. it was set to --prefix=/user only when $DESTDIR is set. so
if user installs ceph using -DCMAKE_INSTALL_PREFIX, these python
packages still go to /usr, which is unexpected.
* ceph-disk/CMakeLists.txt:
install script into ${CMAKE_INSTALL_SBINDIR} instead of /usr/sbin
Signed-off-by: Kefu Chai <kchai@redhat.com>
Git now can have a history for a short branch in the logs, breaking the
rev parsing. Update to a newer version that fixes this.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
* add keyword "INSTALL_SCRIPT" to distutils_install_module(), so we can
override the install path of ceph-disk script.
* refactor the Distutils.cmake module a little bit, the ${option}
variable out lives the CODE snipplet. so we need to reset it
at the beginning.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
with existing targets like "rbd" or "rados". so we can not reuse the
name in ${name}.pyx. and instead, we should specify the target name
explicitly.
* add distutils_install_cython_module() function to build and install
cython modules.
* we can split build and install of cython module, but the install phase
always tries to build the module. so keep it this way. will look at it
later on.
* move the variables initializations into the Distutils.cmake module.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Findlttng-ust.cmake: detect lttng-gen-tp, and make it a required var
* Findlttng-ust.cmake: fix the detection of lttng
* src/tracing/CMakeLists.txt:
- do not put the generated header files into ${CMAKE_SOURCE_DIR}/src/tracing,
instead we should put the generated files into ${PROJECT_BINARY_DIR}/include.
- do not compile the tracing library using the .c files in the repo,
instead, we should generate them at compile time using lttng-gen-tp,
and compile the genererated .c files.
* CMakeLists.txt: make the lttng-use package REQUIRED if WITH_LTTNG=ON
* src/CMakeLists.txt: let librados depend on librados-tp if LTTNG is
enabled: librados includes tracing/librados.h. the same applies to
librbd, libos and libosd. cmake could deduce the dependencies if we
add the lttng header files under include/tracing into the
corresponding target, as the lttng header files are already listed
as OUTPUT and GENERATED files, but this requires us to put all the
involved targets in the same CMakeLists.txt. in this case, three
CMakeLists.txts are impacted. so we create a new target for each
lttng header, and let the impacted target depend on it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
