Build Boost using the src/boost submodule, unless overridden
by -DWITH_SYSTEM_BOOST.
If -DBOOST_J=<n> is provided, builds Boost with <n> jobs.
Boost builds in the configured Build directory.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
msg/async: ibverbs/rdma support
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mellanox <adirl@mellanox.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Fixes here:
Pull civetweb from new branch/commit.
Build civetweb.c w/ USE_IPV6 (w/o depending on extra ceph-only build
logic inside of civetweb.)
Make several things "const" to please the compiler.
Remap port string at '+' -> ,; to avoid , overload.
Use new civetweb options validate_http_method=no canonicalize_url_path=no
Signed-off-by: Marcus Watts <mwatts@redhat.com>
This is because of versioning issues between Boost and Beast
on CentOS 7. It is intended that this patch will be reverted
after merging the in-tree Boost facility.
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
The fcgi and expat libraries are only used by rgw, so only make them
hard requirements if WITH_RADOSGW is set.
Signed-off-by: David Disseldorp <ddiss@suse.de>
* renamed target to fio_ceph_objectstore
* moved into src/test/fio subdirectory
* added to cmake build
* added support for DDIR_READ
* added required fio option 'conf' to load a ceph configuration file
* added multiple collections per job to simulate parallelism from pgs
* creates objects of the appropriate size on setup
* added support for multiple jobs that share an ObjectStore instance
Signed-off-by: Casey Bodley <cbodley@redhat.com>
find_package(lttng-ust REQUIRED) fails to find the lttng library without
this change. because find_path(LTTNG_LIBRARY_DIR ...) does not search
in the default library paths. and the second mode of
FIND_PACKAGE_HANDLE_STANDARD_ARGS() does not stop the cmake with a fatal
message, even some of the required vars are missing. so use the
implemetantion from cmake upstream.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Include Python 3 bindings into the cmake build and make packages for them
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
* remove Findtcmalloc.cmake, use Findgperftools.cmake instead.
* tcmalloc is packaged in gperftools-dev, so we can check them in a single
place.
* fix the check of heap-profiler.h, we should enable the check as long
as tcmalloc is enabled or profiler is enabled.
* only check gperftools headers in "include/gperftools", and do not
"include/google" anymore. as we only support the distros which
shipping recent gperftools-dev package with "include/gperftools".
and "google/*.h" are deprecated.
* set ALLOC_LIBS with GPERFTOOLS_TCMALLOC_LIBRARY, so we can link
against tcmalloc with full path.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* AIO_LIBS is now named AIO_LIBARIES, and there is not point to print
out its path
* USE_NSS is not defined if NSS is not checked, so if(USE_NSS) is better
here.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this behaviour matches the autotools. and we should not enable profiling
unless asked to do so.
Fixes: http://tracker.ceph.com/issues/16804
Signed-off-by: Kefu Chai <kchai@redhat.com>
for example
```
include_directories(${LEVELDB_PREFIX}/include)
```
does not make sense if `LEVELDB_PREFIX` is not defined.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
find_package() modules.
Signed-off-by: Kefu Chai <kchai@redhat.com>
libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.
Signed-off-by: Kefu Chai <kchai@redhat.com>
please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.
* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
"ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"
Signed-off-by: Kefu Chai <kchai@redhat.com>
The cmake build should select NSS or CryptoPP consistently with
current autoconf logic, as used in our RHEL builds (-DWITH_NSS),
and the SSL linkage must match (for NSS, -lssl3).
With this change, there is no direct linkage with libssl when NSS
is selected (I still see indirect linkage via libcurl, in turn
via libssh2, looking into that).
With guidance from upstream and downstream, NSS builds are now
selected by default.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
so it would be easier to figure out what allocator we are using later,
instead of poking around ${Foo}_FOUND variables.
Signed-off-by: Kefu Chai <kchai@redhat.com>
we decided to drop the static libraries from the *-dev debian packages.
so no more tricky changes for supporting creating both libraries.
* partially revert 1dbfb26: s/BUILD_SHARED_LIBS/ENABLE_SHARED/
- add_library() command use BUILD_SHARED_LIBS as the default library
type, and all intermediate libraries should static ones. it would
cause unnecessary confusion if some developer uses add_library() without
specifying the library type, and ends up with a .so. so we use our
own setting variables for specifying the type of user facing libraries.
* revert ac47440
- the BUILD_SHARED_LIBS setting is also populated to the gmock libraries,
which should be compiled as static ones. otherwise the unit tests will
crash on exit.
Signed-off-by: Kefu Chai <kchai@redhat.com>
so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* BUILD_SHARED_LIB is a cmake variable respected by add_library().
so we can simply expose it as an option instead of reinventing our own.
* and mark internal libraries as STATIC explicitly
* and add keyutil to librbd as secret.cc is using it. and secret.cc is
linked into librbd.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Findlttng-ust.cmake: detect lttng-gen-tp, and make it a required var
* Findlttng-ust.cmake: fix the detection of lttng
* src/tracing/CMakeLists.txt:
- do not put the generated header files into ${CMAKE_SOURCE_DIR}/src/tracing,
instead we should put the generated files into ${PROJECT_BINARY_DIR}/include.
- do not compile the tracing library using the .c files in the repo,
instead, we should generate them at compile time using lttng-gen-tp,
and compile the genererated .c files.
* CMakeLists.txt: make the lttng-use package REQUIRED if WITH_LTTNG=ON
* src/CMakeLists.txt: let librados depend on librados-tp if LTTNG is
enabled: librados includes tracing/librados.h. the same applies to
librbd, libos and libosd. cmake could deduce the dependencies if we
add the lttng header files under include/tracing into the
corresponding target, as the lttng header files are already listed
as OUTPUT and GENERATED files, but this requires us to put all the
involved targets in the same CMakeLists.txt. in this case, three
CMakeLists.txts are impacted. so we create a new target for each
lttng header, and let the impacted target depend on it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Moved all the libraries in CMAKE_BINARY_DIR/lib
and all the binaries in CMAKE_BINARY_DIR/bin. Set
various environment variables for test-ceph-helpers.
Put those variables throughout
qa/workunits/ceph-helpers.sh.
NOTE: This is a very rough draft of these fixes.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
* pass the environment variables using `env` to pass environment
varibles with space(s) in them to the COMMAND in add_custom_target.
otherwise, cmake will try to quote the space with "\". this breaks the
generated command line.
* add a comment for ccache to note that we do not expect ccache to speed
up linking. we use it as the linker's launcher to workaround
https://bugs.python.org/issue8027. to be specific,
distutils.UnixCCompiler.link overwrites the first linker CLI's arg
using the the first C++ compiler's CLI arg, if "env" is not used to
launch the linker. this breaks the cythonization of our pybind APIs.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Use cmake's FindOpenSSL. Actually use the library definitions that
come out of it. Also use the include path. If SSL is disabled,
then build without any of this. The result will be a version
of civetweb that includes openssl support using dlopen() - ie,
what we used to do. This is probably a bit silly, and I hope
nobody actually expects to use it that way, but it's mostly harmless.
Signed-off-by: Marcus Watts <mwatts@redhat.com>
Use find_package to enforce the dependency up front (we should be
using this pattern throughout top-level CMakeLists.txt).
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Implement a new external authenticator based on LDAP and the
new external token format.
External LDAP auth now works, at least with openldap/X.500 style
naming and ldaps:// (SSL).
The latter is AD-friendly, but since AD uses dnattr=cn (IIRC)
everywhere, AD will need testing.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
This is temporary, until a way to stop repetitive
unecessary generation of headerfiles is figured out.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
* use ${FUSE_LIBRARIES} instead of "fuse" when referencing libfuse.
* do not compile FuseStore.cc if !HAVE_LIBFUSE.
* CMakeLists.txt: s/HAVE_FUSE/HAVE_LIBFUSE/, to match with the one
defined for config.h.in.cmake, otherwise we need two vars:
HAVE_LIBFUSE, and HAVE_FUSE
Signed-off-by: Kefu Chai <kchai@redhat.com>
* this change is the cmake port of 911e7a0.
the rocksdb's Makefile will detect the installed libbz2 and libz4 by
its own. if the building env happens to have these libraries installed,
a link time dependency is introduced. so we are forced to link against
them.
* do not REQUIRE BZip2 in cmake anymore
* only link against bzip2 if it is detected
Signed-off-by: Kefu Chai <kchai@redhat.com>
Option WITH_XIO now OFF but required by default,
Findxio.cmake now using
FIND_PACKAGE_HANDLE_STANDARD_ARGS.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Set LTTNG option to ON by default, added
Findlttng.cmake, added CMakeLists.txt in
src/tracing that generates .h files using
lttng-gen-tp, added WITH_LTTNG to
config-h.in.cmake.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Added a FindJeMalloc.cmake file, changed
allocator checking logic (tcmalloc is default),
replaced TCMALLOC_LIBS to ALLOC_LIBS, added
HAVE_LIB(ALLOCATOR) to config-h.in.cmake.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Added CMakeLists.txt in src/kv, replaced
os_mon_objs with kv_objs, linked libkv to os,
added HAVE_LIBROCKSDB to config-h.in.cmake,
tweaked rocksdb unittest, added bluestore
unittests, made rocksdb build with
add_custom_target instead of custom_command.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
musl libc does not provide sys_siglist. It does provide a reentrant implementation of strsignal().
Added a cmake build option for using reentrant strsignal() impls instead of sys_siglist.
Signed-off-by: John Coyle <dx9err@gmail.com>
This is what autotools does with AC_FUNC_STRERROR_R bit. It takes us
from
$ monmaptool --print nonexistant
monmaptool: monmap file nonexistant
monmaptool: couldn't open nonexistant: (2)
to
$ monmaptool --print nonexistant
monmaptool: monmap file nonexistant
monmaptool: couldn't open nonexistant: (2) No such file or directory
on cmake - otherwise we end up using GNU-specific version as if it was
the POSIX version.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
res_query in musl libc is thread safe. Added a WITH_THREAD_SAFE_RES_QUERY cmake build option to disable default res_query lock.
Signed-off-by: John Coyle <dx9err@gmail.com>
man pages have to be preprocessed now, and can't be installed directly.
skip installing them until we add the cmake-fu to copy what man/Makefile.am
is doing
Signed-off-by: Casey Bodley <casey@cohortfs.com>
Libnss needs also to check for libnspr and add the related
-l targets to CRYPTO_LIBS, otherwise the build will fail.
Make sure USE_NSS is set correctly to 0 if cryptopp found
and in case nss was found set USE_CRYPTOPP also to 0.
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
XioMessenger implements a Ceph Messenger provider for Accelio,
a high-performance messaging transport by Mellanox. Current
Accelio is layered on ibverbs, and supports Infiniband, ROCE,
and other RDMA transports. Future Accelio verions will support
alternative transports (including TCP), and flexible transport
selection.
config: cluster_rdma drives messenger creation
ceph_mds ceph_mon and ceph_osd use XioMessengers for cluster
communication when cluster_rdma is set
Move XioMessenger to msg/xio.
This matches the other new Messenger locations.
test: tests for tcp and xio messengers
(Not tests only.)
buffer: add subclass for xio buffers
xio: convert to Connection::send_message interface
config: -x, --xio as aliases for client_rdma
ceph-fuse: create xio messenger if client_rdma
Find XioMessenger.h and QueueStrategy.h in msg/xio.
ceph-syn: create xio messenger if client_rdma
librados: create xio messenger if client_rdma
Find XioMessenger.h and QueueStrategy.h in msg/xio.
Restore non-abort from Xio Mon integration.
Fix xio_client send count, again.
xio: must signal cond under mutex lock
xio: dispatch strategies support ms_fast_dispatch
xio: config variable xio_port_shift
remove set_port_shift() from XioMessenger, and just use the value
from the configuration
xio: don't depend on g_ceph_context for dout
XioMessenger now uses its own cct for all logging operations
the accelio log function, however, still depends on a global
CephContext. so we maintain an extra one, separate from g_ceph_context,
in XioMessenger.cc that is initialized on first construction and a
reference is held indefinitely
script: cephfsnew to automate pool and fs creation
Use new on_ow_msg_send_complete hook.
Replaces on_msg_delivered for one-way message style.
Prototype new xio_discon behavior.
On shutdown, XioPortal threads should not exit before Accelio
finalizes all sessions.
Inline join_sessions, it needs sh_mtx held across wait loop.
Fix assert on Cond::Signal. Adds Cond2.
Avoid deadlock, xio_disconnect can deliver a session teardown event.
Also Mutex2.
(Note, Mutex2 and Cond2 are replaced by standard C++ downstream.)
Restore SimpleDispatcher Timings.
The simple_client/simple_server timings are based on a ping/pong
of messages between the client and server, unlike those of the
xio_client/server programs, which are one-way (so their corresponding
1-way bandwidth is appx. 2x what the test reports). We assert
that the results are in general comparable, because in both setups,
a fixed number of messages (def. 50) is maintained in flight.
Wrap Accelio mempool in XioPool, add stats.
To enable stat prints, set xio_trace_mempool. Currently, prints
to stdout at each 64K messages sent or received.
Restore _send_message(..).
Fix merge errors in simple_client, simple_dispatcher.
xio: fix for size in pool stats
Add in/outbound msg counters to XioPoolStats.
Pool stats are easier to read.
Pool stats are easier to read, and if enabled, print on session
teardown. This is a convenient time to view stats, and with a small
Make pool stats counters atomic.
Track requests using hook ctor/dtor.
Lockless, portal thread provides atomicity.
Adapt to recent changes on Accelio for_next
* Accelio options now of opaque type
* on_msg_err with extra direction param
* RDMA behavior now governed by 2 new options
XIO_OPTNAME_MAX_INLINE_DATA
XIO_OPTNAME_MAX_INLINE_HEADER
* Separated send and recv queue depth
xio_messenger: Change xio optname queue depth msgs
* Set 16k threshold to rdma buffers instead of send
* Change xio optname for queue depth msgs
XIO_OPTNAME_SND/RCV_QUEUE_DEPTH_MSGS
xio_messenger: Protect Accelio queue depth.
(Minimal send flow control.)
The guard is per xio_connection, and considering batches.
Increment happens only if xio_send_msg succeeded, decrement in
on_ms_ow_send_complete and on_msg_error. Note that we don't need
atomics because counters are touched only in the correct portal
thread.
Find XioMsg.h in msg/xio
Find XioMessenger.h and QueueStrategy.h in msg/xio (tests).
Adapt to 2 Accelio API changes.
1. xio_context_stop loop takes only 1 argument
2. xio_connect() now takes a structure argument, by reference
Set CMP0046 iif CMake version >= 3
Move XioMessenger to msg/xio
xio: fix for segfault on xio_connect()
No more Mutex2, Cond2.
xio: number of portal threads is configurable
xio: only create additional portals on bind()
xio: use QueueStrategy(1) as default
xio: Messenger factory accepts ms_type "xio"
xio: use ms_type instead of client,cluster_rdma
removing the ability to configure the client and cluster networks
separately in favor of a single global messenger type
--xio is now a command-line alias for --ms_type xio
all daemons now use the Messenger::create() factory function instead of
conditionally creating XioMessengers
the OSD and Monitor classes no longer need separate messengers to
deal with both tcp/rdma clients
xio: portal binding honors ms_bind_port_min,max
xio: remove xio_port_shift
port shifting is no longer necessary, because we won't create both tcp
and xio messengers for the same service
Use Accelio sglist helper macros.
xio: make xio buffer unshareable
xio: Nuke special_handling.
Replace GENERIC with MON (requested by Sage).
Signed-off-by: Casey Bodley <casey@cohortfs.com>
Signed-off-by: Vu Pham <vu@mellanox.com>
Signed-off-by: Matt Benjamin <matt@cohortfs.com>
CMake Ceph Build System (Firefly)
CMake. Add tests.
Respace src/CMakeLists.txt.
CMake. Spacing cleanups.
CMake for Firefly is Triumphant
CMake for Giant
Adapt to Giant.
Fix installation for scripts and man pages
Fix CEPH_LIBDIR and CEPH_PKGLIBDIR defines
Add erasure-code libraries
uses try_compile() to detect support for -msse flags
Fix rados object classes
Propagate Casey's cls library change to src/test.
Fix CMake build for Hammer.
Try-add rados and common to librbd link.
Fix name and linkage of libec_lrc.
Rename arch/neon.c arm.c
Fix libcommon.a dependencies (some unit tests).
Authors:
Ali Maredia <ali@cohortfs.com>
Casey Bodley <casey@cohortfs.com>
Adam Emerson <aemerson@cohortfs.com>
Marcus Watts <mdw@cohortfs.com>
Matt Benjamin <matt@cohortfs.com>
Signed-off-by: Matt Benjamin <matt@cohortfs.com>
