common,test,cmake: various changes to re-enable build on osx
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Among other things, pass -O3, so that it goes faster.
Huge thank you to Yann Collet for identifying this as a build issue!
Signed-off-by: Sage Weil <sage@redhat.com>
- static linking #define to get the advanced stuff
- hint src size to compressor
- no flush needed before end
- check for error
It's about twice as 25% faster now for 16KB chunks.
Note that a quick test to use resetCStream and reusing the same stream
did not appear to have any effect.
Signed-off-by: Sage Weil <sage@redhat.com>
It is not really our business to debug python, boost, or our other
dependencies. Mark them as system includes.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Fix for:
[src/compressor/lz4/LZ4Compressor.h:49]: (style) Checking if
unsigned variable 'compressed_len' is less than zero.
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
use an existing thread-local random engine instead of constructing,
seeding, and locking a separate engine
Signed-off-by: Casey Bodley <cbodley@redhat.com>
clang on osx emits functions with leading underscore, but the isa-l
assembly's functions have no leading underscore. we could label the
function declaration like `int foo asm("foo")` to remove the leading
underscore, but isa-l is a git submodule, let do this later. and in the
meanwhile, disable this plugin on osx.
Signed-off-by: Kefu Chai <kchai@redhat.com>
do not enable lz4 compressor plugin unless HAVE_LZ4 is ON. so, for
example, when the compressor type is "random", the compressor factory
won't complain with:
load failed dlopen(): "/usr/lib64/ceph/compressor/libceph_lz4.so: cannot
open shared object file: No such file or directory"
anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This upgrade brings
- Complete rewrite of DEFLATE optimizations resulting in 5X better
throughput and compression ratios comapred to zlib, lz4, lzo and 2X
better decompression performance when compared to zlib.
- AVX512 improvements to multi-buffer versions of MD5, SHA-1 and SHA-256
cryptographic hashing functions resulting in 3X better in performance
compared to the AVX2 generation.
This update improves bluestore compression and potential to improve
dedup performance.
Testing this patch with objectstore fio yeided a max of 37% increase in
compression performance.
Fio Params:
rw=randwrite,buffer_compress_percentage=50,nr_files=64,direct=1,buffered=0,size=4G,bs=64k
Test Avg BlueStore Compression Time
v2.16(us) v2.18(us)
iodepth=1,jobs=1 241.658 175.476 27.39%
iodepth=2,jobs=2 184.174 145.861 20.80%
iodepth=4,jobs=4 143.617 104.392 27.31%
iodepth=8,jobs=8 146.984 116.505 20.74%
iodepth=16,jobs=8 180.167 112.769 37.41%
Signed-off-by: Ganesh Mahalingam <ganesh.mahalingam@intel.com>
Signed-off-by: Tushar Gohad <tushar.gohad@intel.com>
* reverts c93daaf92b
* add "#include "include/assert.h" to where <cassert> or <assert.h> are
included, or avoid including them if possible.
Signed-off-by: Kefu Chai <kchai@redhat.com>
compare_exchange_strong() is used because on architectures without CaS instructions,
compare_exchange_weak() may fail.
Signed-off-by: Jesse Williamson <jwilliamson@suse.de>
unittest_compressor was failing on arm64 because the zlib
compressor was never initialized, even though it works fine in
non-isal mode
Signed-off-by: Dan Mick <dan.mick@redhat.com>
so we can still compile if certain header file (<sys/auxv.h>) used
for detecting cpu capabilities on ppc is missing on x86 host machine.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Build/link of zstd itself is maybe not ideal, but it works fine.
Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
The problem was due to the attempt of CMake to rewrite the RPATH of
the executables, or shared libraries, that didn't have any RPATH linked
during the building phase.
Currently INSTALL_RPATH is already set for all binaries. This patch
sets an empty INSTALL_RPATH for all binaries that don't depend on
any internal (also being built) target.
Signed-off-by: Ricardo Dias <rdias@suse.com>
prior to this change, libcommon is a convenient library which gets
linked into librados, librbd and libcephfs and all ceph executables.
this incurs some problems:
- double dose of libcommon in memory space and HDD: waste of memory
and disk space.
- if an application links to two libraries including libcommon at the
same time. take librados and libcephfs as an example, they could
interfere with each other by changing the other guy's status.
after this change, libcommon is tuned into a shared library and
renamed to libceph-common. it will be installed into $prefix/lib/ceph,
and packaged in librados2.
ceph.spec.in,debian/librados2.install: package libceph-common in
librados2.
CMakeLists.txt:
- do not link against libboost-* if not necessary.
- s/common/ceph-common/g
- install libceph-common into $prefix/lib/ceph
- set rpath to $prefix/lib/ceph
- link against ceph-common if an executable needs access to non public
symbols in ceph.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Compression and erasure coding plugins are now statically compiled
into libcephd. A new method is added to load them into the
respective registry.
The static libraries are only built when WITH_EMBEDDED is enabled
and existing plugins are unaffected.
Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>
We dynamically enable this if the necessary processor features are present.
Allow this probing to be disabled explicitly.
Signed-off-by: Sage Weil <sage@redhat.com>
4 bytes is enough for small buffers, but fails for larger buffers because
snappy encodes the length as a varint. 8 is enough.
Signed-off-by: Sage Weil <sage@redhat.com>
By using raw deflate instead of default zlib deflate, we're
saving 6 bytes on each compression stream and some CPU time
by not calculating Adler32 during both compression and
decompression (which is waste of time as Bluestore has its
own checksumming code that is *also* executed).
Signed-off-by: Piotr Dałek <git@predictor.org.pl>
