Ceph is a distributed object, block, and file storage platform
Go to file
Erwan Velu 8b6be11a36 tests: Adding parallelism to encoding/readable.sh
When running make -j x check, we face a weird situation where the makefile
targets are spawn in parallel up to "x" but one of those target is very very
long and sequential.

The "readable.sh" test is trying to run ~7.9K tests where 5.3K are actually
executed.

The current code is taking 23mn on a recent laptop (Intel(R) Core(TM)
i7-4810MQ CPU @ 2.80GHz, 32GB of RAM & SSD).

This patch implements parallelism to speed up this process which is not really CPU and
neither IO bound.

By default, readable.sh is now using the number of logical processors to determine
the level of parallelism (by using nproc). If needed, defining the MAX_PARALLEL_JOBS
variable will override this default value.

On the same system, where nproc=8, the resulting execution time is 5m55 seconds :
4x faster than the original code.

The global 'make check' is therefore getting faster too and dropped from 30 to
16 minutes : 2x faster than the original code.

Signed-off-by: Erwan Velu <erwan@redhat.com>
2016-04-05 09:36:25 +02:00
admin
bin
ceph-erasure-code-corpus@c332279082
ceph-object-corpus@47fbf8c6ae
cmake/modules
debian debian/rules: put init-ceph in /etc/init.d/ceph, not ceph-base 2016-03-31 21:13:13 -07:00
doc releases: what is merged where and when ? 2016-03-31 15:10:40 +02:00
etc set 128MB tcmalloc cache size by bytes 2016-04-04 13:41:52 +08:00
examples
fusetrace
keys
m4
man
mirroring
qa tests: Adding parallelism helpers in ceph-helpers.sh 2016-04-05 09:36:25 +02:00
rpm
selinux
share
src tests: Adding parallelism to encoding/readable.sh 2016-04-05 09:36:25 +02:00
systemd
udev
.gitignore
.gitmodule_mirrors
.gitmodules
.mailmap
.organizationmap
.peoplemap
AUTHORS
autogen.sh
ceph.spec.in Merge pull request #7857 from tchaikov/wip-14844 2016-03-30 14:47:34 +08:00
ChangeLog
CMakeLists.txt
CodingStyle
configure.ac configure: Add -D_LARGEFILE64_SOURCE to Linux build. 2016-03-31 12:30:45 -04:00
CONTRIBUTING.rst
COPYING
COPYING-GPL2
COPYING-LGPL2.1
do_autogen.sh
doc_deps.deb.txt
Doxyfile
INSTALL
install-deps.sh
make_dist.sh
make-debs.sh
Makefile.am
NEWS
PendingReleaseNotes
pom.xml
README
README.aix
README.cmake
README.md
README.solaris
README.xio
run-cmake-check.sh
run-make-check.sh
SubmittingPatches.rst

============================================ Ceph - a scalable distributed storage system

Please see http://ceph.com/ for current info.

Contributing Code

Most of Ceph is licensed under the LGPL version 2.1. Some miscellaneous code is under BSD-style license or is public domain. The documentation is licensed under Creative Commons Attribution-ShareAlike (CC BY-SA). There are a handful of headers included here that are licensed under the GPL. Please see the file COPYING for a full inventory of licenses by file.

Code contributions must include a valid "Signed-off-by" acknowledging the license for the modified or contributed file. Please see the file SubmittingPatches.rst for details on what that means and on how to generate and submit patches.

We do not require assignment of copyright to contribute code; code is contributed under the terms of the applicable license.

Build Prerequisites

The list of Debian or RPM packages dependencies can be installed with:

./install-deps.sh

Note: libsnappy-dev and libleveldb-dev are not available upstream for Debian Squeeze. Backports for Ceph can be found at ceph.com/debian-leveldb.

Building Ceph

Autotools

Developers, please refer to the Developer Guide for more information, otherwise, you can build the server daemons, and FUSE client, by executing the following:

./autogen.sh
./configure
make

(Note that the FUSE client will only be built if libfuse is present.)

CMake

Prerequisite: CMake 2.8.11

Build instructions:

mkdir build
cd build
cmake [options] /path/to/ceph/src/dir
make

(Note that /path/to/ceph/src/dir can be in the tree and out of the tree)

Dependencies

The configure script will complain about any missing dependencies as it goes. You can also refer to debian/control or ceph.spec.in for the package build dependencies on those platforms. In many cases, dependencies can be avoided with --with-foo or --without-bar switches. For example,

./configure --with-nss         # use libnss instead of libcrypto++
./configure --without-radosgw  # do not build radosgw
./configure --without-tcmalloc # avoid google-perftools dependency

Building packages

You can build packages for Debian or Debian-derived (e.g., Ubuntu) systems with

sudo apt-get install dpkg-dev
dpkg-checkbuilddeps        # make sure we have all dependencies
dpkg-buildpackage

For RPM-based systems (Red Hat, SUSE, etc.),

rpmbuild

Building the Documentation

Prerequisites

The list of package dependencies for building the documentation can be found in doc_deps.deb.txt:

sudo apt-get install `cat doc_deps.deb.txt`

Building the Documentation

To build the documentation, ensure that you are in the top-level `/ceph directory, and execute the build script. For example:

admin/build-doc