Ceph is a distributed object, block, and file storage platform
Go to file
Dongmao Zhang 75332450e3 Fix rados_shutdown hang forever when using radosstriper
Dear list,

I have met this when I was using radosstriper C API. My program is
roughly like this:

    rados_striper_aio_write
    rados_aio_flush
    rados_aio_wait_for_safe
    rados_aio_release
    rados_striper_destroy
    rados_ioctx_destroy
    rados_shutdown /Hangs here/

In most time, this works well, But the programm occasionally
hangs forever. Output of gstack:

Thread 1 (Thread 0x7fe0afba0760 (LWP 18509)):
0 0x000000330f20822d in pthread_join () from /lib64/libpthread.so.0
1 0x000000347566cea2 in Thread::join(void**) () from
/usr/lib64/librados.so.2
2 0x00000034755ac535 in librados::RadosClient::shutdown() () from
/usr/lib64/librados.so.2
3 0x0000003475592269 in rados_shutdown () from /usr/lib64/librados.so.2
4 0x0000000000402349 in main ()

Thread 4 (Thread 0x7fe0ab14d700 (LWP 18541)):
0 0x000000330f20e264 in __lll_lock_wait () from /lib64/libpthread.so.0
1 0x000000330f209508 in _L_lock_854 () from /lib64/libpthread.so.0
2 0x000000330f2093d7 in pthread_mutex_lock () from
/lib64/libpthread.so.0
3 0x0000003475633af1 in Mutex::Lock(bool) () from
/usr/lib64/librados.so.2
4 0x00000034755abd37 in librados::RadosClient::put() () from
/usr/lib64/librados.so.2
5 0x0000003475592501 in librados::Rados::shutdown() () from
/usr/lib64/librados.so.2
6 0x00007fe0afbba9f7 in
libradosstriper::RadosStriperImpl::CompletionData::~CompletionData() ()
from /usr/lib64/libradosstriper.so.1
7 0x00007fe0afbbaad9 in
libradosstriper::RadosStriperImpl::WriteCompletionData::~WriteCompletionData()
() from /usr/lib64/libradosstriper.so.1
8 0x00007fe0afbc1d75 in RefCountedObject::put() () from
/usr/lib64/libradosstriper.so.1
9 0x00007fe0afbc224d in
libradosstriper::MultiAioCompletionImpl::safe_request(long) () from
/usr/lib64/libradosstriper.so.1
10 0x00000034755c5ce8 in librados::C_AioSafe::finish(int) () from
/usr/lib64/librados.so.2
11 0x00000034755a0e89 in Context::complete(int) () from
/usr/lib64/librados.so.2
12 0x000000347564d4c8 in Finisher::finisher_thread_entry() () from
/usr/lib64/librados.so.2
13 0x000000330f2079d1 in start_thread () from /lib64/libpthread.so.0
14 0x000000330eae886d in clone () from /lib64/libc.so.6

It is obvious that librados::Rados::shutdown is not a thread-safe
function here. It will hang forever. The culprit of this is when
CompletionData is released, it will first notify
"rados_aio_wait_for_safe" to continue, and CompletionData will call
put() to release other data. But if the main thread(Thread 1 here) runs
fast enough, rados_striper_destroy will be executed before other
thread(Thread 4 here)'s releasing refcnf. In this situation, main thread
runs Rados::shutdown() while other thread runs Rados::shutdown() in the
same time.

My suggestion is to let RadosStriperImpl::aio_flush to block until all
the CompletionData has been released. This makes sure other thread will
never call rados_shutdown.
2014-10-31 10:52:20 +08:00
admin be nitpicky about missing references 2014-03-24 13:01:38 -04:00
ceph-object-corpus@bb3cee6b85 ceph-object-corpus: rebase onto firefly corpus 2014-05-03 07:59:28 -07:00
debian 0.86 2014-10-07 12:16:58 -07:00
doc doc/release-notes: v0.87 giant (draft) 2014-10-24 15:22:29 -07:00
examples rbd-replay: Convert prep-for-replay.py to rbd-replay-prep.cc 2014-08-21 10:57:33 -07:00
fusetrace remove superfluous second semicolons at end of lines 2014-05-08 10:09:25 +02:00
keys
m4 configure: verify c++11 is present for rocksdb 2014-07-31 21:11:24 -07:00
man rbd-replay: Add --dump-perf-counters option 2014-09-11 16:48:08 -07:00
qa Merge pull request #2733 from dachary/wip-9372-injectargs 2014-10-23 12:32:37 -07:00
rpm Move fedora patch file (used by ceph.spec.in) to rpm/ subdir 2014-08-27 12:56:43 -07:00
share ceph-post-file: single command to upload a file to cephdrop 2013-08-16 17:59:11 -07:00
src Fix rados_shutdown hang forever when using radosstriper 2014-10-31 10:52:20 +08:00
systemd systemd: ceph-mds and ceph-mon need networking too 2014-08-27 09:22:59 -07:00
udev udev: /dev/disk/by-parttypeuuid/$type-$uuid 2013-06-17 09:49:53 -07:00
.gitignore examples: Add example for the librados C API 2014-08-21 10:57:28 -07:00
.gitmodule_mirrors build: add ceph-erasure-code-corpus to gitmodule_mirrors 2014-10-08 21:22:02 +02:00
.gitmodules .gitmodules: ignoring changes in rocksdb submodule 2014-10-24 11:14:03 -07:00
.mailmap mailmap: Jiantao He name normalization 2014-10-20 14:07:00 -07:00
.organizationmap mailmap: Wyllys Ingersoll affiliation 2014-10-20 14:07:00 -07:00
.peoplemap mailmap: Ilya Dryomov transition from Inktank to Red Hat 2014-10-19 17:40:15 -07:00
AUTHORS AUTHORS 2014-07-17 21:33:22 -07:00
autogen.sh autogen.sh: debug with -x 2014-07-31 21:11:25 -07:00
ceph.spec.in rpm: 95-ceph-osd-alt.rules is not needed for centos7 / rhel7 2014-10-13 08:16:40 -07:00
ChangeLog
CodingStyle
configure.ac check rocksdb library when '--with-librocksdb' option is enabled 2014-10-23 13:20:58 +08:00
CONTRIBUTING.rst doc: Add a contributing file at the root of repo 2014-09-06 00:23:24 +05:30
COPYING erasure-code: HTML display of benchmark results 2014-08-04 11:42:05 +02:00
COPYING-LGPL2.1
deps.deb.txt add rocksdb bz2 dep 2014-07-31 21:11:24 -07:00
deps.rpm.txt add rocksdb bz2 dep 2014-07-31 21:11:24 -07:00
do_autogen.sh Default configure with nss instead of cryptopp 2014-10-21 08:44:43 +02:00
doc_deps.deb.txt README: move package dependencies into separate files 2014-05-20 21:56:05 -07:00
Doxyfile
INSTALL Update INSTALL to mention the submodules/recursive 2014-05-31 11:22:41 +02:00
Makefile.am Makefile.am: make sure everything was build before make check 2014-10-08 20:19:07 +02:00
NEWS
PendingReleaseNotes doc/release-notes: v0.85 2014-09-08 12:19:34 -07:00
pom.xml update license for libcephfs 2014-08-26 15:52:37 -04:00
README README: symlink from README.md 2014-08-06 08:15:35 -07:00
README.md README.md: word wrap 2014-08-06 08:16:21 -07:00
SubmittingPatches SubmittingPatches: clarify "github fork" in preference to "clone" 2014-03-06 21:11:18 -08:00

============================================ Ceph - a scalable distributed storage system

Please see http://ceph.com/ for current info.

Contributing Code

Most of Ceph is licensed under the LGPL version 2.1. Some miscellaneous code is under BSD-style license or is public domain. The documentation is licensed under Creative Commons Attribution-ShareAlike (CC BY-SA). There are a handful of headers included here that are licensed under the GPL. Please see the file COPYING for a full inventory of licenses by file.

Code contributions must include a valid "Signed-off-by" acknowledging the license for the modified or contributed file. Please see the file SubmittingPatches for details on what that means and on how to generate and submit patches.

We do not require assignment of copyright to contribute code; code is contributed under the terms of the applicable license.

Build Prerequisites

debian-based

The list of debian package dependencies can be found in deps.deb.txt:

sudo apt-get install `cat deps.deb.txt`

Note: libsnappy-dev and libleveldb-dev are not available upstream for natty, oneiric, and squeeze. Backports for Ceph can be found at ceph.com/debian-leveldb.

rpm-based

The list of RPM package dependencies can be found in deps.rpm.txt:

sudo yum install `cat deps.rpm.txt`

Building Ceph

Developers, please refer to the Developer Guide for more information, otherwise, you can build the server daemons, and FUSE client, by executing the following:

./autogen.sh
./configure
make

(Note that the FUSE client will only be built if libfuse is present.)

Dependencies

The configure script will complain about any missing dependencies as it goes. You can also refer to debian/control or ceph.spec.in for the package build dependencies on those platforms. In many cases, dependencies can be avoided with --with-foo or --without-bar switches. For example,

./configure --with-nss         # use libnss instead of libcrypto++
./configure --without-radosgw  # do not build radosgw
./configure --without-tcmalloc # avoid google-perftools dependency

Building packages

You can build packages for Debian or Debian-derived (e.g., Ubuntu) systems with

sudo apt-get install dpkg-dev
dpkg-checkbuilddeps        # make sure we have all dependencies
dpkg-buildpackage

For RPM-based systems (Red Hat, SUSE, etc.),

rpmbuild

Building the Documentation

Prerequisites

The list of package dependencies for building the documentation can be found in doc_deps.deb.txt:

sudo apt-get install `cat doc_deps.deb.txt`

Building the Documentation

To build the documentation, ensure that you are in the top-level `/ceph directory, and execute the build script. For example:

admin/build-doc