Current code allows multiple MDRequests to concurrently acquire a
remote lock. But a lock ACK message wakes all requests because they
were all put to the same waiting queue. One request gets the lock,
the rest requests will re-send the OP_WRLOCK/OPWRLOCK slave requests
and trigger assertion on remote MDS. The fix is disable concurrently
acquiring remote lock, send OP_WRLOCK/OPWRLOCK slave request only
if there is no on-going slave request.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Use "srcdn->is_auth() && destdnl->is_primary()" to check if the MDS is
inode exportor of rename operation is not reliable, This is because
OP_FINISH slave request may race with subtree import. The fix is use
a variable in MDRequest to indicate if the MDS is inode exportor.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
MDCache::handle_cache_expire() processes dentries after inodes, so the
MDCache::maybe_eval_stray() in MDCache::inode_remove_replica() always
fails to remove stray inode because MDCache::eval_stray() checks if the
stray inode's dentry is replicated.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
We should not defer processing caps if the inode is auth pinned by MDRequest,
because the MDRequest may change lock state of the inode later and wait for
the deferred caps.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Locker::simple_excl() and Locker::scatter_mix() miss is_rdlocked
check; Locker::file_excl() miss is_rdlocked check and is_wrlocked
check.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
In some rare case, Locker::acquire_locks() drops all acquired locks
in order to auth pin new objects. But Locker::drop_locks only drops
explicitly acquired remote locks, does not drop objects' version
locks that were implicitly acquired on remote MDS. These leftover
locks break locking order when re-acquiring _locks and may cause
dead lock.
The fix is indroduce DROPLOCKS slave request which drops all acquired
lock on remote MDS.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
The slaves for two phrase commit should be mdr->more()->witnessed
instead of mdr->more()->slaves. mdr->more()->slaves includes MDS
for remote auth pin and lock
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Anchor table updates for a given inode is fully serialized on client side.
But due to network latency, two commit requests from different clients can
arrive to anchor server out of order. The anchor table gets corrupted if
updates are committed in wrong order.
The fix is track on-going anchor updates for individual inode and delay
processing commit requests that arrive out of order.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
CDir::assimilate_dirty_rstat_inodes() may encounter frozen inodes that
are being renamed. Skip these frozen inodes because assimilating inode's
rstat require auth pinning the inode.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
When handling cross authority rename, the master first sends OP_RENAMEPREP
slave requests to witness MDS, then sends OP_RENAMEPREP slave request to
the rename inode's auth MDS after getting all witness MDS' acknowledgments.
Before receiving the OP_RENAMEPREP slave request, the rename inode's auth
MDS may change lock state of the rename inode and send lock messages to
witness MDS. But the witness MDS may already received the OP_RENAMEPREP
slave request and changed the source inode's authority. So the witness MDS
send lock acknowledgment message to wrong MDS and trigger assertion.
The fix is, firstly the master marks rename inode as ambiguous and send a
message to ask the rename inode's auth MDS to mark the inode as ambiguous,
then send OP_RENAMEPREP slave requests to the witness MDS, finally send
OP_RENAMEPREP slave request to the rename inode's auth MDS after getting
all witness MDS' acknowledgments.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
When replaying an directory rename operation, MDS need to find old parent of
the renamed directory to adjust auth subtree. Current code searchs the cache
to find the old parent, it does not work if the renamed directory inode is not
in the cache. EMetaBlob for directory rename contains at most one null dentry,
so MDS can use null dentry to find old parent of the renamed directory. If
there is no null dentry in the EMetaBlob, the MDS was witness of the rename
operation and there is not auth subtree underneath the renamed directory.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Server::_rename_prepare() adds null dest dentry to the EMetaBlob if
the rename operation overwrites a remote linkage. This is incorrect
because null dentry are processed after primary and remote dentries
during journal replay. The erroneous null dentry makes the dentry of
rename destination disappear.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Discover reply that adds replica dentry and inode can race with rename
if slave request for rename sends discover and waits, but waked up by
reply for different discover.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Locker::simple_eval() checks if the loner wants CEPH_CAP_GEXCL to
decide if it should change the lock to EXCL state, but it checks
if CEPH_CAP_GEXCL is issued to the loner to decide if it should
change the lock to SYNC state. So if the loner wants CEPH_CAP_GEXCL,
but doesn't have CEPH_CAP_GEXCL, Locker::simple_eval() will keep
switching the lock state.
Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
We were using a per-process counter combined with the pid. A short
running process can easily loop through and reuse the same pid later.
Instead, go for 48 bits of randomness and the pid. This way if we get
a dup pid we'll only get a dup nonce once out of 2^48 tries.
Avoids #3630 when running a libcephfs test in a loop (so that the pid
is eventually reused). This is a better fix than the broken
8b59908370. The real solution on the MDS
side involves cleaning up the msgr/MDS interaction with session
shutdown.
Signed-off-by: Sage Weil <sage@inktank.com>
make_request() clear out req->reply and frees req; we can't inspect
it here.
Instead, just assume that extra_bl is the create flag/ino if it is
present. Old code does not include an extra_bl on CREATE, and new code
will have the same first bytes for compatibility.
Signed-off-by: Sage Weil <sage@inktank.com>
Unprotect examines all pools, so use blanket x before 0.54. After
that, use class-read restricted by object_prefix to rbd_children.
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Clone needs to actually re-read the header to make sure the image is
still protected before returning. Additionally, it needs to consider
the image protected *only* if the protection status is protected -
unprotecting does not count. I thought I'd already fixed this, but
can't find the commit.
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Since 58890cfad5, regular {rbd_}open()
would fail with -EPERM if the user did not have write access to the
pool, since a watch on the header was requested.
For many uses of read-only access, establishing a watch is not
necessary, since changes to the header do not matter. For example,
getting metadata about an image via 'rbd info' does not care if a new
snapshot is created while it is in progress.
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
20496b8d2b forgot to do this. Without
this change, all class methods required regular read permission in
addition to class-read or class-write.
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
This prevented a read-only user from being able to unprotect a
snapshot without write permission on all pools. This was masked before
by the CLS_METHOD_PUBLIC flag.
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
We shouldn't look at Pipe::state in SimpleMessenger::_lookup_pipe() without
holding pipe_lock. Instead, use an atomic that we set to non-zero only
when transitioning to the terminal STATE_CLOSED state.
Signed-off-by: Sage Weil <sage@inktank.com>
Exercise some rare races by injecting delays before taking locks
via the 'ms inject internal delays' option.
Signed-off-by: Sage Weil <sage@inktank.com>
When a pipe is faulting and shutting down, we have to drop pipe_lock to
take msgr lock and then remove the entry. The Pipe in this case will
have STATE_CLOSED. Handle this case in all places we do a lookup on
the rank_pipe hash so that we effectively ignore entries that are
CLOSED.
This fixes a race introduced by the previous commit where we won't use
the CLOSED pipe and try to register a new one, but the old one is still
registered.
See bug #3675.
Signed-off-by: Sage Weil <sage@inktank.com>
If we have a con that refs a pipe but it is closed, don't use it. If
the ref is still there, it is only because we are racing with fault()
and it is about to (or just was) be detached. Either way,
Signed-off-by: Sage Weil <sage@inktank.com>
