so there are only two ways to override the number of daemons to start
- using the env var CEPH_NUM_{MON|OSD|MGR|MDS} or {MON|OSD|MGR|MDS}
- command line options: --{mon,osd,mds}_num
do prevent a daemon from running, set the corrresponding env var to 0.
Signed-off-by: Kefu Chai <kchai@redhat.com>
It's possible for the Sequencer to go away while the OpSequencer still has
txcs in flight. We were handling the case where the osr was on the
deferred_queue, but it may be off the deferred_queue but waiting for the
commit to happen, and we still need to wait for that.
Fix this by introducing a 'zombie' state for the osr, in which we keep the
osr in the osr_set.
Clean up the OpSequencer methods and a few other method names.
Signed-off-by: Sage Weil <sage@redhat.com>
We've been avoiding doing this for a while and it has finally caught up
with us: the SharedBlob may outlive the split due to deferred IO, and
a read on the child collection may load a competing Blob and SharedBlob
and read from the on-disk blocks that haven't been written yet.
Fix by preserving the one-SharedBlob-instance invariant by moving cache
items to the new Collection and cache shard like we should have from the
beginning.
Signed-off-by: Sage Weil <sage@redhat.com>
We can't use a bare Collection since we get/put refs, the last put will
delete it, and the dtor asserts nref == 0 (no faking a ref and deliberately
leaking!).
Signed-off-by: Sage Weil <sage@redhat.com>
Otherwise cache items survive beyond umount into the next mount cycle!
Also, ensure that we flush_cache *before* clearing coll_map, as some cache
items have references back to the Collection.
Signed-off-by: Sage Weil <sage@redhat.com>
These can survive as long as the txc, which can be longer than the
Collection. Make sure we have a valid ref as both finish_write and
~SharedBlob use coll for the SharedBlobSet (and coll->store->cct for
debug).
Signed-off-by: Sage Weil <sage@redhat.com>
We were modifying bufferlists in place, and kludging around it by making
full copies elsewhere. Instead, never modify a buffer.
This fixes issues where the buffer we submit to ObjectStore ends up in
the cache and we modify in place later, corrupting the implementation's
copy. (This was affecting BlueStore.)
Rearrange the data methods to be next to each other and clean them up a
bit too.
Signed-off-by: Sage Weil <sage@redhat.com>
If we have no non-deferred IO to flush, and we are running bluefs on a
single shared device, then we can rely on the bluefs flush to make our
current batch of deferred ios stable.
Separate deferred into a "done" and "stable" list. If we do sync, put
everything from "done" onto "stable". Otherwise, after we do our kv
commit via bluefs, move "done" to "stable" then.
Signed-off-by: Sage Weil <sage@redhat.com>
When the Sequencer goes away it get deregistered. If there are still
deferred IOs in flight, we need to wait for those too.
Signed-off-by: Sage Weil <sage@redhat.com>
Allow several deferred writes to accumulate before we submit them. In
general we have no time pressure, and on HDD (and perhaps sometimes SSD)
it is beneficial to accumulate and batch these so that they result in
fewer seeks. On HDD, this is particularly true of seeks away from the
journal. And on sequential workloads this can avoid seeks. In may even
allow the block layer or SSD firmware to merge IOs and perform fewer
writes.
Signed-off-by: Sage Weil <sage@redhat.com>
If a blob is shared, we can't discard deallocated regions: there may
be deferred buffers in flight and we might get a read via the clone.
Signed-off-by: Sage Weil <sage@redhat.com>
In a simple HDD workload with queue depth of 1, we halve our throughput
because the kv thread does a full commit twice per IO: once for the
initial commit, and then again to clean up the deferred write record. The
second wakeup is unnecessary; we can clean it up on the next commit.
We do need to do this wakeup in a few cases, though, when draining the
OpSequencers: (1) on replay during startup, and (2) on shutdown in
_osr_drain_all().
Send everything through _osr_drain_all() for simplicity.
This doubles HDD qd=1 IOPS from ~50 to ~100 on my 7200 rpm test device
(rados bench 30 write -b 4096 -t 1).
Signed-off-by: Sage Weil <sage@redhat.com>
First, eliminate the work queue--it's useless. We are dispatching aio and
should not block. And if a single thread isn't sufficient to do it, it
probably means we should be parallelizing kv_sync_thread too (which is our
only caller that matters).
Repurpose the old osr-list -> txc-list-per-osr queue structure to manage
the queuing. For any given osr, dispatch one batch of aios at a time,
taking care to collapse any overwrites so that the latest write wins.
Signed-off-by: Sage Weil <sage@redhat.com>
Make osr_set refcounts so that it can tolerate a Sequencer destruction
racing with flush or a Sequencer that outlives the BlueStore instance
itself.
Signed-off-by: Sage Weil <sage@redhat.com>
The old implementation is racy and doesn't actually work. Instead, rely
on a list of all OpSequencers and drain them all.
Signed-off-by: Sage Weil <sage@redhat.com>
This ensures that we don't trim an onode from the cache while it has a
txc that is still in flight. Which in turn ensures that if we try to read
the object, we will have any writing buffers available.
Signed-off-by: Sage Weil <sage@redhat.com>
BlueStore collection methods only need preceding transactions to be
applied to the kv db; they do not need to be committed.
Note that this is *only* needed for collection listings; all other read
operations are immediately safe after queue_transactions().
Signed-off-by: Sage Weil <sage@redhat.com>
Currently this is the same as flush, but more precisely it is an internal
method that means all txc's must complete. Update _wal_apply() to use it
instead of flush(), which is part of the public Sequencer interface.
Signed-off-by: Sage Weil <sage@redhat.com>
This reverts 3e40595f3c
The individual throttles have their own set of perfcounters; no need to
duplicate them here.
Signed-off-by: Sage Weil <sage@redhat.com>
The throttle is really about limiting deferred IO; we do not need to
actually remove the deferred record from the kv db before queueing more.
(In fact, the txc that queues more will do the cleanup.)
Signed-off-by: Sage Weil <sage@redhat.com>
We can unblock flush()ing threads as soon as we have applied to the kv db,
while the callbacks must wait until we have committed.
Move methods around a bit to better match the execution order.
Signed-off-by: Sage Weil <sage@redhat.com>
The only remaining flush() users only need to see previous txc's applied
to the kv db (e.g., _omap_clear needs to see the records to delete them).
Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/os/bluestore/BlueStore.h
We do not release extents until after any deferred IO, so this flush() is
unnecessary.
Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/os/bluestore/BlueStore.cc
We rely on the buffer cache to avoid reading any deferred write data. In
order for that to work, we have to ensure the entire block whose
overwrite is deferred is in the buffer cache. Otherwise, a write to 0~5
that results in a deferred write could break a subsequent read from 5~5
that reads the same block from disk before the deferred write lands.
Signed-off-by: Sage Weil <sage@redhat.com>
It does not matter if we update the freelist in the initial commit or when
cleaning up the deferred transaction; both will eventually update the
persistent kv freelist. We maintain one case to ensure that legacy
deferred events (from a kraken upgrade) release when they are replayed.
What matters while online is the Allocator, which has an independent
in-memory copy of the freelist to make decisions. And we can delay that
as long as we want. To avoid any concerns about deferred writes racing
against released blocks, just defer any release until the txc is fully
completed (including any deferred writes). This ensures that even if we
have a pattern like
txc 1: schedule deferred write on block A
txc 2: release block A
txc 1+2: commit
txc 2: done!
txc 1: do deferred write
txc 1: done!
then txc 2 won't do its release because it is stuck behind txc 1 in the
OpSequencer queue:
...
txc 1: reaped
txc 2: reaped (and extents released to alloc)
This builds in some delay in just-released space being usable again, but
it should be a very small amount of space relative to the size of the
store!
Signed-off-by: Sage Weil <sage@redhat.com>
"wal" can refer to both the rocksdb wal (effectively, or journal) and the
"wal" events we include in it (mainly promises to do future IO or release
extents to the freelist). This is super confusing!
Instead, call them 'deferred'.. deferred transactions, ops, writes, or
releases.
Signed-off-by: Sage Weil <sage@redhat.com>
