Commit Graph

116340 Commits

Author SHA1 Message Date
Kefu Chai
20d7d5180a
Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires
Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-10-21 15:23:06 +08:00
Rishabh Dave
6f95ebf04a doc/cephfs: add sample cephfs-shell.conf
And elaborate the description about it.

Fixes: https://tracker.ceph.com/issues/43028
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-10-21 10:10:38 +05:30
Kefu Chai
af4d035d3c
Merge pull request #37403 from athanatos/sjust/wip-seastore-gc
crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-10-21 11:01:42 +08:00
Sebastian Wagner
687ade71be
Merge pull request #37731 from adk3798/cephadm-logs
doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-10-21 00:51:49 +02:00
Adam King
5224c5baf4 doc: document additional journal args in cephadm logs command
Signed-off-by: Adam King <adking@redhat.com>
2020-10-20 16:08:28 -04:00
Samuel Just
5f3348e06a test/crimson/seastore/test_btree_lba_manager: fix get_overlap
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
33f61b832a test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
0266f5851e crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs
insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
95ddd9cfc8 crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals
Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
88fd179489 crimson/common/fixed_kv_node_layout: add support for node dependent values
node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
0f5b5d0043 test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
513ec5333b crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions
Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
2ac5ba6fbf test/crimson/seastore: extract common test logic for transaction_maanager initialization
This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
debd83d175 crimson/os/seastore/segment_cleaner: add online gc
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
c8d3d47dfe test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs
We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
562224b70c crimson/os/seastore/lba_manager: add get_physical_extent_if_live
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
37783172a6 crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
c300ec9630 os/seastore/journal: replay_segment: skip replay on deltas to released segments
See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
41df0a17b4 crimson/os/seastore/journal: fix get_journal_seq
current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
7b242b56ef crimson/os/seastore: add journal_replay check and debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
f3b79db4e4 crimson/os/seastore/journal: add segment_header_t operator<<
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
085acb425f crimson/os/seastore/cache: add duplicate_for_write debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
7328acd052 crimson/os/journal: check offset against segment length in replay
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
db86ad6011 crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
b7edb6c8da crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
12caf7851d crimson/os/seastore/journal: improve encode_record debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
820383ce4a crimson/os/seastore/journal: use scan_segment to add extent scan
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
8981c67aad crimson/os/seastore/journal: generalize replay scan
We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
1f8199f7f5 crimson/os/seastore/journal: record extents types and laddr in record metadata
We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
4bc64de8ea crimson/os/seastore/journal: use metadata size for block base rather than assuming one block
Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
bf1912289b crimson/os/seastore: add space accounting to segment_cleaner and wire in
Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:04 -07:00
J. Eric Ivancich
5b994f9059 rgw: rgw-orphan-list should use "plain" formatted rados ls output
The previous version that used "json-pretty" output for `rados ls`
added complications due to json's escaping of special characters. So
this version returns to the "plain" output for `rados ls` but deals
with entries (oids) that might have namespaces and/or locators as
well.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2020-10-20 14:04:34 -04:00
Jason Dillaman
ca4f546f63 librbd: remove io::ReadResult::set_clip_length
Combined the functionality with the newer 'set_image_extents' call
to simplify the call-site logic for IO reads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-10-20 13:10:06 -04:00
Patrick Donnelly
e5f917aa1c
Merge PR #37707 into master
* refs/pull/37707/head:
	doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-20 08:21:22 -07:00
Jason Dillaman
f53c3eef2b librbd: remove remainder of ImageCache API hooks
With the incorporation of the ImageDispatcher, there is no longer a
need for the ImageCache (and related) classes which were added as
a temporary workaround to incorporate the PWL cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-10-20 11:13:11 -04:00
Xiubo Li
de093c61ac client: add command_lock support
The Client::command_table member could get rid of the client_lock.
And this could make the client have a higher concurrency.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-20 08:14:49 -04:00
Xiubo Li
b835cad14d client: defer swapping the newmap to mdsmap
This is preparing for adding the command lock support.

For the new mds map we could access it without the client_lock, so
before swaping it to the `mdsmap` member, it will be safe to access.

With this we can defer to hold the client_lock and it will be safe
for the new command_lock to avoid possible dead lock in future or
something else.

Fixes: https://tracker.ceph.com/issues/46620
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-20 08:14:48 -04:00
Xiubo Li
931ba6a441 client: try to grab the client_lock when needed
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-20 08:14:48 -04:00
Xiubo Li
9d4c506eae client: clean up the client code
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-10-20 08:14:48 -04:00
Kefu Chai
9745a4cd19
Merge pull request #37692 from tchaikov/wip-bump-up-pytest
pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-10-20 20:07:55 +08:00
Kefu Chai
4583c33dcd pybind/mgr/dashboard: disable too-many-branches lint warning
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
71979e9b46 pybind/mgr/dashboard: use setUpClass for initializeing class
instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
34da2a1185 pybind/mgr/dashboard: refactor overlong statement
to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
69fa3b9add pybind/mgr/dashboard: reindent to appease lint
silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
128778f25e pybind/mgr/dashboard: s/pytest<4/pytest/
to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Kefu Chai
c3f8832785 pybind/mgr/dashboard: bump flake8 to 3.8.4
to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Kefu Chai
fa9e2bfd4b tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip
as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Lenz Grimmer
2437ef8648
Merge pull request #36948 from bk201/wip-46492
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-10-20 12:58:51 +02:00
Sridhar Seshasayee
e563e8a84e osd: Handle possible future work items returned from mclock scheduler.
It's quite possible for a scheduler like mClock to return no work items
based on parameters like reservation, weight and limit. For e.g., if
a client exceeds its limit for a given time period, the scheduler returns
a response indicating that the operation in its queue will be scheduled at
a later point in time. The osd in such a scenario must handle this case.

mClockScheduler::dequeue() now either returns an item to be scheduled or
a time (double) indicating when the op will be ready for dequeuing. The
next dequeue is attempted in the subsequent cycle after waiting until the
'future_time' expires or when the worker thread is notified as part of
OSD::ShardedOpWQ::_process() within the sharded threadpool worker loop.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2020-10-20 13:05:51 +05:30
Pritha Srivastava
bf3f3ba675 rgw/gc: fixing the condition when marker for a queue is
always reset to empty which causes RGWGC::list to get stuck in
a loop, which ultimately is broken out of when the queue's truncated
flag is false.

1. Check for entries size also while evaluating whether objects cache for
a gc object should be marked as 'transitioned' in case of cls_rgw_gc_list.
When there are no entries, we get back a return value of 0, and the
object cache is not marked as 'transitioned'.

2. Also for the last gc object, we need to check whether the queue is still
under process and set the correct flag.

Missing the two conditions above causes the GC::list to loop continously
over the same gc object.

Fixes: https://tracker.ceph.com/issues/47909

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
2020-10-20 11:57:06 +05:30