Commit Graph

116129 Commits

Author SHA1 Message Date
Xuehan Xu
951f47fc1c qa/suites: add crimson-rados/thrash tests
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-22 11:27:35 +08:00
Lenz Grimmer
6bd8fa8679
Merge pull request #37088 from Devp00l/wip-37408
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2020-10-21 15:06:52 +02:00
Lenz Grimmer
7d9d3a60d5
Merge pull request #37416 from tspmelo/wip-rbd-clone
mgr/dashboard: Disable RBD clone action when conditions are not met

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2020-10-21 15:06:27 +02:00
Daniel Gryniewicz
0477cc2b0e
Merge pull request #37396 from SMIL-IT/fix-keycount
rgw: fix: S3 API KeyCount incorrect return.
2020-10-21 08:48:03 -04:00
Mykola Golub
656386e9c0
Merge pull request #37727 from dillaman/wip-librbd-readresult-clip
librbd: remove io::ReadResult::set_clip_length

Reviewed-by: Mykola Golub <mgolub@suse.com>
2020-10-21 11:14:24 +03:00
Mykola Golub
2ee4aecb02
Merge pull request #37726 from dillaman/wip-librbd-remove-image-cache
librbd: remove remainder of ImageCache API hooks

Reviewed-by: Mykola Golub <mgolub@suse.com>
2020-10-21 11:13:35 +03:00
Kefu Chai
20d7d5180a
Merge pull request #37614 from smithfarm/wip-revert-suse-podman-requires
Revert "rpm: on SUSE, podman is required for cephadm to work" and "spec: Podman (temporarily) requires apparmor-abstractions on suse"

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-10-21 15:23:06 +08:00
Kefu Chai
af4d035d3c
Merge pull request #37403 from athanatos/sjust/wip-seastore-gc
crimson/seastore: add online gc to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-10-21 11:01:42 +08:00
Sebastian Wagner
687ade71be
Merge pull request #37731 from adk3798/cephadm-logs
doc: document additional journal args in cephadm logs command

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-10-21 00:51:49 +02:00
Adam King
5224c5baf4 doc: document additional journal args in cephadm logs command
Signed-off-by: Adam King <adking@redhat.com>
2020-10-20 16:08:28 -04:00
Samuel Just
5f3348e06a test/crimson/seastore/test_btree_lba_manager: fix get_overlap
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
33f61b832a test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
0266f5851e crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs
insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
95ddd9cfc8 crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals
Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
88fd179489 crimson/common/fixed_kv_node_layout: add support for node dependent values
node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
0f5b5d0043 test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
513ec5333b crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions
Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
2ac5ba6fbf test/crimson/seastore: extract common test logic for transaction_maanager initialization
This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:12 -07:00
Samuel Just
debd83d175 crimson/os/seastore/segment_cleaner: add online gc
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
c8d3d47dfe test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs
We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
562224b70c crimson/os/seastore/lba_manager: add get_physical_extent_if_live
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
37783172a6 crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
c300ec9630 os/seastore/journal: replay_segment: skip replay on deltas to released segments
See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
41df0a17b4 crimson/os/seastore/journal: fix get_journal_seq
current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
7b242b56ef crimson/os/seastore: add journal_replay check and debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
f3b79db4e4 crimson/os/seastore/journal: add segment_header_t operator<<
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
085acb425f crimson/os/seastore/cache: add duplicate_for_write debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
7328acd052 crimson/os/journal: check offset against segment length in replay
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
db86ad6011 crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
b7edb6c8da crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
12caf7851d crimson/os/seastore/journal: improve encode_record debugging
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
820383ce4a crimson/os/seastore/journal: use scan_segment to add extent scan
Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
8981c67aad crimson/os/seastore/journal: generalize replay scan
We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
1f8199f7f5 crimson/os/seastore/journal: record extents types and laddr in record metadata
We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
4bc64de8ea crimson/os/seastore/journal: use metadata size for block base rather than assuming one block
Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:08 -07:00
Samuel Just
bf1912289b crimson/os/seastore: add space accounting to segment_cleaner and wire in
Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>
2020-10-20 12:27:04 -07:00
Jason Dillaman
ca4f546f63 librbd: remove io::ReadResult::set_clip_length
Combined the functionality with the newer 'set_image_extents' call
to simplify the call-site logic for IO reads.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-10-20 13:10:06 -04:00
Patrick Donnelly
e5f917aa1c
Merge PR #37707 into master
* refs/pull/37707/head:
	doc: document the mds_autoscaler mgr plugin

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-20 08:21:22 -07:00
Jason Dillaman
f53c3eef2b librbd: remove remainder of ImageCache API hooks
With the incorporation of the ImageDispatcher, there is no longer a
need for the ImageCache (and related) classes which were added as
a temporary workaround to incorporate the PWL cache.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2020-10-20 11:13:11 -04:00
Kefu Chai
9745a4cd19
Merge pull request #37692 from tchaikov/wip-bump-up-pytest
pybind/mgr/dashboard: s/pytest<4/pytest/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-10-20 20:07:55 +08:00
Kefu Chai
4583c33dcd pybind/mgr/dashboard: disable too-many-branches lint warning
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
71979e9b46 pybind/mgr/dashboard: use setUpClass for initializeing class
instead of relying on __init__(), use setUpClass() to initialize class
for testing. it turns out in pytest > 4, __init__() is called for the
test class but the attributes of the instantiated class is in turn overriden.

so we have to use setUpClass to do this job.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
34da2a1185 pybind/mgr/dashboard: refactor overlong statement
to silence lint warning like:

services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent':'./services/tcmu_service.py:64:39: E126 continuation line over-indented for hanging indent'}
2: 1     E126 continuation line over-indented for hanging indent

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
69fa3b9add pybind/mgr/dashboard: reindent to appease lint
silences warning like:

************* Module dashboard.services.tcmu_service
services/tcmu_service.py:64:0: C0330: Wrong continued indentation (add 18 spaces).
                    or [[0, 0]])[-1][1] / 1000000000
                    ^                 | (bad-continuation)

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:13:12 +08:00
Kefu Chai
128778f25e pybind/mgr/dashboard: s/pytest<4/pytest/
to address following failure:

    The user requested pytest<4
    pytest-cov 2.10.1 depends on pytest>=4.6

when building the target of "mgr-dashboard-virtualenv"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Kefu Chai
c3f8832785 pybind/mgr/dashboard: bump flake8 to 3.8.4
to address the failure of

The conflict is caused by:
    flake8 3.7.8 depends on pycodestyle<2.6.0 and >=2.5.0
    autopep8 1.5.4 depends on pycodestyle>=2.6.0

when building the "mgr-dashboard-venv" target

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Kefu Chai
fa9e2bfd4b tools/setup-virtualenv.sh: pass --use-feature=2020-resolver to pip
as long as pip supports this option, pass it to `pip install`

to silence warnings and errors like:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

autopep8 1.5.4 requires pycodestyle>=2.6.0, but you'll have pycodestyle 2.5.0 which is incompatible.
pytest-cov 2.10.1 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-10-20 19:09:25 +08:00
Lenz Grimmer
2437ef8648
Merge pull request #36948 from bk201/wip-46492
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2020-10-20 12:58:51 +02:00
Milind Changire
50e680cb50 doc: document the mds_autoscaler mgr plugin
The MDS Autoscaler provides automation to maintain a required count of
MDSs in the system.
The MDS count in the system can be afected by:
* max_mds config option
* standby_count_wanted config option
* death of an active MDS Rank

Fixes: https://tracker.ceph.com/issues/46884
Signed-off-by: Milind Changire <mchangir@redhat.com>
2020-10-20 09:19:48 +05:30
Patrick Donnelly
4e2f47fc4e
Merge PR #37646 into master
* refs/pull/37646/head:
	mds: account for closing sessions in hit_session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-10-19 19:28:40 -07:00