RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-03-11 02:39:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,446 Commits 219 Branches 396 Tags 1.4 GiB
ccd68c668d
Commit Graph

131446 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Casey Bodley
ccd68c668d rgw: add json encoding of bucket layout types 
adds a "layout" section to RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
9fd4db2bf7 json: add encode_json() overload for string_view 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
5c7b68606e rgw: refactor per-entry reshard logic into separate function 
this cuts down on nesting and avoids the need for goto

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
1c1bd6447c rgw: reshard adds a log layout for the new index 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
122da8f14a radosgw-admin: try reshard even if bucket is resharding 
allow reshard in case a previous reshard failed. if the reshard is
actually still in progress, we'll fail to get the reshard lock

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
fa59327279 test: fix threading for FaultInjector death tests 
addresses test timeout and warning message:

[WARNING] /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-death-test.cc:1121:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads. See https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests-and-threads for more explanation and suggested solutions, especially if this is the last message you see before your test times out.

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
6156f0475f radosgw-admin: remove fault injection options from usage 
these are only used for testing, not administration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
d99f1fd2a5 rgw: RGWBucketReshard doesn't need a friend 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
ed40166b1d rgw: fix decode of cls_rgw reshard types 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
91d5ba628f rgw: BucketReshardManager takes layouts 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
824fdc91af rgw: clean up uses of BucketShard::init() without info 
the rgw_bucket overload of BucketShard::init() has to look up the bucket
info. use the RGWBucketInfo overload when we have one

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:32 -04:00
Casey Bodley
6e6f59b945 qa/rgw: disable coredumps for reshard fault injection 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
2a51bc6ce1 test/rgw: add test_bucket_reshard() for fault injection testing 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
8c2bac38f8 test/rgw: test_rgw_reshard.py exec_cmd() can return error code 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
1cb18ba018 radosgw-admin: 'bucket reshard' returns positive error codes 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
b7041fff3f rgw: reshard first cleans up old-style reshards 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
0f9870ba39 rgw: refactor reshard init/cleanup with fault injection 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e4c3bbe3ae rgw: add typedef for ReshardFaultInjector 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
38bb228934 rgw: BucketReshardManager stores BucketReshardShards by value 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
2ef324bb19 rgw: remove unused attrs from RGWBucketReshard 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
eb304f4bc5 rgw: reshard tests with fault injection 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
69fc20f0c8 common: add generic FaultInjector 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
9424b1b8b2 rgw: Better error handling 
- function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
cb376a4c44 rgw: use a helper function to handle repetitive num_shards check 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
d0214ab7f7 rgw: code organization fixes and error handling 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
6a922e10c4 rgw: clean up fixes for extra spaces and indentation. 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
eb8c6b7585 rgw: helper function to set reshard status from rgw::BucketReshardState. 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
a4af6ea82a rgw: minor fixes 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
3d3ecb2a1e rgw: failing to reshard, restore old indexes 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
fbe6842165 rgw: allow clean_index to take const bucket_index_layout_generation& 
fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
7df7b275b0 rgw: - take non-optional gen_id in svc_bi_rados functions 
- call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Manjarabad Jagannath
112aea9e19 rgw: make RGWBucketInfo non-const in functions using guard_reshard() 
- remove unused 'instance_id' from cls_rgw_reshard_entry
 - other minor fixes

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Manjarabad Jagannath
680bfdcfeb rgw: target_layout to be optional 
Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
f78b1a2ead rgw: use rgw::BucketReshardState for reshard state change 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
9f68c54fc1 rgw: fetch updated bucketinfo rather than bucket_id in guard_reshard 
To be able to read the updated layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
bd0b0e5d98 rgw: use BucketReshardState states in appropriate places 
This replaces cls_rgw_reshard_status.

Remove unused parameters in cls_rgw_bucket_instance_entry.
Other minor cleanup fixes.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
b6c91c506c rgw/dynamic-resharding: Allow clean_index to take generation number 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
7348a8397a rgw/dynamic-resharding: reshard bucket using the same instance id 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
faf5b79876 rgw/dynamic-resharding: remove creation of new bucket instance 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e75c6233c9 test/rgw: add --gen to radosgw-admin/help.t 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
844fafc135 rgw: add generations to error repo with binary format 
adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
643e9beafa rgw: add generation support to BILog admin APIs 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
d7b6b49b6c rgw: remove rgw_bucket_shard_sync_info::full_marker 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e66dd889b3 rgw: disable backoff on data sync error_retry_time 
this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Adam C. Emerson
e432f4a4be qa/rgw: rgw_multisite_tests task loads default plugins 
the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e7f3ac92a7 qa/rgw: disable multisite tests for 'bucket sync disable' 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
ecad0e388a qa/rgw: add missing meta checkpoint to test_multipart_object_sync 
if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
5e58ef856c rgw: RGWSyncBucketCR holds lease over Init state too 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
3184aa7b18 rgw: fix up BucketShardIncrementalSync log message 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
799bd1bcac rgw: move full sync from SyncBucketShard to SyncBucket 
renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00