RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-01-02 09:02:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,435 Commits 212 Branches 394 Tags 1.5 GiB
6e6f59b945
Commit Graph

131435 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Casey Bodley
6e6f59b945 qa/rgw: disable coredumps for reshard fault injection 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
2a51bc6ce1 test/rgw: add test_bucket_reshard() for fault injection testing 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
8c2bac38f8 test/rgw: test_rgw_reshard.py exec_cmd() can return error code 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
1cb18ba018 radosgw-admin: 'bucket reshard' returns positive error codes 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
b7041fff3f rgw: reshard first cleans up old-style reshards 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
0f9870ba39 rgw: refactor reshard init/cleanup with fault injection 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e4c3bbe3ae rgw: add typedef for ReshardFaultInjector 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
38bb228934 rgw: BucketReshardManager stores BucketReshardShards by value 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
2ef324bb19 rgw: remove unused attrs from RGWBucketReshard 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
eb304f4bc5 rgw: reshard tests with fault injection 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
69fc20f0c8 common: add generic FaultInjector 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
9424b1b8b2 rgw: Better error handling 
- function update_bucket() handles updating bucket state

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
cb376a4c44 rgw: use a helper function to handle repetitive num_shards check 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
d0214ab7f7 rgw: code organization fixes and error handling 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
6a922e10c4 rgw: clean up fixes for extra spaces and indentation. 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
eb8c6b7585 rgw: helper function to set reshard status from rgw::BucketReshardState. 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
a4af6ea82a rgw: minor fixes 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
3d3ecb2a1e rgw: failing to reshard, restore old indexes 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
fbe6842165 rgw: allow clean_index to take const bucket_index_layout_generation& 
fix bi_get() to get objects after being resharded

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
7df7b275b0 rgw: - take non-optional gen_id in svc_bi_rados functions 
- call init_index() on target layout during reshard process.
       Takes const rgw::bucket_index_layout_generation&

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Manjarabad Jagannath
112aea9e19 rgw: make RGWBucketInfo non-const in functions using guard_reshard() 
- remove unused 'instance_id' from cls_rgw_reshard_entry
 - other minor fixes

Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Manjarabad Jagannath
680bfdcfeb rgw: target_layout to be optional 
Signed-off-by: Shilpa Manjarabad Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
f78b1a2ead rgw: use rgw::BucketReshardState for reshard state change 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
9f68c54fc1 rgw: fetch updated bucketinfo rather than bucket_id in guard_reshard 
To be able to read the updated layout.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
bd0b0e5d98 rgw: use BucketReshardState states in appropriate places 
This replaces cls_rgw_reshard_status.

Remove unused parameters in cls_rgw_bucket_instance_entry.
Other minor cleanup fixes.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
b6c91c506c rgw/dynamic-resharding: Allow clean_index to take generation number 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
7348a8397a rgw/dynamic-resharding: reshard bucket using the same instance id 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
faf5b79876 rgw/dynamic-resharding: remove creation of new bucket instance 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e75c6233c9 test/rgw: add --gen to radosgw-admin/help.t 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
844fafc135 rgw: add generations to error repo with binary format 
adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Shilpa Jagannath
643e9beafa rgw: add generation support to BILog admin APIs 
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
d7b6b49b6c rgw: remove rgw_bucket_shard_sync_info::full_marker 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e66dd889b3 rgw: disable backoff on data sync error_retry_time 
this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Adam C. Emerson
e432f4a4be qa/rgw: rgw_multisite_tests task loads default plugins 
the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
e7f3ac92a7 qa/rgw: disable multisite tests for 'bucket sync disable' 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
ecad0e388a qa/rgw: add missing meta checkpoint to test_multipart_object_sync 
if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
5e58ef856c rgw: RGWSyncBucketCR holds lease over Init state too 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
3184aa7b18 rgw: fix up BucketShardIncrementalSync log message 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
799bd1bcac rgw: move full sync from SyncBucketShard to SyncBucket 
renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
d78907825f radosgw-admin: 'bucket sync checkpoint' waits for full sync 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
1e90a3e753 radosgw-admin: 'bucket sync status' displays new full sync status 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
50e6b3a5e4 rgw: add rgw_read_bucket_full_sync_status() 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
cce09513a5 rgw: InitBucketShardSyncStatus always sets state to Incremental 
full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
6bf0a9e250 rgw: split SyncBucket from SyncBucketShard 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
06c30057f4 rgw: add InitBucketFullSyncStatusCR 
a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
d9946514f0 rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
5403c85ae5 rgw: system objects can set exclusive on set_attrs() 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
444c5fff47 rgw: add exclusive flag to RGWSimpleRadosWriteCR 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
aa1aca4ebe rgw: generalize error handling in RGWShardCollectCR 
RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00
Casey Bodley
293f6612e6 rgw: add sync_bucket_shard_cr() factory function 
allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>
 2022-05-27 15:47:31 -04:00