Commit Graph

138734 Commits

Author SHA1 Message Date
Casey Bodley
0bd813f069 test/pybind/cephfs: convert from nose to pytest
* use a fixture for setup_test()
* invoke test_cephfs.py with pytest

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-07-06 11:02:35 -04:00
Casey Bodley
af04457a43 test/pybind/rbd: convert from nose to pytest
* use fixtures for temporary images and groups
* use pytest.skip instead of nose.SkipTest
* replace setUp/tearDown with setup/teardown_method
* add @pytest.mark.skip_if_crimson
* replace nose assertions

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-07-06 11:02:11 -04:00
Casey Bodley
744c8b0841
Merge pull request #52216 from cbodley/wip-61838
qa/s3tests: filter on 'sts_tests' and 'webidentity_tests'

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2023-07-06 09:59:31 -04:00
Matan
9465f7e9b9
Merge pull request #50458 from Matan-B/wip-matanb-c-perf-basic
qa/suites/crimson-rados: Introduce performance testing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Mark Nelson <mark.nelson@clyso.com>
2023-07-06 15:42:05 +03:00
Ronen Friedman
de051591de
Merge pull request #52293 from ronen-fr/wip-rf-consttime
common: make some common ctors and accessors constexpr

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
2023-07-06 09:28:41 +03:00
Patrick Donnelly
58df861608
Merge PR #52290 into main
* refs/pull/52290/head:
	pybind/rbd: drop GIL when calling into librbd
	pybind/cephfs: drop gil during cephfs callouts

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
2023-07-05 13:10:13 -04:00
Pedro Gonzalez Gomez
4de76bf550
Merge pull request #50558 from rhcs-dashboard/fix-dataUnits-graph-plotting
mgr/dashboard: fix a bug where data would plot wrongly

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2023-07-05 12:29:20 +02:00
Ronen Friedman
540d4ec12b common: make some common ctors and accessors constexpr
Add the constexpr qualifier to:
- some utime_t ctors and accessors;
- a shard_id_t ctor;
- ceph_le operator

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2023-07-05 04:35:45 -05:00
Matan
e509829189
Merge pull request #51852 from Matan-B/wip-matanb-crimson-watch-notify-logs
crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-07-05 11:33:31 +03:00
Ilya Dryomov
fa09e736f3
Merge pull request #52086 from pkalever/fix-mirror-status-desc
rbd-mirror: fix image replayer shut down description on force promote

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-07-05 00:22:53 +02:00
zdover23
c5d4dbf405
Merge pull request #52207 from zdover23/wip-doc-2023-06-27-rados-operations-crush-map-edits-2-of-x-2nd-attempt
doc/rados: edit crush-map-edits (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-07-05 05:22:33 +10:00
Zac Dover
ae04debe9e doc/rados: edit crush-map-edits (2 of x)
Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-07-05 05:11:25 +10:00
zdover23
baeb4d541e
Merge pull request #52308 from zdover23/wip-doc-2023-07-05-seqdiag-test
doc/dev: remove seqdiag assets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2023-07-05 03:55:15 +10:00
Zac Dover
ce96ddd4c7 doc/dev: remove seqdiag assets
Remove seqdiag assets to determine whether the docs can be built if they
are absent. (Currently they cannot be built when they are present.) If
this works, then these diagrams will be replaced, probably with .png
files.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-07-05 03:37:10 +10:00
Matan Breizman
780f6f55cd qa/tasks/cbt.py: Install collectl manually
See: d4a04809fd

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2023-07-04 11:42:34 +00:00
Matan Breizman
abdbfa2275 qa/suites/crimson-rados: Introduce performance testing
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2023-07-04 11:42:34 +00:00
zdover23
0f303c0149
Merge pull request #51422 from yuvalif/wip-yuval-cephadm-lua-doc
doc/rgw/lua: add info uploading a script in cephadm deployment

Reviewed-by: Zac Dover <zac.dover@proton.me>
2023-07-04 03:47:40 +10:00
Ilya Dryomov
1a60f66b0a pybind/rbd: drop GIL when calling into librbd
This was missing for rbd_mirror_peer_site_add() and
rbd_get_data_pool_id().

While at it, add a test for data_pool_id().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-07-03 19:07:21 +02:00
Casey Bodley
813fb284c7
Merge pull request #52247 from cbodley/wip-57905
rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2023-07-03 11:19:36 -04:00
Mark Kogan
19f97fc5a9
Merge pull request #52249 from mkogan1/wip-t61661-d3n-valg
rgw: d3n: fix valgrind reported leak related to libaio ops
2023-07-03 17:06:21 +03:00
Soumya Koduri
0c5a77f4c9
Merge pull request #52160 from soumyakoduri/wip-skoduri-lc-acl
rgw/set_attrs: Do not modify mtime

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-07-03 10:20:48 +05:30
J. Eric Ivancich
7d93aa83bb
Merge pull request #51983 from clwluvw/aws4-date-header
rgw: pick http_date in case of http_x_amz_date absence

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-07-02 16:30:26 -04:00
Patrick Donnelly
5d249a4c97
pybind/cephfs: drop gil during cephfs callouts
This has disastorous consequences including the possibility of deadlock.
In the best case, you have the rmdir holding the GIL until the MDS
responds!

Fixes: https://tracker.ceph.com/issues/61869
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-07-02 12:05:08 -04:00
Matan Breizman
3548489f62 crimson/osd/watch: don't print bufferlists
The bufferlists clutter the logs and don't infrom of anything useful:
```
DEBUG 2023-05-26 23:36:29,780 [shard 2] osd - send_completion sending notify replies: {notify_reply_t{watcher_gid=4220, watcher_cookie=139786421930752, bl=buffer::list(len=0,

)}, notify_reply_t{watcher_gid=4220, watcher_cookie=139786422032096, bl=buffer::list(len=0,

)}}
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2023-07-02 13:57:49 +00:00
Matan Breizman
b1c05f9ace crimson/osd/watch: Add logs around Watch/Notify
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2023-07-02 13:57:49 +00:00
Matan
c42efbf587
Merge pull request #49044 from rzarzynski/wip-crimson-ertr-plain-rets
crimson/ertr: let ErrVisitorT return plain value if ValueFuncT returns seastar::future

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2023-07-02 14:02:01 +03:00
Matan
55fcd87221
Merge pull request #52032 from Matan-B/wip-matanb-crimson-no-ignore
crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-07-02 13:53:25 +03:00
Matan
6bd7cd4c38
Merge pull request #51147 from liu-chunmei/multicore-osd
Crimson/osd: support multicore osd

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-07-02 13:15:02 +03:00
Matan
1f1800b26e
Merge pull request #51916 from cyx1231st/wip-seastar-msgr-multi-core-2
crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-07-02 13:05:04 +03:00
Anthony D'Atri
00edceddba
Merge pull request #52280 from zdover23/wip-doc-2023-07-01-radosgw-multisite
doc/radosgw: refine "Zones" in multisite.rst
2023-07-01 09:46:19 -04:00
Zac Dover
8dab6c066c doc/radosgw: refine "Zones" in multisite.rst
Describe in greater detail the function of Zones in serving S3 objects,
as requested by Alexander Patrakov here:
https://github.com/ceph/ceph/pull/49981#issuecomment-1615866374.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-07-01 21:51:15 +10:00
zdover23
f79906237a
Merge pull request #52277 from zdover23/wip-doc-2023-07-01-radosgw-s3select-overview
doc/radosgw: s/s3select/S3 Select/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-07-01 21:29:24 +10:00
Zac Dover
71c327ef4f doc/radosgw: s/s3select/S3 Select/
Correct several misspelling of "S3 Select". Hat tip to Anthony D'Atri,
who caught this in an earlier PR.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-07-01 21:14:30 +10:00
J. Eric Ivancich
892fd71829
Merge pull request #47595 from 5cs/fix-lc-head-marker
rgwlc: fix lc head marker point to non-exist lc entry

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-06-30 11:33:15 -04:00
J. Eric Ivancich
fb9d481188
Merge pull request #50667 from ivancich/wip-add-reupload-testing
qa/rgw: test that multipart re-upload does not leave any orphans

Reviewed-by: Cory Snyder <csnyder@iland.com>
2023-06-30 11:30:27 -04:00
Soumya Koduri
88a05a4600
Merge pull request #52252 from soumyakoduri/wip-skoduri-dbstore
rgw/dbstore: Update README

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-06-30 17:44:02 +05:30
Yingxin
ee16718511
Merge pull request #52217 from zhscn/fix-file-local-variable
crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2023-06-30 10:32:24 +08:00
zdover23
cb5c7ecf4a
Merge pull request #52250 from zdover23/wip-doc-2023-06-29-radosgw-s3select-basic-workflow
doc/radosgw: edit "Basic Workflow" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-06-30 02:00:13 +10:00
Zac Dover
4d2c09b683 doc/radosgw: edit "Basic Workflow" in s3select.rst
Edit the "Basic Workflow" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-06-30 01:27:52 +10:00
Soumya Koduri
467c76e3dd rgw/dbstore: Update README
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2023-06-29 16:00:43 +05:30
Mark Kogan
f834e10cfe rgw: d3n: fix valgrind reported leak related to libaio ops
Fixes: https://tracker.ceph.com/issues/61661

The valgrind leak indication is a false positive in this case,
it is because the libaio internal thread have not timed out yet
when radosgw is terminated.

```
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...
```
for the sake of teuthology reducing the timeout

waiting for 2 minutes for example like below would also prevent the leak report
```
❯ env
LD_LIBRARY_PATH=/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/:$LD_LIBRARY_PATH
PYTHONPATH=$PYTHONPATH:/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/cython_modules/lib.3
RAGWEED_CONF=$(realpath ./ragweed.conf) RAGWEED_STAGES=prepare,check tox
-- -v |& ccze -Aonolookups ;   sleep 2m | pv -t ; pkill radosgw
```

Signed-off-by: Mark Kogan <mkogan@redhat.com>
2023-06-29 08:13:49 +00:00
Radoslaw Zarzynski
5cd4a0ec53
Merge pull request #52008 from rishabh-d-dave/improvements-authmon
mon: improvements and cleanup for AuthMonitor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-06-29 09:32:03 +02:00
Yingxin Cheng
a870946423 crimson/osd/heartbeat: relax the order of replacement reset and accept
With the new implementation in messenger, the order of replacement reset
and accept events cannot be determined because they are from different
connections.

Modify the heatbeat logic to tolerate the both cases.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2023-06-29 14:27:58 +08:00
Casey Bodley
b91ff3fd86 rgw: fetch_remote_obj() preserves RGW_ATTR_COMPRESSION of encrypted objects
if the source object was both compressed and encrypted, preserve its
original compression attribute so it can be decompressed the same way it
was originally compressed

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-06-28 15:19:19 -04:00
Casey Bodley
ea45da747d rgw: fetch_remote_obj() will never verify etags of encrypted objects
fetch_remote_obj() transfers objects in their encrypted form, so does
not have access to the decrypted data for checksum verification

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-06-28 15:19:19 -04:00
Casey Bodley
8d35759eb8 rgw: rgwx-skip-decrypt also skips decompression of encrypted objects
compression is applied before encryption. so if we skip decryption, we
can't decompress either

Fixes: https://tracker.ceph.com/issues/57905

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-06-28 15:19:19 -04:00
Ilya Dryomov
a0082f63e8
Merge pull request #52157 from pkalever/log-improve
rbd-mirror: add some assorted logs

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-06-28 11:45:50 +02:00
Soumya Koduri
31666d71aa rgw: Do not update mtime in set_attrs
As per https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
the only way for users to modify object metadata is to make a copy of the object and
set the metadata. Hence retain old mtime for any other attr changes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2023-06-28 13:11:26 +05:30
zdover23
2767f5a72c
Merge pull request #52194 from zdover23/wip-doc-2023-06-26-radosgw-s3select-overview
doc/radosgw: edit "Overview" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-06-28 16:43:08 +10:00
Zac Dover
e051dd1a75 doc/radosgw: edit "Overview" in s3select.rst
Edit the "Overview" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-06-28 16:16:00 +10:00