Commit Graph

141815 Commits

Author SHA1 Message Date
Patrick Donnelly
23af075146
Merge PR #54726 into main
* refs/pull/54726/head:
	PendingReleaseNotes: announce cephfs-shell avail. on rhel9
	qa: test fs:shell on all distros
	qa: add cephfs-shell to installed rpm packages
	ceph.spec.in: enable support for cephfs-shell by default via EPEL9

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-12-11 08:37:39 -05:00
Rishabh Dave
d174be7556
Merge pull request #54779 from rishabh-d-dave/fs-release-notes
cephfs: add a couple of release notes

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-12-11 16:32:48 +05:30
Ilya Dryomov
8d4bf56d2a
Merge pull request #54848 from idryomov/wip-diff-iterate-test-fixup
test/librbd: close image in DiffIterateDeterministic

Reviewed-by: Mykola Golub <mgolub@suse.com>
2023-12-11 11:47:22 +01:00
Ilya Dryomov
97dd6442aa
Merge pull request #54847 from idryomov/wip-journaling-test-fixups
test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-11 11:08:08 +01:00
Samuel Just
9b597a74ca
Merge pull request #53054 from xxhdx1985126/wip-crimson-pg-proj-last-update
crimson/osd/pg: set PG::projected_last_update when submitting transactions

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2023-12-10 20:01:05 -08:00
Samuel Just
a538294a55
Merge pull request #53078 from xxhdx1985126/wip-crimson-pg-recovery
crimson/osd/pg_recovery: avoiding duplicated object recovering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
2023-12-10 20:00:31 -08:00
Samuel Just
cbd79400c0
Merge pull request #53127 from xxhdx1985126/wip-crimson-obc-iteration
crimson/osd/pg: discard watches' states after iterating all cached obcs

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2023-12-10 19:59:30 -08:00
Samuel Just
6532501f8b
Merge pull request #53409 from xxhdx1985126/wip-crimson-recover-head-first
crimson/osd/pg_recovery: recover head before recovering clones

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-12-10 19:58:55 -08:00
Samuel Just
6b6c84f1ed
Merge pull request #53985 from xxhdx1985126/wip-63186
crimson/osd: store "final pool info" for pools that are deleted

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-12-10 19:57:57 -08:00
Samuel Just
84325b2ec1
Merge pull request #53306 from xxhdx1985126/wip-clone-wait-head
crimson/osd/osd_operations/client_request: recover the head and other necessary objects before proceeding

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-12-10 19:56:45 -08:00
Yingxin
330161855e
Merge pull request #53130 from cyx1231st/wip-crimson-drop-mempool
mempool: avoid true sharing for the counters, with crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
2023-12-11 09:39:50 +08:00
zdover23
e324cbf35f
Merge pull request #54845 from zdover23/wip-doc-2023-12-09-radosgw-role-2
doc/radosgw: edit front matter - role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-12-10 23:58:46 +01:00
Zac Dover
9d10361961 doc/radosgw: edit front matter - role.rst
Clarify the front matter in doc/radosgw/role.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-12-11 07:38:34 +10:00
Yuval Lifshitz
eddde0d3ea
Merge pull request #54789 from yuvalif/wip-yuval-remove-lua-devel
build/rgw/lua: remove luarocks support from rhel8

reviewed-by: batrick, cbodley, mchangir
2023-12-10 22:47:53 +02:00
Yuval Lifshitz
b0a77a53ab build/rgw/lua: remove lua-devel as a runtime dependency
this is due to the fact that lua-devel
cannot be installed in teuthology for rhel8

this would fail the build of some luarocks packages

Fixes: https://tracker.ceph.com/issues/63672

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2023-12-10 08:09:35 +00:00
Ilya Dryomov
be40bbbb12 test/librbd: close image in DiffIterateDeterministic
... to avoid valgrind reporting a memory leak on ImageCtx.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-09 22:35:58 +01:00
Ilya Dryomov
d790b57b4b test/librbd: actually alternate overlaps in DiscardWithPruneWriteOverlap
Make sense of the inner loop in write_thread.  The crash on
"it != m_events.end()" assert reproduces even faster this way.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-09 21:35:55 +01:00
Ilya Dryomov
de397f7588 test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap
For reasons that I think no longer apply today, set_val() and
set_val_or_die() refuse to set "type: str" config options that aren't
marked as "can be changed at runtime" -- set_val() returns an error and
set_val_or_die() terminates the process.  What is and isn't marked as
"can be changed at runtime" seems to be pretty much random both within
and outside of RBD, so let's just refactor how config is set here.

While at it, I realized that reproducer config is underspecified:

- for rbd_cache_policy and rbd_cache_writethrough_until_flush settings
  to matter, rbd_cache must be set to true and rbd_cache_max_dirty must
  be set to a positive number

- order should be set explicitly, because rbd_default_order can be as
  low as 12 (for 4096-byte objects), interfering with the logic of the
  test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-09 21:35:55 +01:00
Adam King
ec21a43687
Merge pull request #54378 from adk3798/cephadm-jinja2-logrotate
cephadm: move logrotate configs to jinja2 template

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2023-12-09 14:56:34 -05:00
Anthony D'Atri
95a800c98b
Merge pull request #54846 from zdover23/wip-doc-2023-12-09-radosgw-role
doc/radosgw: format POST statements
2023-12-09 09:16:22 -05:00
Zac Dover
1c0901cf85 doc/radosgw: format POST statements
Format the POST methods so that they appear in the rendered text as
examples of POST API calls and not as plain old unformatted text, which
is how they looked before this commit. The content of these API calls
remains to be tested and confirmed to work, but this is a first step.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-12-09 04:46:00 +01:00
Ilya Dryomov
d5122b9c79
Merge pull request #54839 from cbodley/wip-librbd-strands
librbd: include asio strand headers in librbd.cc

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-08 23:47:31 +01:00
Casey Bodley
512fc74fb6
Merge pull request #54827 from cbodley/wip-vstart-dbstore
vstart: add --rgw_store option for rados|dbstore|posix

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2023-12-08 21:57:20 +00:00
Casey Bodley
e94de28235
Merge pull request #54546 from sonetto-lmz/post-etag
rgw: fix post obj does not return the response etag field

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2023-12-08 19:45:28 +00:00
Casey Bodley
ecd96ca8ff
Merge pull request #54622 from BBoozmen/oozmen_lc_work_time
[rgw][lc][rgw_lifecycle_work_time] adjust timing if the configured end time is less than the start time

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-12-08 19:44:56 +00:00
Casey Bodley
dc634cc4d6
Merge pull request #54668 from qiuxinyidian/rgw-frontend
add checking for rgw frontend init

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-12-08 19:44:35 +00:00
Casey Bodley
e6224ce777
Merge pull request #54738 from cbodley/wip-63485
rgw/iam: admin/system users ignore iam policy parsing errors

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2023-12-08 19:43:54 +00:00
Casey Bodley
e15e6568ab
Merge pull request #54749 from cbodley/wip-rgw-zone-old-pools
rgw/zone: remove check for (very) old pools config

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
2023-12-08 19:43:31 +00:00
Casey Bodley
4304ca95d3 librbd: include asio strand headers in librbd.cc
Fixes: https://tracker.ceph.com/issues/63682

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-12-08 11:24:57 -05:00
Casey Bodley
260644a763 vstart: add --rgw_store option for rados|dbstore|posix
enables dbstore for rgw_backend_store and rgw_config_store, allowing
vstart to run without any mons or osds. database files are put under
the dev subdirectory

when rgw_store=posix, the posix filter is added on top of dbstore

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-12-08 10:09:43 -05:00
Rishabh Dave
6f581fb588 cephfs: move release note for PR #41779 to right spot
Release note for PR #41779 applies releases after Reef, not  Reef
onwards. Move the release note accordingly.

Fixes: https://github.com/ceph/ceph/pull/41779
Fixes: https://tracker.ceph.com/issues/47264
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2023-12-08 15:11:29 +05:30
zdover23
191fefc4d1
Merge pull request #54751 from zdover23/wip-doc-2023-12-02-radosgw-multisite-starting-the-gateway
doc/radosgw: add gateway starting command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-12-08 06:09:15 +01:00
Adam King
eed127258c
Merge pull request #54773 from phlogistonjohn/jjm-cephadm-zipapp-templating-again
cephadm: simplify loading jinja2 templates from zipapp

Reviewed-by: Adam King <adking@redhat.com>
2023-12-07 14:33:12 -05:00
Casey Bodley
9cba1f1d5e
Merge pull request #54278 from smanjara/wip-fix-rgw-full-sync
rgw/multisite: fix marker_tracker flush() call in incremental sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-12-07 15:05:47 +00:00
Casey Bodley
7970cd8475
Merge pull request #54810 from mkogan1/rgw-wip-t63445-valg-supp-rb
rgw: d3n: fix valgrind reported leak related to libaio worker threads

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-12-07 15:03:39 +00:00
zdover23
b4102be9e0
Merge pull request #54809 from joscollin/wip-doc-mirror-add-supported-file-types
doc: add supported file types in cephfs-mirroring.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-12-07 08:41:19 +01:00
Ilya Dryomov
cc1951a5d6
Merge pull request #54377 from baergj/fix-journal-tid-association-try2-upstream
librbd: Append one journal event per image request

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-06 21:11:19 +01:00
Casey Bodley
77e2e02079
Merge pull request #54799 from cbodley/wip-osd-scrub-signed-warning
osd/scrubber: fix signed comparison warning

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2023-12-06 13:12:57 +00:00
Jos Collin
3d44e5dfdc
doc: add supported file types in cephfs-mirroring.rst
Signed-off-by: Jos Collin <jcollin@redhat.com>
2023-12-06 18:36:47 +05:30
Mark Kogan
dd258d8da0 rgw: d3n: fix valgrind reported leak related to libaio worker threads
which sporadically reproduces on teuthology ubuntu instances

happens because a race between RGW shutdown occurring before
the libaio worker threads had terminated

to fix, reduced the libaio threads inactivity shutdown time

ref:
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...

Fixes: https://tracker.ceph.com/issues/63445

Signed-off-by: Mark Kogan <mkogan@ibm.com>
2023-12-06 14:48:40 +02:00
Ilya Dryomov
1ce887c639
Merge pull request #54755 from idryomov/wip-63654
librbd: fix regressions in ObjectListSnapsRequest

Reviewed-by: Mykola Golub <mgolub@suse.com>
2023-12-06 11:49:32 +01:00
Nizamudeen A
1039443c0f
Merge pull request #54804 from rhcs-dashboard/upgrade-grafana-9.4.12
monitoring: upgrade grafana container to 9.4.12

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
2023-12-06 15:38:41 +05:30
Ilya Dryomov
05c4c92505
Merge pull request #54670 from trociny/wip-cython-3
pybind/rbd: fix compilation with cython3

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2023-12-06 10:52:09 +01:00
zdover23
7aa81b735d
Merge pull request #54790 from zdover23/wip-doc-2023-12-05-radosgw-rgw-cache-nginx-link-update
doc/radosgw: update link in rgw-cache.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-12-06 09:32:27 +01:00
Zac Dover
ec7c515490 doc/radosgw: add gateway starting command
Add a command that properly starts (or restarts) the RADOS gateway after
RGW settings have been changed. This commit has been added in response
to an issue reported anonymously on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-12-06 09:25:32 +01:00
Yuval Lifshitz
70fbeb1746
Merge pull request #54215 from kchheda3/wip-fix-kafka-break
rgw/notification: Kafka persistent notifications not retried and removed even when the broker is down 

reviewd-by: cbodley, yuvalif
2023-12-06 10:21:42 +02:00
Nizamudeen A
dd0a5aac96 monitoring: upgrade grafana container to 9.4.12
Fixes the CVEs mentioned here: https://grafana.com/blog/2023/06/06/grafana-security-release-new-grafana-versions-with-security-fixes-for-cve-2023-2183-and-cve-2023-2801/

Signed-off-by: Nizamudeen A <nia@redhat.com>
2023-12-06 11:56:44 +05:30
Casey Bodley
8cd25c3624
Merge pull request #54787 from cbodley/wip-versioned-variant-inline
common: use inline for monostate dencoders

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2023-12-06 01:55:57 +00:00
Joshua Baergen
7436b4c990 test/librbd: Add workaround for a journaling deadlock that this test exposes
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
2023-12-05 14:25:46 -07:00
Joshua Baergen
4a8fa2da72 librbd: Append one journal event per image request
In the case where an image request is split across multiple object
extents and journaling is enabled, multiple journal events are appended.
Prior to this change, all object requests would wait for the last
journal event to complete, since journal events complete in order and
thus the last one completing implies that all prior journal events were
safe at that point.

The issue with this is that there's nothing stopping that last journal
event from being cleaned up before all object requests have stopped
referring to it. Thus, it's entirely possible for the following sequence
to occur:
1. An image request gets split into two image extents and two object
   requests. Journal events are appended (one per image extent).
2. The first object request gets delayed due to an overlap, but the
   second object request gets submitted and starts waiting on the last
   journal event (which also causes a C_CommitIOEvent to be instantiated
   against that journal event).
3. Journaling completes, and the C_CommitIOEvent fires. The
   C_CommitIOEvent covers the entire range of data that was journaled in
   this event, and so the event is cleaned up.
4. The first object request from above is allowed to make progress; it
   tries to wait for the journal event that was just cleaned up which
   causes the assert in wait_event() to fire.

As far as I can tell, this is only possible on the discard path today,
and only recently. Up until 21a26a7528
(librbd: Fix local rbd mirror journals growing forever), m_image_extents
always contained a single extent for all I/O types; this commit changed
the discard path so that if discard granularity changed the discard
request, m_image_extents would be repopulated, and if the request
happened to cross objects then there would be multiple m_image_extents.

It appears that the intent here was that there should be one journal
event per image request and the pending_extents kept track of what had
completed thus far. This commit restores that 1:1 relationship.

Fixes: https://tracker.ceph.com/issues/63422
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
2023-12-05 14:25:46 -07:00