Commit Graph

140347 Commits

Author SHA1 Message Date
zdover23
aee1f43e26
Merge pull request #53613 from zdover23/wip-doc-2023-09-23-architecture-8-of-x
doc/architecture: "Edit HA Auth"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-24 03:52:15 +10:00
Zac Dover
c71cd84ec9 doc/architecture: "Edit HA Auth"
Rewrite the explanation of how a client authenticates against a monitor.
This is a rewrite of a single paragraph, and has been set apart in its
own PR so that it can receive the maximum amount of scrutiny that the
upstream Ceph community can muster.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-24 03:11:34 +10:00
Casey Bodley
3cdfef2cf6
Merge pull request #53097 from mkogan1/rgw-wip-t62141-valg-supp
valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-23 16:11:38 +01:00
Kefu Chai
53579861ac
Merge pull request #49405 from tchaikov/clang-16
run-make-check.sh: use clang-17 if available

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
2023-09-23 20:23:15 +08:00
Rishabh Dave
56412b71a2
Merge pull request #51890 from lxbsz/wip-61394
test: unmount the mountpoint just before exiting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-23 14:54:14 +05:30
Kefu Chai
0934413254 run-make-check.sh: use clang-17 if available
now that clang-17 has been released, let's use it if available.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-09-23 15:07:37 +08:00
Kefu Chai
cbb06d6b0b
Merge pull request #53546 from mcv21/deb_package_fixes
debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2023-09-23 13:43:12 +08:00
Casey Bodley
1378703106
Merge pull request #50611 from cbodley/wip-rgw-sal-list-buckets
rgw/sal: list_buckets() returns RGWBucketEnts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
2023-09-22 23:20:20 +01:00
Casey Bodley
958578aec1 rgw/sal: StoreBucket no longer wraps RGWBucketEnt
`sal::Bucket` no longer needs to wrap `RGWBucketEnt` to support user
bucket listings, so can be represented by `RGWBucketInfo` alone. the
bucket stats interfaces that relied on RGWBucketEnt internally now
return their result as either `RGWBucketEnt` or `RGWStorageStats`

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 12:49:07 -04:00
Casey Bodley
50836121a6 rgw/sal: list_buckets() returns RGWBucketEnts
`sal::User::list_buckets()` no longer returns a map of `sal::Bucket`
handles. it now uses `std::span<RGWBucketEnt>` for input and output.
`RGWBucketEnt` contains all of the information we need to satisfy
ListBuckets requests, and also stores the `rgw_bucket` key for use with
`Driver::get_bucket()` where a `sal::Bucket` handle is necessary

`sal::BucketList` contains the span of results and the `next_marker`.
the `is_truncated` flag was removed in favor of `!next_marker.empty()`

the checks for `user->get_max_buckets()` on bucket creation now use a
paginated `check_user_max_buckets()` helper function that limits the
number of allocated entries to `rgw_list_buckets_max_chunk`

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 12:49:07 -04:00
Matthew Vernon
e4393e574c debian: call dh_python3 for ceph-{base,common,fuse,volume}
In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
67abe9c49a debian: radosgw.init to installinit, remove auto_build override
Installation of init scripts properly belongs with dh_installinit, so
move the installation there.

That means we no longer need the override of dh_auto_build, which
simplifies the rules file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
efd469e1b1 debian: specify a dependency on python3 for cephadm
cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
96edd84e47 debian: remove obsolete ceph-base.docs, restore dh_installdocs
debian/ceph-base.docs only referred to a README that doesn't exist, so
remove it. Because dpkg-source doesn't reflect deletions from debian/
cf the orig.tar.gz, also remove the file in dh_auto_clean.

Then do away with the removal of the empty override of dh_installdocs;
the main benefit of which here is that debian/copyright gets installed
in all of the built packages, which otherwise lack a copyright
file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
74188c2780 debian: correct maintainer address
This means that debian/control matches changelog entries, and that the
Maintainer address is up to date.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
86863990ca debian: dh compat to 12, necessary init/systemd adjustments
Bring the dh compat level to 12, the most recent supported by the
oldest supported Ubuntu LTS release, 20.04. This necessitates changes
to how initscripts & systemd packaging are done.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
842d6aed00 debian/copyright: update syntax, maintainer, add license stanzas
Update the header paragraph to link to the canonical URL for the
format, and point to dev@ceph.io as the Contact.

Also add License: stanzas to reflect the licences in use (and refer to
fuller versions in /usr/share/common-licenses/ as appropriate).

This means that packages containing this copyright file are better in
compliance with the licences concerned.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
7ec0f6975b debian: add missing item separators in debian/control
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
bf09f66545 debian: remove {Build-,}Depends on essential packages
Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
eaaf80725c debian: specify interpreters for ceph-mon and ceph-osd postinsts
These were previously missing. The requirement for interpreters is in
Debian policy section 10.4:
https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

Debian's packaging already adds the #! to these two postinsts. In
practice, a text executible without a #! line will likely be executed
by the calling shell, so a lot of the time we'd get away with it
unless the administrator is using an incompatible shell like tcsh.

This behaviour of shells is documented in POSIX section 1(e)(i)(b)
here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
c14ed9bf1b debian: Build-Depend on g++ 11 or greater
Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.

This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.

Fixes: https://tracker.ceph.com/issues/61845

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:31:50 +01:00
Rishabh Dave
9fedc1e062
Merge pull request #52524 from dparmar18/wip-62072
tools/cephfs_mirror: only perform actions if init succeed

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-22 20:15:04 +05:30
Venky Shankar
0aedb3cfdc Merge PR #53407 into main
* refs/pull/53407/head:
	Client: Fix nonblocking-io zero by read

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-22 20:11:49 +05:30
Venky Shankar
8a84a6cc70 Merge PR #53279 into main
* refs/pull/53279/head:
	qa: descrease pgbench scale factor to 32 for postgresql database test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2023-09-22 20:11:12 +05:30
Venky Shankar
3b06ea9915 Merge PR #52747 into main
* refs/pull/52747/head:
	cephfs-mirror: restart failed/blocklisted replayer instances
	cephfs-mirror: do not run concurrent C_RestartMirroring context

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-22 20:10:42 +05:30
Venky Shankar
8f9fc152c5 Merge PR #53418 into main
* refs/pull/53418/head:
	qa: use exisitng ignorelist override list for fs:mirror[-ha]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
2023-09-22 20:09:44 +05:30
Rishabh Dave
954ed30f3c
Merge pull request #48027 from lxbsz/wip-57154
mds/client: check the cephx mds auth access in client side

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2023-09-22 20:08:10 +05:30
Casey Bodley
9b7cd7fc7d
Merge pull request #52576 from cfsnyder/wip-62075
rgw: add radosgw-admin bucket check olh/unlinked commands

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 15:33:25 +01:00
Rishabh Dave
c4f968b931
Merge pull request #52042 from rishabh-d-dave/better-errmsg-for-perm
MDSAuthCaps: print better error message for perm flag in MDS caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2023-09-22 20:02:40 +05:30
Rishabh Dave
3234c696de
Merge pull request #47533 from rishabh-d-dave/ceph-fuse-cmd
qa/cephfs: cleanup code for ceph-fuse command

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-22 20:01:13 +05:30
Casey Bodley
503c855f86
Merge pull request #52259 from cbodley/wip-61859
qa/rgw: disable 'rgw lc debug interval' in most suites

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>
2023-09-22 15:08:36 +01:00
Ronen Friedman
5ecd20f620
Merge pull request #53531 from ronen-fr/wip-rf-squeue2
osd/scrub: extract scrub initiation code out of the OSD

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-09-22 16:46:38 +03:00
Ilya Dryomov
f0c48dfcd6
Merge pull request #53311 from idryomov/wip-62711
qa/suites/{rbd,krbd}: disable POOL_APP_NOT_ENABLED health check

Reviewed-by: Ramana Raja <rraja@redhat.com>
2023-09-22 15:44:35 +02:00
Cory Snyder
6b057fe554 rgw: radosgw-admin bucket check should only print index entries with --check-objects flag
Printing all index entries can be very time consuming for large
buckets and the inability to switch this behavior off makes it
cumbersome to use the command for fixing bucket stats. This was
also preventing the command from outputting recalculated bucket
stats when the --fix flag wasn't specified.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
2023-09-22 08:35:22 +00:00
Rishabh Dave
104fd674b0
Merge pull request #53528 from rishabh-d-dave/cephfs-qa-mdtest
qa/cephfs: fix build failure for mdtest project

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-22 11:56:03 +05:30
Casey Bodley
b975270d4e qa/rgw: enable s3tests for 'storage classes'
other than the rgw/lifecycle subsuite, all of the other subsuites were just
adding storage classes to the rgw task, but not the s3tests task that
enables the storage class test cases

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 17:51:59 -04:00
Casey Bodley
d46af6d3de qa/rgw: disable 'rgw lc debug interval' in most suites
this has a performance cost, and the teuthology environment can
already be sluggish. if lifecycle expiration/transition testing
was limited to the rgw/lifecycle and rgw/cloud-transition suites,
the others would probably run significantly faster

Fixes: https://tracker.ceph.com/issues/61859

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 17:51:59 -04:00
Casey Bodley
c7f2bc42d9 qa/s3tests: use cloudtier_tests and lifecycle_tests to control filters
when `extra_attrs` was made additive, these tests were adding both
'not lifecycle_expiration' and 'lifecycle_expiration' to the command
line. use a yaml flag instead to control whether or not the s3tests task
adds the 'not' filter

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 17:51:59 -04:00
Casey Bodley
62245cb5f3 qa/rgw: remove duplicate overrides in rgw/lifecycle
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 17:51:59 -04:00
Ilya Dryomov
0b68a8b4c0 qa/suites/krbd: disable POOL_APP_NOT_ENABLED health check
... same as for rbd suite.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-09-21 23:36:07 +02:00
Ilya Dryomov
9e884ddeec qa/suites/rbd: drop POOL_APP_NOT_ENABLED from ignorelists
With "mon warn on pool no app = false" in the config, it's obviously
redundant.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-09-21 23:36:07 +02:00
Ilya Dryomov
e64830eb8e qa/suites/rbd: disable POOL_APP_NOT_ENABLED health check
Commit 990806e635 ("mon, qa: issue pool application warning even
if pool is empty") made it impossible to create a pool without raising
a (bogus) health alert.  See [1] for details.

[1] https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/ZTDYC5HN677RR26EB4P6PORN6L2IFH4R/

Fixes: https://tracker.ceph.com/issues/62711
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2023-09-21 23:36:07 +02:00
Casey Bodley
e554b28b9e
Merge pull request #53448 from cbodley/wip-62378
rgw/crypt: don't deref null manifest_bl

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2023-09-21 22:15:37 +01:00
Casey Bodley
70973ad678
Merge pull request #53045 from smanjara/wip-shilpa-revert-51772
rgw/multisite: fixes assertion failure during realm reload

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 20:59:59 +01:00
Cory Snyder
570adec5bb rgw: prevent another leftover bucket index olh entry scenario
If a call to bucket_index_link_olh or bucket_index_unlink_instance
fails, its associated pending xattr may have prevented the olh object
from being removed by another thread. We should do a best effort
cleanup attempt for this case by calling update_olh before returning
an error to the caller.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
2023-09-21 19:27:57 +00:00
zdover23
773799febb
Merge pull request #53493 from zdover23/wip-doc-2023-09-18-architecture-7-of-x
doc/architecture: "Edit HA Auth" (one of several)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-22 03:41:21 +10:00
Casey Bodley
b75cc5c10e rgw/admin: 'buckets list' takes --marker
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 13:37:16 -04:00
Casey Bodley
d9dfd382c1 rgw: remove Bucket::update_container_stats()
callers use Bucket::read_stats() to load bucket stats

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-21 13:37:16 -04:00
Casey Bodley
044b507077
Merge pull request #53505 from cbodley/wip-62771
rgw/sal: get_placement_target_names() returns void

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2023-09-21 18:30:07 +01:00
Yuval Lifshitz
94ac81981f
Merge pull request #52392 from yuvalif/wip-yuval-trace-name
rgw: rename request traces and change for tags

reviewed-by: cbodley
2023-09-21 20:12:01 +03:00