Commit Graph

140625 Commits

Author SHA1 Message Date
Kamoltat
0972dbf225 qa/workunits: modified tests for noautoscale flag change
modified:

`qa/workunits/mon/test_noautoscale_flag.sh`
`qa/workunits/cephtool/test.sh`

adding test coverage to files mentioned above

Fixes: https://tracker.ceph.com/issues/61922

Signed-off-by: Kamoltat <ksirivad@redhat.com>
2023-09-25 12:46:36 +00:00
Adam King
a6d174c2d6
Merge pull request #53482 from adk3798/cephadm-refactor-decorators
cephadm: start decorators.py in cephadmlib

Reviewed-by: John Mulligan <jmulligan@redhat.com>
2023-09-25 08:32:39 -04:00
Pedro Gonzalez Gomez
99c06f594c
Merge pull request #53479 from rhcs-dashboard/upgrade-ui-popover-fix
mgr/dashboard: remove empty popover when there are no health warns

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
2023-09-25 12:24:58 +02:00
Cory Snyder
32fb6a1a68 rgw: fix output formatting of bucket index check admin api
The bucket index check admin API was previously returning invalid
JSON.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
2023-09-25 10:06:45 +00:00
Aashish Sharma
bfdcabe9d9 mgr/dashboard: Fix user/bucket count in rgw overview dashboard
Donot consider buckets/users count from daemons that have similar realm
name

Fixes: https://tracker.ceph.com/issues/62964

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2023-09-25 13:46:58 +05:30
Nizamudeen A
98a10dc0e7
Merge pull request #53512 from rhcs-dashboard/promqls-enhancements
mgr/dashboard: fix prometheus queries subscriptions

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2023-09-25 13:15:11 +05:30
zdover23
d855643ddb
Merge pull request #53631 from zdover23/wip-doc-2023-09-25-architecture-9-of-x
doc/architecture: edit "HA Authentication"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-25 15:06:22 +10:00
Zac Dover
cb680a7865 doc/architecture: edit "HA Authentication"
Edit "High Availability Authentication" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-25 14:53:29 +10:00
liubingrun
e47bb22f69 rgw: improve the efficiency of buffer list utilization of chunk upload
Reduced waste of buffer::ptr by receiving multiple chunks and filling them into the buffer

AWSv4ComplMulti::recv_body() just receive one chunk and fill it into buffer.
Each 4MB buffer is actually only utilizing 64KB, leading to frequent buffer allocations.
~800GB virtual memory consumption has been observed.

Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
2023-09-24 15:43:18 -04:00
Adam King
2015892b58
Merge pull request #53575 from rkachach/fix_issue_62838
mgr/cephadm: removing double quotes from the generated nvmeof config

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
2023-09-24 09:48:20 -04:00
Adam King
e91a621ead cephadm: black format initial decorators.py
Signed-off-by: Adam King <adking@redhat.com>
2023-09-24 09:44:32 -04:00
Adam King
2fe7c87bd1 cephadm: start decorators.py in cephadmlib
Originally, wanted to move all the decorators into
their own files. Unfortunately, that isn't possible
at this time as most of them depend on things that
are still within cephadm.py This includes

list_daemons
_rm_cluster
is_fsid
termcolor
ContainerInfo
Ceph

and I'm sure I'm missing some others. We'll have to
revisit this again later when more of these things
have moved, or they can be slowly moved as their
dependencies are.

Signed-off-by: Adam King <adking@redhat.com>
2023-09-24 09:44:27 -04:00
Adam King
d80e4215f1
Merge pull request #53468 from phlogistonjohn/jjm-cephadm-move-container-types
cephadm: move container types

Reviewed-by: Adam King <adking@redhat.com>
2023-09-24 09:39:47 -04:00
zdover23
aee1f43e26
Merge pull request #53613 from zdover23/wip-doc-2023-09-23-architecture-8-of-x
doc/architecture: "Edit HA Auth"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-24 03:52:15 +10:00
Zac Dover
c71cd84ec9 doc/architecture: "Edit HA Auth"
Rewrite the explanation of how a client authenticates against a monitor.
This is a rewrite of a single paragraph, and has been set apart in its
own PR so that it can receive the maximum amount of scrutiny that the
upstream Ceph community can muster.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-24 03:11:34 +10:00
Casey Bodley
3cdfef2cf6
Merge pull request #53097 from mkogan1/rgw-wip-t62141-valg-supp
valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-23 16:11:38 +01:00
Kefu Chai
53579861ac
Merge pull request #49405 from tchaikov/clang-16
run-make-check.sh: use clang-17 if available

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
2023-09-23 20:23:15 +08:00
Rishabh Dave
56412b71a2
Merge pull request #51890 from lxbsz/wip-61394
test: unmount the mountpoint just before exiting

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-23 14:54:14 +05:30
Kefu Chai
0934413254 run-make-check.sh: use clang-17 if available
now that clang-17 has been released, let's use it if available.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2023-09-23 15:07:37 +08:00
Ronen Friedman
3413c4248f rgw/test: fix compiler warning
Fixing a compiler warning regarding ambiguity of
the overloaded operator '==' (as it allows a one-sided
const operand)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2023-09-23 01:24:05 -05:00
Ronen Friedman
4f1a4fe714 rgw/test: add std:: qualifiers to 'move'
to silence compiler warnings.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2023-09-23 01:13:58 -05:00
Ronen Friedman
3e9269fb49 tools: add std:: qualifiers to 'move'
to silence compiler warnings.
e.g. (ceph_dedup_tool.cc:1104:32: warning: unqualified call to
'std::move' [-Wunqualified-std-cast-call]
     estimate_threads.push_back(move(ptr));
                               ^
                               std::

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2023-09-23 01:13:47 -05:00
Kefu Chai
cbb06d6b0b
Merge pull request #53546 from mcv21/deb_package_fixes
debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2023-09-23 13:43:12 +08:00
Casey Bodley
1378703106
Merge pull request #50611 from cbodley/wip-rgw-sal-list-buckets
rgw/sal: list_buckets() returns RGWBucketEnts

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
2023-09-22 23:20:20 +01:00
Dhairya Parmar
9e80a71ee1 qa: fix "no orch backend set" in nfs suite
Fixes: https://tracker.ceph.com/issues/62870
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-23 03:21:42 +05:30
Casey Bodley
06f6a3e6e8 qa/rgw/sts: keycloak task installs java manually
java had already been installed automatically before centos 9. add an
override to install the jdk-17 packages manually

Fixes: https://tracker.ceph.com/issues/62536

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 17:37:36 -04:00
Cory Snyder
4728daa555 rgw: fix radosgw-admin bucket check stat calculation bug
Fixes a regression with radosgw-admin bucket check stat
calculation and bucket reshard stat calculation when
there are objects that have transitioned from unversioned
to versioned. The bug was introduced in
152aadb71b.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
2023-09-22 21:17:58 +00:00
Cory Snyder
340522f9ae rgw: add test case to reproduce bucket check stats bug for versioned bucket
Reproduces a regression where radosgw-admin bucket check incorrectly counts
objects that started as unversioned and later transitioned to versioned.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
2023-09-22 21:17:55 +00:00
sayantani11
3537737108
script/backport-resolve-issue: Update script with latest versions
Signed-off-by: Sayantani Saha <ii.sayantani.ii@gmail.com>
2023-09-23 02:39:34 +05:30
Casey Bodley
7ba05a2916 rgw: s3website doesn't prefetch for web_dir() check
this function only needs to check for existence of the given path.
the sal::Object is destroyed before the function returns, so it's
wasteful to prefetch its data

Fixes: https://tracker.ceph.com/issues/62938

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 15:27:40 -04:00
Casey Bodley
958578aec1 rgw/sal: StoreBucket no longer wraps RGWBucketEnt
`sal::Bucket` no longer needs to wrap `RGWBucketEnt` to support user
bucket listings, so can be represented by `RGWBucketInfo` alone. the
bucket stats interfaces that relied on RGWBucketEnt internally now
return their result as either `RGWBucketEnt` or `RGWStorageStats`

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 12:49:07 -04:00
Casey Bodley
50836121a6 rgw/sal: list_buckets() returns RGWBucketEnts
`sal::User::list_buckets()` no longer returns a map of `sal::Bucket`
handles. it now uses `std::span<RGWBucketEnt>` for input and output.
`RGWBucketEnt` contains all of the information we need to satisfy
ListBuckets requests, and also stores the `rgw_bucket` key for use with
`Driver::get_bucket()` where a `sal::Bucket` handle is necessary

`sal::BucketList` contains the span of results and the `next_marker`.
the `is_truncated` flag was removed in favor of `!next_marker.empty()`

the checks for `user->get_max_buckets()` on bucket creation now use a
paginated `check_user_max_buckets()` helper function that limits the
number of allocated entries to `rgw_list_buckets_max_chunk`

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 12:49:07 -04:00
Matthew Vernon
e4393e574c debian: call dh_python3 for ceph-{base,common,fuse,volume}
In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
67abe9c49a debian: radosgw.init to installinit, remove auto_build override
Installation of init scripts properly belongs with dh_installinit, so
move the installation there.

That means we no longer need the override of dh_auto_build, which
simplifies the rules file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
efd469e1b1 debian: specify a dependency on python3 for cephadm
cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
96edd84e47 debian: remove obsolete ceph-base.docs, restore dh_installdocs
debian/ceph-base.docs only referred to a README that doesn't exist, so
remove it. Because dpkg-source doesn't reflect deletions from debian/
cf the orig.tar.gz, also remove the file in dh_auto_clean.

Then do away with the removal of the empty override of dh_installdocs;
the main benefit of which here is that debian/copyright gets installed
in all of the built packages, which otherwise lack a copyright
file.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
74188c2780 debian: correct maintainer address
This means that debian/control matches changelog entries, and that the
Maintainer address is up to date.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
86863990ca debian: dh compat to 12, necessary init/systemd adjustments
Bring the dh compat level to 12, the most recent supported by the
oldest supported Ubuntu LTS release, 20.04. This necessitates changes
to how initscripts & systemd packaging are done.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:04 +01:00
Matthew Vernon
842d6aed00 debian/copyright: update syntax, maintainer, add license stanzas
Update the header paragraph to link to the canonical URL for the
format, and point to dev@ceph.io as the Contact.

Also add License: stanzas to reflect the licences in use (and refer to
fuller versions in /usr/share/common-licenses/ as appropriate).

This means that packages containing this copyright file are better in
compliance with the licences concerned.

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
7ec0f6975b debian: add missing item separators in debian/control
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
bf09f66545 debian: remove {Build-,}Depends on essential packages
Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:

https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
eaaf80725c debian: specify interpreters for ceph-mon and ceph-osd postinsts
These were previously missing. The requirement for interpreters is in
Debian policy section 10.4:
https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

Debian's packaging already adds the #! to these two postinsts. In
practice, a text executible without a #! line will likely be executed
by the calling shell, so a lot of the time we'd get away with it
unless the administrator is using an incompatible shell like tcsh.

This behaviour of shells is documented in POSIX section 1(e)(i)(b)
here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:32:03 +01:00
Matthew Vernon
c14ed9bf1b debian: Build-Depend on g++ 11 or greater
Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.

This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.

Fixes: https://tracker.ceph.com/issues/61845

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
2023-09-22 17:31:50 +01:00
Rishabh Dave
9fedc1e062
Merge pull request #52524 from dparmar18/wip-62072
tools/cephfs_mirror: only perform actions if init succeed

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-22 20:15:04 +05:30
Venky Shankar
0aedb3cfdc Merge PR #53407 into main
* refs/pull/53407/head:
	Client: Fix nonblocking-io zero by read

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-22 20:11:49 +05:30
Venky Shankar
8a84a6cc70 Merge PR #53279 into main
* refs/pull/53279/head:
	qa: descrease pgbench scale factor to 32 for postgresql database test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2023-09-22 20:11:12 +05:30
Venky Shankar
3b06ea9915 Merge PR #52747 into main
* refs/pull/52747/head:
	cephfs-mirror: restart failed/blocklisted replayer instances
	cephfs-mirror: do not run concurrent C_RestartMirroring context

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-22 20:10:42 +05:30
Venky Shankar
8f9fc152c5 Merge PR #53418 into main
* refs/pull/53418/head:
	qa: use exisitng ignorelist override list for fs:mirror[-ha]

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
2023-09-22 20:09:44 +05:30
Rishabh Dave
954ed30f3c
Merge pull request #48027 from lxbsz/wip-57154
mds/client: check the cephx mds auth access in client side

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
2023-09-22 20:08:10 +05:30
Casey Bodley
9b7cd7fc7d
Merge pull request #52576 from cfsnyder/wip-62075
rgw: add radosgw-admin bucket check olh/unlinked commands

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-22 15:33:25 +01:00