Commit Graph

131363 Commits

Author SHA1 Message Date
Laurent Barbe
bf0951353a cephadm: ceph.manual.gpg cleanup
Cleanup file /etc/apt/trusted.gpg.d/ceph.manual.gpg when repo is removed

Signed-off-by: Laurent Barbe <laurent@ksperis.com>
2022-07-29 09:33:36 +02:00
Laurent Barbe
79c805546c cephadm: Fix repo_gpgkey should return 2 vars
when option --gpg-url is specified, the name used for the gpg filename is missing and throws an exception
this adds the string "manual" to the gpg key : /etc/apt/trusted.gpg.d/ceph.manual.gpg

Fixes: https://tracker.ceph.com/issues/56950

Signed-off-by: Laurent Barbe <laurent@ksperis.com>
2022-07-28 17:38:44 +02:00
Venky Shankar
b76b6ea168
Merge pull request #44486 from batrick/fs-workload-kclient-switches
qa: reduce fs:workload use of random selection

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-25 06:54:53 +05:30
Zack Cerza
6fee777d60
Merge pull request #44967 from batrick/nested-subset-doc
doc/dev: document nested subsets in teuthology
2022-05-24 10:23:48 -06:00
Yuri Weinstein
123c57400e
Merge pull request #46099 from NitzanMordhai/wip-nitzan-rados-test-delete-pools-parallel
systest: fix race with pool delete after socket injection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2022-05-24 08:02:26 -07:00
Yuri Weinstein
5442a42857
Merge pull request #45825 from NitzanMordhai/wip-nitzan-watch-notify-reconnect-107
librados/watch_notify: reconnect after socket injection

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
2022-05-24 08:01:45 -07:00
Yuri Weinstein
714cdc4e87
Merge pull request #45805 from ifed01/wip-ifed-more-stats
os/bluestore: more debug information from BlueStore

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2022-05-24 08:01:19 -07:00
Yuri Weinstein
bc02a55d3a
Merge pull request #45409 from NUABO/tancz
mon: add proxy to cache tier options

Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-05-24 07:59:36 -07:00
Adam King
7dbddc008c
Merge pull request #46324 from ceph/dgalloway-nodockerio
qa: Pull from quay.io not docker.io

Reviewed-by: Adam King <adking@redhat.com>
2022-05-24 10:27:03 -04:00
kalebskeithley
77e3e0d39d
Merge pull request #46366 from kalebskeithley/rocksdb
rocksdb: build with rocksdb-7.y.z
2022-05-24 07:06:50 -04:00
Rishabh Dave
d4ff8e3a2f
Merge pull request #46185 from dparmar18/fix_rm_error_output
cephfs-shell: fix rm cmd's error output

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
2022-05-24 16:13:48 +05:30
Rishabh Dave
9368f4fe6c
Merge pull request #46160 from dparmar18/fix_tracker_40860
cephfs-shell: fix rmdir cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>
2022-05-24 16:10:26 +05:30
Ernesto Puerta
0ec4ba41ad
Merge pull request #46249 from rhcs-dashboard/rbd-mirroring-commands
mgr/dashboard: enable/disable/promote/demote/resync mirroring images

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2022-05-24 12:38:14 +02:00
Venky Shankar
90ff84d0ac
Merge pull request #45728 from lxbsz/wip-55144
client: stop retrying the request when exceeding 256 times

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-24 07:54:16 +05:30
Venky Shankar
266b73b147
Merge pull request #46034 from mchangir/mgr-snap_schedule-remove-subvolume-support
mgr/snap_schedule: remove subvol interface

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-24 07:13:32 +05:30
Yingxin
08de9222d1
Merge pull request #46351 from cyx1231st/wip-seastore-cleaner-cleanup
crimson/os/seastore/segment_cleaner: minor cleanups

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-24 09:34:48 +08:00
Venky Shankar
16d173a0fd
Merge pull request #46314 from rishabh-d-dave/qa-fs-fsid
qa/vstart_runner: store FSID in LocalContext instances

 Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-24 06:56:13 +05:30
Kaleb S. KEITHLEY
eea10281e6 rocksdb: build with rocksdb-7.y.z
RocksDB 7, specifically 7.2.2 has landed in Fedora 37/rawhide.

https://tracker.ceph.com/issues/55730

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2022-05-24 08:33:37 +08:00
Neha Ojha
f0aeb2ec29
Merge pull request #46346 from ljflores/wip-lflores-testing-recovery
qa/suites/rados/thrash-erasure-code-big/thrashers: add `osd max backfills` setting to mapgap and pggrow

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-05-23 16:26:27 -07:00
Vikhyat Umrao
cd08198132
Merge pull request #46124 from SMIL-Infra/reopen_cluster_logs
mon/LogMonitor: reopen log files on SIGHUP
2022-05-23 12:49:35 -07:00
Ernesto Puerta
09b3210d64
Merge pull request #46233 from rhcs-dashboard/fix-overlapping-form-icons
mgr/dashboard: form field validation icons overlap with other icons

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-05-23 20:30:08 +02:00
Casey Bodley
51969f86a2
Merge pull request #45985 from soumyakoduri/wip-skoduri-dbstore-tests
rgw/qa: Run s3tests on dbstore backend in teuthology

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-05-23 12:55:41 -04:00
Pere Diaz Bou
8bd89415fe mgr/dashboard: expose image mirroring commands as endpoints
Expose:
  - enable/disable mirroring in image
  - promote/demote (primary and non-primary)
  - resync
  - snapshot mode:
    - mirror image snapshot (manual snapshot)
    - schedule

Fixes: https://tracker.ceph.com/issues/55645
Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
2022-05-23 14:20:50 +02:00
Xiubo Li
8a50329d56 client: stop retrying the request when exceeding 256 times
The type of 'retry_attempt' in 'MetaRequest' is 'int', while in
'ceph_mds_request_head' the type of 'num_retry' is '__u8'. So in
case the request retries exceeding 256 times, the MDS will receive
a incorrect retry seq.

In this case it's ususally a bug in MDS and continue retrying the
request makes no sense. For now let's limit it to 256. In future
this could be fixed in ceph code, so avoid using the hardcode here.

Fixes: https://tracker.ceph.com/issues/55144
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2022-05-23 14:05:25 +08:00
Venky Shankar
48e9ada562
Merge pull request #45844 from lxbsz/wip-55253
client: switch AT_NO_ATTR_SYNC to AT_STATX_DONT_SYNC

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-23 11:27:33 +05:30
Venky Shankar
4cbb268803
Merge pull request #46337 from rishabh-d-dave/missing-arg-exit-code
cephfs-shell: fixes related to Cmd2ArgparseError

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2022-05-23 11:05:03 +05:30
Soumya Koduri
a7397100aa rgw/qa: Run s3tests on dbstore backend in teuthology
Add a new test-suite to run s3tests on dbstore backend.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
2022-05-21 23:49:47 +05:30
Kefu Chai
b539372437
Merge pull request #44264 from tchaikov/wip-mipel
cmake: do not use GCC extension when detecting 16-byte atomic op

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-05-21 20:27:06 +08:00
Kefu Chai
a7c1c50717
Merge pull request #45835 from tchaikov/wip-55189
common/ceph_json: dump bool using f->dump_bool()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2022-05-21 20:10:30 +08:00
Kefu Chai
73ccf6ead8
Merge pull request #46355 from tchaikov/wip-no-annobin
ceph.spec.in: disable annobin plugin if compile with gcc-toolset

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2022-05-21 17:34:02 +08:00
Kefu Chai
6677658992 common/ceph_json: dump bool using f->dump_bool()
as per https://www.json.org/json-en.html, JSON encodes bool as
"true" or "false", without the quotes. before this change, the quotes
are always added when encoding boolean values.

but this change is not backward compatible.

encode_json()'s bool overload is used by rgw. it uses JSONObj
defined in common/ceph_json.h to decode JSON-encoded structs.
and it does not differentiate bool from str when decoding a boolean
value despite that it could have check the "quoted" member variable
of JSONObj for validating the type of value. so we should be fine.

Fixes: https://tracker.ceph.com/issues/55189
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-05-21 16:32:14 +08:00
Ronen Friedman
6db901185c
Merge pull request #46293 from ronen-fr/wip-rf-sched-test
test/osd: unit-tests for the scrubber scheduler

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-05-21 10:03:17 +03:00
Adam King
9476907f60
Merge pull request #45467 from phlogistonjohn/jjm-format-exp2
A module and decorator for generically handling format= in python mgr modules

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2022-05-20 18:36:24 -04:00
Adam King
54cdc1d47e
Merge pull request #45786 from adk3798/staggered-upgrade
mgr/cephadm: staggered upgrade

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2022-05-20 18:33:36 -04:00
Kefu Chai
5a78fe1764 ceph.spec.in: disable annobin plugin if compile with gcc-toolset
we are seeing FTBFS failures when compiling the packages with seastar
enabled, like:

Package gcc-toolset-9-annobin-9.08-4.el8.x86_64 is already installed.
...
Package redhat-rpm-config-130-1.el8.noarch is already installed.
...
CFLAGS=-O2 -g -pipe -Wall -Werror=format-security  -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
...
-- The CXX compiler identification is GNU 9.2.1
....
    Building CXX object CMakeFiles/cmTC_386fb.dir/testCXXCompiler.cxx.o
    /opt/rh/gcc-toolset-9/root/usr/bin/c++   -O2 -g -pipe -Wall -Werror=format-security  -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -o CMakeFiles/cmTC_386fb.dir/testCXXCompiler.cxx.o -c /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.0-209-g3e01cd41/rpm/el8/BUILD/ceph-17.2.0-209-g3e01cd41/x86_64-redhat-linux-gnu/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    cc1plus: fatal error: inaccessible plugin file /opt/rh/gcc-toolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/plugin/gcc-annobin.so expanded from short plugin name gcc-annobin: No such file or directory

the root cause is that redhat-rpm-config expects gcc-annobin:

$ cat /usr/lib/rpm/redhat/redhat-annobin-cc1 *cc1_options: + %{!-fno-use-annobin:%{!iplugindir*:%:find-plugindir()} -fplugin=gcc-annobin}

while this plugin is not installed by gcc-toolset-9-annobin, which installs annobin.so though:

$ rpm -ql gcc-toolset-9-annobin| grep annobin.so /opt/rh/gcc-toolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/plugin/annobin.so /opt/rh/gcc-toolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/plugin/annobin.so.0 /opt/rh/gcc-toolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/plugin/annobin.so.0.0.0

but gcc-toolset-8-annobin provides this file. upgrading to
gcc-toolset-11 does not help. see https://centos.pkgs.org/8-stream/centos-appstream-x86_64/gcc-toolset-11-annobin-plugin-gcc-10.23-1.el8.x86_64.rpm.html
so, the intermediate solution would be to disable the plugin, if
we want to use gcc-toolset to build rpm packages.

in this change, _annotated_build is undefined to prevent the compiler
from adding extra information to the binary. in general this change
shuold be safe, without these information, it'd be hard to tell if
the binary is hardened or what ABI version it expects. see
also https://fedoraproject.org/wiki/Changes/Annobin

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-05-20 14:32:18 +08:00
Rishabh Dave
ab19827858 qa/cephfs: remove temporary files
These temporary files don't matter for test execution with teuthology
but they do matter for execution with vstart_runner.py since the test
fails if these files exist already. And tests are often run repeatedly
with vstart_runner.py, unlike with teuthology.

Fixes: https://tracker.ceph.com/issues/55719
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2022-05-20 10:58:33 +05:30
Yingxin Cheng
d8f5acb8f6 crimson/os/seastore/cache: ignore maybe unused variables
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-05-20 11:27:02 +08:00
Yingxin Cheng
521a316daa crimson/os/seastore/segment_cleaner: cleanup logs
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-05-20 11:27:02 +08:00
Yingxin Cheng
90276e9fae crimson/os/seastore/segment_cleaner: move implementations to cc
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-05-20 11:26:56 +08:00
Samuel Just
c6e5724e1b
Merge pull request #46335 from cyx1231st/wip-seastore-onode-fix-unaligned-access
crimson/onode-staged-tree: fix unaligned reference to shard_pool_t::pool

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-05-19 19:16:29 -07:00
Samuel Just
b5901f9a42
Merge pull request #46333 from athanatos/sjust/wip-55705
crimson: generalize journal_seq_t to tolerate different paddr_t types

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2022-05-19 19:14:29 -07:00
Samuel Just
449a1e1d01
Merge pull request #46310 from cyx1231st/wip-seastore-tune-cleaner
crimson/os/seastore: increase journal size and decrease rewrite-dirty size

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-05-19 19:11:44 -07:00
Kefu Chai
0d45ecaf87
Merge pull request #46122 from tchaikov/wip-pmem
blk/pmem: refactor pmem_check_file_type() using std::filesystem

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-05-20 07:55:39 +08:00
Laura Flores
40062676c2 qa/suites/rados/thrash-erasure-code-big/thrashers: add osd max backfills setting to mapgap and pggrow
All `rados/thrash-erasure-code-big` tests that die due to the “wait_for_recovery” timeout have one thing in common: They contain either `thrashers/pggrow` or `thrashers/mapgap`.

The difference between pggrow and mapgap vs. all other non-offending thrashers (default, careful, fastread, and morepggrow) is that they lack an override setting for `osd max backfills`. `osd max backfills` is the max number of backfill operations allowed to/from an OSD. The higher the number, the quicker the recovery. By default, this value is 1. On all of the non-offending thrashers (default, careful, fastread, and morepggrow), the default 1 value gets overridden in their .yaml files with a value > 1. This is not the case for pggrow and mapgap, however, as they lack an `osd max backfills` override setting.

The mclock op scheduler is known to override `osd max backfills` with a high value, but all of the thrash-erasure-code-big thrashers have their op queue set to “debug_random”, which chooses randomly between op queues (the debug_random op queue is set to override the default mclock_scheduler in qa/config/rados.yaml). So, coupled with the “debug_random” op queue, the low `osd max backfill` setting is causing some tests to time out in recovery.

WITHOUT `osd max backfills`, as they are now, “mapgap” and “pggrow” tests die due to timed-out recovery about 17/100 times, as seen here with a pggrow test: http://pulpito.front.sepia.ceph.com/lflores-2022-05-18_14:24:29-rados:thrash-erasure-code-big-master-distro-default-smithi/

WITH `osd max backfills` specified, as I have suggested in this PR, 99/100 tests passed, with one test failing for a different reason:
http://pulpito.front.sepia.ceph.com/lflores-2022-05-17_22:40:27-rados:thrash-erasure-code-big-master-distro-default-smithi/

I also scheduled 145 tests WITH `osd max backfills` that are a mix of pggrow and mapgap thrashers. 144/145 tests passed, with one test failing for a different reason. http://pulpito.front.sepia.ceph.com/lflores-2022-05-17_15:27:54-rados:thrash-erasure-code-big-master-distro-default-smithi/

Fixes: https://tracker.ceph.com/issues/51076
Signed-off-by: Laura Flores <lflores@redhat.com>
2022-05-19 18:29:00 -05:00
Radoslaw Zarzynski
b3e725ac49
Merge pull request #45562 from tchaikov/wip-vstart-with-pmem
vstart.sh: add options to enable PMEM bluestore backend

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-05-20 00:08:57 +02:00
Adam King
6a68def64e doc/cephadm: staggered upgrade docs
Signed-off-by: Adam King <adking@redhat.com>
2022-05-19 17:24:56 -04:00
Adam King
0a46fcb423 mgr/cephadm: unit test for staggered upgrade param validation
Signed-off-by: Adam King <adking@redhat.com>
2022-05-19 17:24:56 -04:00
Adam King
791e1d20b3 qa/suites/orch/cephadm: staggered upgrade test
Signed-off-by: Adam King <adking@redhat.com>
2022-05-19 17:24:56 -04:00
Adam King
62a06c4647 mgr/cephadm: autopep8 changes
Signed-off-by: Adam King <adking@redhat.com>
2022-05-19 17:24:56 -04:00
Adam King
c1f3497b43 mgr/cephadm: make use of new upgrade control parameters
Fixes: https://tracker.ceph.com/issues/54135

Signed-off-by: Adam King <adking@redhat.com>
2022-05-19 17:24:56 -04:00