Commit Graph

132938 Commits

Author SHA1 Message Date
Kefu Chai
a49a475403 common/ceph_context: fix the formatting
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-07-28 07:46:55 +08:00
Adam King
6c795c597a
Merge pull request #47178 from adk3798/logrotate-owner
cephadm: add "su root root" to cephadm.log logrotate config

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com
2022-07-27 19:46:04 -04:00
Kefu Chai
d3a7e5e564
Merge pull request #46618 from aclamk/wip-dout-for-alien
crimson/alienstore: Add douts

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-07-28 07:45:39 +08:00
Adam King
f1cbe5225e
Merge pull request #47269 from adk3798/fix-repo-test
qa/workunits/cephadm: update test_repos master -> main

Reviewed-by: Guillaume Abrioux gabrioux@redhat.com
2022-07-27 19:45:11 -04:00
Richael Zhuang
4ad4e1fbc7 .gitmodules: update isa-l_crypto source address
Update address to https://github.com/intel/isa-l_crypto for it's
moved under intel repo.

Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>
2022-07-28 07:41:26 +08:00
Kefu Chai
65307131e4
Merge pull request #44499 from SMIL-Infra/fix-missed-lease-ack
osd/PeeringState: fix missed recheck_readable from laggy

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-28 07:40:09 +08:00
Kefu Chai
f9aaf7fe15
Merge pull request #45135 from lmlg/fix-32-bit
rgw,test: Fix build errors in 32 bit platforms

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2022-07-28 07:39:00 +08:00
Kefu Chai
d7fe19d05b
Merge pull request #43005 from wxypro/timeout_conn_msg_opt
msg: Add target_addr to log when connections timed out

Reviewed-by: Kefu Chai <kchai@redhat.com>
2022-07-28 07:38:14 +08:00
Kefu Chai
98c074a93f
Merge pull request #44040 from richael02/update-isal-crypto-version
crypto: update isa-l_crypto version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-28 07:34:59 +08:00
Kefu Chai
9b76560557
Merge pull request #44762 from Zhiwei-Dai/wip-zlib-compressor-isal-for-aarch64
compressor/zlib: isa-l optimization for zlib algorithm on aarch64

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-28 07:34:04 +08:00
Marcus Watts
b5880caa50 rgw: better tenant id from the uri on anonymous access
When anonymous tries access public bucket, it gets 404,
because rgw doesn't check tenant correctly.

A previous fix for this broke legacy implicit tenants,
because it didn't check for anonymous access.  This version
restricts its behavior to the anonymous user.

Fixes: https://tracker.ceph.com/issues/48001 https://tracker.ceph.com/issues/48382

Original fix by
Author: Rafał Wądołowski <rafal@rafal.net.pl>
Signed-off-by: Rafał Wądołowski <rwadolowski@cloudferro.com>

This fix
Signed-off-by: Marcus Watts <mwatts@redhat.com>
2022-07-27 16:59:47 -04:00
Daniel Gryniewicz
3b8fb3d498 RGW - Zipper - Add temporary Filter testing
In order to test filters, we need to be able to instantiate one and
stack it over a store.  Add a temporary config option to specify a
filter that can be used for testing.  This will be replaced with a
better system once the loadable module work is done.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-07-27 13:32:53 -04:00
Daniel Gryniewicz
fc78286fb3 RGW - Zipper - Filter base implementation
This is the base implementation for filters.  It passes each call on
each object through to the next layer down.  This allows filters to
subclass these, and only implement the bits they care about, leaving the
rest as pass-through.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-07-27 13:32:53 -04:00
Daniel Gryniewicz
fcd7b14a67 RGW - Zipper - make Serializer use unique_ptr
Serializer needs to use a unique_ptr like the rest of the SAL API.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-07-27 13:32:53 -04:00
Jan Sobczak
9ddc5b7737 ceph-volume: support symlinks as devices
This makes ceph-volume support passing symlinks as devices.

Fixes: https://tracker.ceph.com/issues/49103

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-07-27 16:48:53 +02:00
Guillaume Abrioux
01112ba689 ceph-volume: drop self.abspath in Device()
seems useless to have both self.path and self.abspath attributes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-07-27 16:41:26 +02:00
Guillaume Abrioux
e439ada4c2
Merge pull request #47167 from guits/drop-bs-tool-call
ceph-volume: drop `ceph-bluestore-tool` call in Device._parse()
2022-07-27 16:34:08 +02:00
myoungwon oh
92ca9c082c seastore/rbm: rename works
1. rename NVMeBlockDevice to RBMDevice
2. rename PosixNVMeDevice to NVMeBlockDevice

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2022-07-27 21:09:30 +09:00
Yingxin
78a23ebcad
Merge pull request #47290 from aisakaki/wip-omap-clear
crimson/os/seastore/omap: remove unnecessary extent loads when clearing

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2022-07-27 17:54:30 +08:00
Yingxin Cheng
133b1f83f8 crimson/os/seastore: set tail to the last-known journal head under the full trim
Rather than setting to the start_seq of the current record during replay
time.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-07-27 16:24:06 +08:00
Yingxin Cheng
d4c80b9a1b crimson/os/seastore/async_cleaner: abort when the alloc_tail goes backwards
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-07-27 16:22:12 +08:00
Yingxin Cheng
db7a484eb2 crimson/os/seastore: distinguish from commit-time and submit-time sequences
Also, the committed journal head should be updated atomically with the
rest of the complete-commit logics.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-07-27 16:20:40 +08:00
Guillaume Abrioux
24899a1f8d ceph-volume: fix is_ceph_disk_member()
`dev['NAME']` can't match `part` given that it's the name of the
parent device being compared to the partition name.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-07-27 09:59:16 +02:00
Ronen Friedman
0ebb32213a osd/scrub: pre-prepare the description text of the current state
The existing code accesses the states of the scrub FSM via the
standard "state iterator", which is definitely not thread-safe.
It is also a bit time consuming.

As it happened, new code that tried to create the state string
while not holding the PG lock - crashed the OSD.
This PR changes the relevant lines to allow for safe and fast
access to the state description text.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2022-07-27 05:48:07 +00:00
myoungwon oh
163b51f2ef seastore/rbm: rename NVMeManager to BlockRBManager
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2022-07-27 14:24:36 +09:00
Samuel Just
93e15f59de
Merge pull request #46755 from aravind-wdc/zns
Crimson: Update zns support

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2022-07-26 22:08:18 -07:00
myoungwon oh
5b622828a7 seastore/rbm: use DEBUG instead of logger().debug()
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2022-07-27 13:53:14 +09:00
Xinyu Huang
1c81d6b50c crimson/os/seastore/omap: remove unnecessary extent loads when clearing.
Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
2022-07-27 02:29:58 +00:00
Kefu Chai
e70ca62d9c
Merge pull request #47285 from rzarzynski/wip-crimson-monc-discard_result
crimson/monc: replace then_unpack() with discard_result() if possible

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-27 08:58:42 +08:00
Luciano Lo Giudice
71ab500a2f rgw,test: Fix build errors in 32 bit platforms
When building on 32 bit platforms (tested on armhf), 2 errors were detected:
The first involves the compiler being unable to disambiguate a call to
deserialize an integer (apparently because there isn't a specialization for it).
The second involves invalid usage of the builtin reinterpret_cast.

Signed-off-by: Luciano Lo Giudice <luciano.logiudice@canonical.com>
2022-07-27 08:56:55 +08:00
Kefu Chai
31aa539f26 osd: use fmt/ranges.h for printing ranges
now that fmt/ranges.h provides a generic mechinary for printing
ranges. let's just use it.

the output format is changed a little bit when it comes to
set and map, but as long as it makes sense, and we don't use
a tool to parse the logging messages for extracting information
from the messages printed by fmt::format(), we should be safe.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-07-27 07:16:11 +08:00
Samuel Just
eb6b3ba259
Merge pull request #47287 from tchaikov/wip-crimson-cleanup
crimson/osd: remove unused variable

Reviewed-by: Samuel Just <sjust@redhat.com>
2022-07-26 12:33:00 -07:00
Kefu Chai
a9bc62927d crimson/osd: remove unused variable
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-07-27 00:41:52 +08:00
Radoslaw Zarzynski
f307cae1a1 crimson/monc: replace then_unpack() with discard_result() if possible
More understandable this way.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 16:24:23 +00:00
Kefu Chai
9a74fd607a
Merge pull request #47279 from rzarzynski/wip-crimson-fix-repopreply-capture
crimson/osd: don't capture a local, on-stack address of MOSDRepOpReply

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-26 23:24:50 +08:00
Kefu Chai
f8eafce937
Merge pull request #47277 from rzarzynski/wip-crimson-renew_on_auth_reply
crimson/monc: renew tickets and rotating keys on MAuthReply

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-07-26 23:23:44 +08:00
zdover23
f5857acd68
Merge pull request #47082 from rzarzynski/wip-doc-crimson-cephadm
doc/crimson: document installing crimson with cephadm

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2022-07-27 00:45:09 +10:00
Guillaume Abrioux
0ee8351e4d ceph-volume: filter out non-existing devices
[1] has broken teuthology tests.

```
[root@smithi097 /]# ls -l /sys/block/
total 0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-3 -> ../devices/virtual/block/dm-3
lrwxrwxrwx 1 root root 0 Jul 25 14:15 dm-4 -> ../devices/virtual/block/dm-4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop1 -> ../devices/virtual/block/loop1
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop2 -> ../devices/virtual/block/loop2
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop3 -> ../devices/virtual/block/loop3
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop4 -> ../devices/virtual/block/loop4
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop5 -> ../devices/virtual/block/loop5
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop6 -> ../devices/virtual/block/loop6
lrwxrwxrwx 1 root root 0 Jul 25 14:13 loop7 -> ../devices/virtual/block/loop7
lrwxrwxrwx 1 root root 0 Jul 25 14:12 nvme0n1 -> ../devices/pci0000:00/0000:00:02.0/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1c1n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme1/nvme1c1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme1n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2c2n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme2/nvme2c2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme2n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys2/nvme2n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3c3n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme3/nvme3c3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme3n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys3/nvme3n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4c4n1 -> ../devices/virtual/nvme-fabrics/ctl/nvme4/nvme4c4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:18 nvme4n1 -> ../devices/virtual/nvme-subsystem/nvme-subsys4/nvme4n1
lrwxrwxrwx 1 root root 0 Jul 25 14:12 sda -> ../devices/pci0000:00/0000:00:1f.2/ata5/host4/target4:0:0/4:0:0:0/block/sda
[root@smithi097 /]#
```

devices like `nvme1c1n1` exist in `/sys/block` but aren't present in `/dev`

```
[root@smithi097 /]# ls -l /dev/nvme*
crw------- 1 root root  10, 122 Jul 25 14:16 /dev/nvme-fabrics
crw------- 1 root root 240,   0 Jul 25 14:12 /dev/nvme0
brw-rw---- 1 root disk 259,   0 Jul 25 14:15 /dev/nvme0n1
crw------- 1 root root 240,   1 Jul 25 14:16 /dev/nvme1
brw-rw---- 1 root disk 259,   2 Jul 25 15:24 /dev/nvme1n1
crw------- 1 root root 240,   2 Jul 25 14:16 /dev/nvme2
brw-rw---- 1 root disk 259,   4 Jul 25 15:24 /dev/nvme2n1
crw------- 1 root root 240,   3 Jul 25 14:16 /dev/nvme3
brw-rw---- 1 root disk 259,   6 Jul 25 15:24 /dev/nvme3n1
crw------- 1 root root 240,   4 Jul 25 14:16 /dev/nvme4
brw-rw---- 1 root disk 259,   8 Jul 25 15:24 /dev/nvme4n1
[root@smithi097 /]#
```

If a device isn't actually present in `/dev`, `get_block_devs_sysfs()` shouldn't return it.

[1] https://github.com/ceph/ceph/commit/c7f017b21ade3762ba5b7b9688bed72c6b6

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-07-26 16:24:16 +02:00
Guillaume Abrioux
f5f3117510 ceph-volume: drop ceph-bluestore-tool call in Device._parse()
disk.has_bluestore_label() does the same thing without a subprocess call.

Fixes: https://tracker.ceph.com/issues/56622

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-07-26 16:24:02 +02:00
Nizamudeen A
42fd81f1bd
Merge pull request #46945 from rhcs-dashboard/rbd-e2e
mgr/dashboard: rbd-mirroring e2e tests

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2022-07-26 19:51:06 +05:30
Radosław Zarzyński
60b8348a84 doc, crimson: document installing crimson with cephadm
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 13:32:19 +00:00
Radoslaw Zarzynski
15eeada305 crimson/osd: comment on dispatched in OSD::ms_dispatch()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 13:08:12 +00:00
Patrick Donnelly
889e332cf8
Merge PR #47218 into main
* refs/pull/47218/head:
	mon/MDSMonitor: fix check for standby reversion
	mds: update comment about standby-replay daemons holding a rank
	qa: test for standby-replay removal in MDSMonitor::prepare_beacon
	mds: add inject config for synthetic health message

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
2022-07-26 08:59:56 -04:00
Radoslaw Zarzynski
3c45df8182 crimson/osd: move conn into the closure in OSD::ms_dispatch()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:50:54 +00:00
Radoslaw Zarzynski
32a8436721 crimson/osd: don't capture a local, on-stack address of MOSDRepOpReply
This flaw can explain the following crash observed at Sepia:

```
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-13661-g3a14f283/rpm/el8/BUILD/ceph-17.0.0-13661-g3a14f283/x86_64-redhat-linux-gnu/boost/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = MOSDRepOpReply]: Assertion `px != 0' failed.
Aborting on shard 0.
Backtrace:
 0# gsignal in /lib64/libc.so.6
 1# abort in /lib64/libc.so.6
 2# 0x00007FB1346F4CD9 in /lib64/libc.so.6
 3# 0x00007FB13471A3F6 in /lib64/libc.so.6
 4# crimson::osd::OSD::handle_rep_op_reply(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<MOSDRepOpReply>) in ceph-osd
 5# 0x000055C4100FF8D0 in ceph-osd
 6# 0x000055C410100639 in ceph-osd
 7# crimson::osd::OSD::ms_dispatch(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<Message>) in ceph-osd
 8# crimson::net::ChainedDispatchers::ms_dispatch(seastar::shared_ptr<crimson::net::Connection>, boost::intrusive_ptr<Message>) in ceph-osd
 9# 0x000055C4165220EF in ceph-osd
10# 0x000055C416532791 in ceph-osd
11# 0x000055C4165328B2 in ceph-osd
12# 0x000055C40FDD33D5 in ceph-osd
13# 0x000055C40FDEE2CE in ceph-osd
14# 0x000055C40FDEEEF0 in ceph-osd
15# 0x000055C41CA41215 in ceph-osd
16# 0x000055C41CA97959 in ceph-osd
17# 0x000055C41CBF73A0 in ceph-osd
18# 0x000055C41CBF93F5 in ceph-osd
19# 0x000055C41C629A46 in ceph-osd
20# 0x000055C41C62E012 in ceph-osd
21# main in ceph-osd
22# __libc_start_main in /lib64/libc.so.6
23# _start in ceph-osd
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:50:54 +00:00
Radoslaw Zarzynski
eac4c56af0 crimson/monc: renew_rotating_keyring() doesn't assume the clock is monotonic
According to the `seastar::lowres_system_clock` reference:

> This clock has the same granularity as lowres_clock,
> but it is not required to be monotonic and its time
> points correspond to system time.

we should similar check as the classical `MonClient` does:

```cpp
  if ((now > last_rotating_renew_sent) &&
      double(now - last_rotating_renew_sent) < 1) {
    ldout(cct, 10) << __func__ << " called too often (last: "
                   << last_rotating_renew_sent << "), skipping refresh" << dendl;
    return 0;
  }
```

This check is rather paranoidal and the main reason behind it
in crimson is replicating the classical behaviour.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:04:54 +00:00
Radoslaw Zarzynski
b9d2456fa6 crimson/monc: unify handling auth_service_ticket_ttl with classical OSD
In the classical `MonClient` the `auth_service_ticket_ttl` is lower
bounded to `30` units.

```cpp
  utime_t now = ceph_clock_now();
  utime_t cutoff = now;
  cutoff -= std::min(30.0, cct->_conf->auth_service_ticket_ttl / 4.0);
  utime_t issued_at_lower_bound = now;
  issued_at_lower_bound -= cct->_conf->auth_service_ticket_ttl;
  if (!rotating_secrets->need_new_secrets(cutoff)) {
    ldout(cct, 10) << "_check_auth_rotating have uptodate secrets (they expire after " << cutoff << ")" << dendl;
    rotating_secrets->dump_rotating();
    return 0;
  }
```

The unification affects also the debug mesages.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:04:54 +00:00
Radoslaw Zarzynski
7ea3d373b7 crimson/monc: improve debugs around Connection::renew_tickets()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:04:54 +00:00
Radoslaw Zarzynski
ccc4e33c11 crimson/monc: renew tickets and rotating keys on MAuthReply
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-07-26 12:04:39 +00:00
Aravind Ramesh
da260f3797 crimson/zns: manage zone size, capacity and device size
zone_size and zone_capacity are reported in 512B sectors by the
kernel. Update the superblock correctly with zone size/capacity
in bytes.
Currently we are using stat() to get the device size, but it
fails on block devices to get the size. Use BLKGETSIZE ioctl
to get the actual size of the zns device in sectors and convert
to bytes.

Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
2022-07-26 17:04:11 +05:30