Commit Graph

127495 Commits

Author SHA1 Message Date
Radoslaw Zarzynski
3bed1087b7 crimson, monmap: add more debugs around building initial monmap.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-10-25 13:09:22 +00:00
Adam Kupczyk
867f0a249e os/bluestore: Disable compaction then no-column-b is storing allocations to bluefs file
During BlueStore umount we store current allocation state to disk, in form of bluefs file.
If RocksDB was performing compaction during capture of allocator state, it could cause corruption.
Solution is to delete db (stop RocksDB) before state capture.

Fixes: https://tracker.ceph.com/issues/52399
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2021-10-25 13:35:45 +02:00
Nizamudeen A
b4ebca28d8 mgr/dashboard: Fix for form inside form closing issue
After the angular 11 upgrade the form in form behaviour got broken when
one tries to close that "embedded" form. This commit is to fix that
behaviour.

Fixes: https://tracker.ceph.com/issues/53020
Signed-off-by: Nizamudeen A <nia@redhat.com>
2021-10-25 17:03:32 +05:30
Ernesto Puerta
cc5d532c04
Merge pull request #43642 from rhcs-dashboard/pin-version-req-lint
mgr/dashboard: pin a version for autopep8 and pyfakefs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2021-10-25 13:12:12 +02:00
Nizamudeen A
946dab4f60 mgr/dashboard: pin a version for autopep8 and pyfakefs
Fixes: https://tracker.ceph.com/issues/53024
Signed-off-by: Nizamudeen A <nia@redhat.com>
2021-10-25 14:47:36 +05:30
Samuel Just
c9ae57ca50 crimson/os/seastore/.../lba_btree_node.h: add operator== for lba_map_val_t
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-25 01:05:18 -07:00
Samuel Just
b32fbda595 crimson/os/seastore/lba_manager/btree: tolerate op_context_t without pins
Simpler to construct unit tests for the LBABTree itself without pins.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-25 01:05:18 -07:00
Samuel Just
a89353141c test/crimson/seastore/test_btree_lba_manager: factor out btree_test_base
We're going to use this to create tests for LBABtree in isolation.

Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-25 01:05:18 -07:00
Samuel Just
efcb7a30aa crimson/os/seastore/object_data_handler.cc: misc logging improvements
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-25 01:05:18 -07:00
Xuehan Xu
be4c407eff crimson/os/seastore: debug outputs
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-25 01:05:09 -07:00
Samuel Just
157ad2aab6 crimson/os/seastore/.../lba_btree: fix logging for LBABtree::handle_split
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-25 01:05:09 -07:00
Nizamudeen A
dc80f26a50 mgr/dashboard: Proper error handling in module status guard
I've recently introduced a check that verifies if the orch_backend we need for a feature matches the orch_backend of the cluster. But to verify this I need to call configOpt API and some user's don't have the permission to access it. So it'll show this Access Denied page.

Fixes: https://tracker.ceph.com/issues/53021
Signed-off-by: Nizamudeen A <nia@redhat.com>
2021-10-25 12:55:35 +05:30
Mykola Golub
ed9d0b4417
Merge pull request #43633 from ideepika/wip-rbd-full-try-test-fixups
test/librbd: harden RemoveFullTry tests

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
2021-10-24 10:44:00 +03:00
Ilya Dryomov
6278a04ac2 qa/suites/rbd: whitelist POOL_FULL due to quota for test_librbd.sh
RemoveFullTry tests fill up the pool and expect EDQUOT.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-23 22:04:58 +05:30
Ilya Dryomov
641f782ac7 test/librbd: don't run RemoveFullTry tests with persistent writeback cache
Persistent writeback cache messes with pool quota enforcement on the
OSD side and we don't always break out of the write zeroes loop:

  src/test/librbd/test_librbd.cc:2165: Failure
  Value of: off >= quota && off < size
    Actual: false
  Expected: true

Unfortunately persistent writeback cache doesn't honor FUA [1] so the
workaround for volatile caches from the previous patch doesn't apply.

[1] https://tracker.ceph.com/issues/49715

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-23 22:04:58 +05:30
Ilya Dryomov
7f5e1ce326 test/librbd: set FUA on writes in remove_full_try()
Work around write-back or write-around cache getting enabled (e.g. if
one runs the tests with rbd_cache_writethrough_until_flush disabled).

Fixes: https://tracker.ceph.com/issues/52883
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-23 22:04:58 +05:30
myoungwon oh
8865223260 seastore: add device_id_t in RBM
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-24 00:06:55 +09:00
myoungwon oh
2089a7745e seastore: add rbm alloc infos when alloc_extent() is called
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-24 00:05:08 +09:00
myoungwon oh
a65c26a5a8 seastore: free allocated extent if extent is retiered
This commit frees allocated extent  when the extent is retired

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-23 23:33:11 +09:00
myoungwon oh
b950e21573 seastore: make rbm_alloc_delta_t to use pair<paddr_t, size>
This commit makes rbm_alloc_delta_t to use std::pair based on
paddr_t in order that other modules such as cache, which does not
know about RBM, can add free information.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-23 23:09:47 +09:00
Samuel Just
e7d152fce8
Merge pull request #43629 from athanatos/sjust/wip-fix-fsdriver
crimson/tools/store_nbd/fs_driver: fix mount and mkfs to handle new mkfs signature

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-10-22 21:42:44 -07:00
Kai Kang
fe12f39721 cmake: add support for python 3.10
Signed-off-by: Kai Kang <Kai.Kang@windriver.com>
2021-10-22 14:59:28 +08:00
Mykola Golub
82c16d39c3
Merge pull request #43573 from idryomov/wip-create-ioctx-preserve-full-try
librbd: preserve CEPH_OSD_FLAG_FULL_TRY in create_ioctx()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
2021-10-22 09:49:12 +03:00
Mykola Golub
b905c50877
Merge pull request #43568 from idryomov/wip-writesame-fua
librbd: honor FUA op flag for write_same() in write-around cache

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
2021-10-22 09:48:47 +03:00
Samuel Just
2e5aeb8651 crimson/tools/store_nbd/fs_driver: fix mount and mkfs to handle new mkfs signature
Introduced: e5f7ff
Fixes: https://tracker.ceph.com/issues/53012
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-21 23:48:38 -07:00
myoungwon oh
f3c3bf8139 seastore: make public interfaces in RBM to use paddr_t
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-22 15:28:00 +09:00
myoungwon oh
187aeaec67 seastore: add blocks_per_segment to use paddr_t in RBM
Seastore exploits paddr_t to represent its address.
However. the term of "segment" in paddr_t generally be used in LFS,
not random block design---"block" is more general term of RBM.

The problem is that seasore already has been tighly coupled with padd_t,
also it does not make sense that there are two differnt address
such as paddr_t and block_paddr_t.

Therefore, to solve this problem, this commit adds blocks_per_segment, which
describes how many blocks a single segment contains to make RBM be compotible
with paddr_t

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2021-10-22 14:36:48 +09:00
J. Eric Ivancich
0deb744900
Merge pull request #42151 from chapche/wip-tracker-51466
rgw: cls_bucket_list_unordered() might return one redundent entry every time is_truncated is true

 Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
2021-10-21 22:43:24 -04:00
Sage Weil
e4306d84ad Revert "Merge pull request #42099 from aclamk/wip-bluefs-fine-grain-locking-2"
This reverts commit 44e89c613c, reversing
changes made to 552ac66577.

This is causing failures in
ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixCsumVsCompression/2
and ObjectStore/StoreTestSpecificAUSize.SpilloverFixedTest/2

Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-21 22:18:12 -04:00
Sebastian Wagner
b830c555d2 monitoring/prometheus: Add cmake integration
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-22 13:37:31 +13:00
Paul Cuzner
4750ac0d77 mgr/prometheus: add test cases and validation using tox
Focus all tests inside a tests directory, and use pytest/tox to
perform validation of the overall content. tox tests also use
promtool if available to provide rule checks and unittest runs.

In addition to these checks a validate_rules script provides the
format, and content checks against all rules - which is also
called via tox (but can be run independently too)

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-10-22 13:36:40 +13:00
Paul Cuzner
e0dfc02063 mgr/prometheus: track individual healthchecks as metrics
This patch creates a health history object maintained in
the modules kvstore.  The history and current health
checks are used to create a metric per healthcheck whilst
also providing a history feature. Two new commands are added:
ceph healthcheck history ls
ceph healthcheck history clear

In addition to the new commands, the additional metrics
have been used to update the prometheus alerts

Fixes: https://tracker.ceph.com/issues/52638

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-10-22 13:32:39 +13:00
Radoslaw Zarzynski
596318022a crimson/osd: fix network address selection for hearbeat's messengers.
Public and cluster messenger instances binds to `INADDR_ANY` and learns
their local addresses later which allows to deal with some exotic network
arrangements.

````
INFO  2021-10-21 12:18:57,740 [shard 0] osd - picked address v2:0.0.0.0:0/0
ERROR 2021-10-21 12:18:57,740 [shard 0] none - Falling back to public interface
INFO  2021-10-21 12:18:57,740 [shard 0] osd - picked address v2:0.0.0.0:0/0
DEBUG 2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6800/3159356168] do_listen: try listen v2:0.0.0.0:6800/3159356168...
DEBUG 2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6800/3159356168] do_listen: try listen v2:0.0.0.0:6800/3159356168...
INFO  2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6800/3159356168] try_bind: done
INFO  2021-10-21 12:18:57,741 [shard 0] osd -  pg_epoch 84 pg[1.c( v 84'194 (0'0,84'194] local-lis/les=82/83 n=1 ec=13/7 lis/c=82/82 les/c/f=83/83/0 sis=82) [0] r=0 lpr=0 crt=84'194 lcod 0'0 mlcod 0'0 unknown exit Initial 0.723432 0 0.000000
INFO  2021-10-21 12:18:57,741 [shard 0] osd - Exiting state: Initial, entered at 1634818737.017936, 0.0 spent on 0 events
INFO  2021-10-21 12:18:57,741 [shard 0] osd -  pg_epoch 84 pg[1.c( v 84'194 (0'0,84'194] local-lis/les=82/83 n=1 ec=13/7 lis/c=82/82 les/c/f=83/83/0 sis=82) [0] r=0 lpr=0 crt=84'194 lcod 0'0 mlcod 0'0 unknown enter Reset
INFO  2021-10-21 12:18:57,741 [shard 0] osd - Entering state: Reset
INFO  2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(cluster) v2:0.0.0.0:6800/3159356168] try_bind: done
DEBUG 2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6801/3159356168] do_listen: try listen v2:0.0.0.0:6801/3159356168...
INFO  2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6801/3159356168] try_bind: done
INFO  2021-10-21 12:18:57,741 [shard 0] ms - [osd.0(client) v2:0.0.0.0:6801/3159356168] try_bind: done
```

The two messenger instances dedicated to hearbeat should follow this policy.
Unfortunately, crimson -- in contrast to the classical OSD -- uses the public's
and cluster's addresses *learned from clients*.

```
INFO  2021-10-21 12:18:57,747 [shard 0] osd - heartbeat: start front_addrs=v2:172.17.0.1:6801/3159356168, back_addrs=v2:172.17.0.1:6800/3159356168
DEBUG 2021-10-21 12:18:57,748 [shard 0] ms - [osd.0(hb_back) v2:172.17.0.1:6800/3159356168] do_listen: try listen v2:172.17.0.1:6800/3159356168...
DEBUG 2021-10-21 12:18:57,748 [shard 0] ms - [osd.0(hb_front) v2:172.17.0.1:6800/3159356168] do_listen: try listen v2:172.17.0.1:6800/3159356168...
DEBUG 2021-10-21 12:18:57,748 [shard 0] ms - [osd.0(hb_back) v2:172.17.0.1:6801/3159356168] do_listen: try listen v2:172.17.0.1:6801/3159356168...
DEBUG 2021-10-21 12:18:57,748 [shard 0] ms - [osd.0(hb_front) v2:172.17.0.1:6801/3159356168] do_listen: try listen v2:172.17.0.1:6801/3159356168...
```

If a network interface's address is different than the one visible-to and
learned-from clients, all attempts to bind fail with `EADDRNOTAVAIL`
(usually `99` for `errno`).

```
DEBUG 2021-10-21 12:19:13,284 [shard 0] ms - [osd.0(hb_back) v2:172.17.0.1:7550/3159356168] do_listen: try listen v2:172.17.0.1:7550/3159356168...
DEBUG 2021-10-21 12:19:13,284 [shard 0] ms - [osd.0(hb_front) v2:172.17.0.1:7568/3159356168] do_listen: try listen v2:172.17.0.1:7568/3159356168...
INFO  2021-10-21 12:19:13,284 [shard 0] ms - [osd.0(hb_front) v2:172.17.0.1:7568/3159356168] was unable to bind after 3 attempts: generic:99
ERROR 2021-10-21 12:19:13,284 [shard 0] osd - heartbeat messenger bind(v2:172.17.0.1:0/3159356168): generic:99
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-10-21 17:31:17 +00:00
Mark Kogan
2d0b120983
Merge pull request #43503 from mkogan1/wip-fips-md5
rgw: under fips, set flag to allow md5 in select rgw ops
2021-10-21 17:35:53 +03:00
Patrick Donnelly
94b712bdab
Merge PR #43566 into master
* refs/pull/43566/head:
	qa: reduce frag split confs for dir_split counter test

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2021-10-21 10:26:43 -04:00
Daniel Gryniewicz
7ddbd004b7
Merge pull request #43600 from smithfarm/wip-52979
rgw/tracing: unify SO version numbers within librgw2 package

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-21 10:21:12 -04:00
zdover23
3fb19d9bb9
Merge pull request #43586 from sebastian-philipp/docs-email-contact
doc/start: documenting-ceph: Remove email to ceph-docs@redhat.com

Reviewed-by: Zac Dover <zac.dover@gmail.com>
2021-10-21 21:14:01 +10:00
Sebastian Wagner
7b1d5372cb
Merge pull request #43604 from zdover23/wip-doc-2021-10-21-cephadm-install-typo
doc/cephadm: update typo in install.rst

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
2021-10-21 13:08:38 +02:00
Sebastian Wagner
4709ff951a
Merge pull request #43616 from krafZLorG/fix-cephadm-chooseleaf
cephadm: Fix option name osd_crush_chooseleaf_type

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-21 12:53:39 +02:00
krafZLorG
0e7c16ce2c
Fix option name osd_crush_chooseleaf_type
Signed-off-by: Dmitry Kvashnin <dm.kvashnin@gmail.com>
2021-10-21 11:41:39 +03:00
Patrick Donnelly
5dd4274b1a
Merge PR #43508 into master
* refs/pull/43508/head:
	mds/FSMap: do not assert allow_standby_replay on old FSMaps

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-10-20 22:27:40 -04:00
Patrick Donnelly
655507ed4e
Merge PR #43506 into master
* refs/pull/43506/head:
	mon/MDSMonitor: avoid crash when decoding old FSMap epochs

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
2021-10-20 22:26:41 -04:00
Patrick Donnelly
c455b10a08
Merge PR #40183 into master
* refs/pull/40183/head:
	client: remove useless mds member in Fh class
	client: adjust the Fh class members order
	client: switch to use shared_ptr for mds_sessions
	client: minor cleanup to Fh related code
	client: minor cleanup to MetaRequest class
	client: minor cleanup to Client related code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
2021-10-20 22:25:58 -04:00
Patrick Donnelly
04aabf8bee
Merge PR #38752 into master
* refs/pull/38752/head:
	qa: enable dynamic debug support to kclient

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-20 22:24:32 -04:00
Sage Weil
78f8f78bf2 Merge PR #43138 into master
* refs/pull/43138/head:
	mgr/rook: implement `orch device zap` in rook orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
2021-10-20 17:50:21 -04:00
J. Eric Ivancich
0de8e4ae91
Merge pull request #43594 from ivancich/wip-bucket-fix-pool-actual
rgw: have "bucket check --fix" fix pool ids correctly

 Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-10-20 14:38:26 -04:00
J. Eric Ivancich
08b94b52db
Merge pull request #43429 from ivancich/wip-document-rgw_lc_debug_interval
rgw: document rgw_lc_debug_interval configuration option

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-10-20 13:49:20 -04:00
J. Eric Ivancich
7f6677df55 rgw: clean up a little LC logging
Some of the LC logging does not put whitespace between some values and
the thread identifier. This commit adds the whitespace.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2021-10-20 13:00:42 -04:00
J. Eric Ivancich
4c2ac79621 rgw: during "bucket check --fix" set index entry pool id correctly
The current code sets the pool id of bucket index entries to the
bucket index pool rather than the data pool. This fixes that.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2021-10-20 13:00:40 -04:00
J. Eric Ivancich
c0a93a771e
Merge pull request #43553 from ivancich/wip-bucket-fix-pool
rgw: pass "force check filter" through SAL in bucket listing

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-10-20 12:58:39 -04:00