Commit Graph

127529 Commits

Author SHA1 Message Date
Deepika Upadhyay
a7952949a8 qa/suites/rbd: remove baremetal based setup needed for iscsi testing
* replace ceph baremetal deployment with cephadm based deployment

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-10-18 13:21:50 +05:30
Deepika Upadhyay
8a35516294 doc: add iSCSI initiator client configuration section for cephadm
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-10-18 13:21:43 +05:30
Maya Gilad
3f5dca4977 doc added msgr on-wire-compression
Signed-off-by: Maya Gilad <ms.maya.gilad@gmail.com>
2021-10-18 10:03:30 +03:00
Kalpesh Pandya
2916f2439e src/rgw: Fix for malformed url
This PR solves: https://tracker.ceph.com/issues/52738
It is solved by making changes to rgw_url.cc
A test is also added to check it's working.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2021-10-18 12:08:51 +05:30
Patrick Donnelly
abd19aa215
qa: reduce frag split confs for dir_split counter test
Fixes: https://tracker.ceph.com/issues/52949
Fixes: a5675535ba
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-17 21:52:23 -04:00
Ilya Dryomov
db366cab01 test: fix valgrind.supp path in run-rbd-valgrind-unit-tests.sh
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-17 17:24:41 +02:00
Ilya Dryomov
b387fdeeb3 test/librados_test_stub: add IoCtx::{set,get}_pool_full_try() stubs
Because unittest_librbd links not just with librados_test_stub.a but
also with librados.so (why?), these were sneakily picked up from the
latter.  This resulted in get_pool_full_try() reading from invalid
memory and every once in a while returning true, prompting librbd to
write to invalid memory through set_pool_full_try() in create_ioctx()
and cause random crashes.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-17 17:24:41 +02:00
Ilya Dryomov
7cc7efae2a librbd: preserve CEPH_OSD_FLAG_FULL_TRY in create_ioctx()
The obvious use case is an image with a separate data pool but it could
be useful in other places too.

While at it, set_namespace() call in handle_v2_get_data_pool() is
redundant since create_ioctx() already takes care of it.

Fixes: https://tracker.ceph.com/issues/52961
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-17 17:24:41 +02:00
Mykola Golub
429ac06cbb qa/tasks/backfill_toofull: make test work when compression on
The osd backfill reservation does not take compression into account so
we need to operate with "uncompressed" bytes when calculating nearfull
ratio.

Signed-off-by: Mykola Golub <mgolub@suse.com>
2021-10-16 15:28:53 +03:00
Kefu Chai
5db11d3988
Merge pull request #43571 from tchaikov/wip-bl-prepare-iovs
common/bl: bl::prepare_iovs() related cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-16 10:57:05 +08:00
Kefu Chai
4e81734247 common/bl: mark bl::prepare_iovs() const
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-16 08:31:08 +08:00
Kefu Chai
e9678d0b67 common/bl: simplify bl::prepare_iovs()
* consolidate the code to initialize a vector<iov>
* use an iterator to track the current vector<iovec>. less
  repeating this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-16 08:29:59 +08:00
Kefu Chai
f04a6cea54 common/bl: define iov_vec_t using a dedicated struct
more readable this way

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-16 07:48:58 +08:00
Kefu Chai
573661de50 common/bl: move bl::prepare_iovs() to .cc file
to reduce the compilation time by having a smaller header file.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-16 07:35:25 +08:00
Anthony D'Atri
00b232a3a9
Merge pull request #43478 from anthonyeleven/anthonyeleven/fix-52830
common/options: document rgw_lc_debug_interval configuration option
2021-10-15 15:34:30 -07:00
Samuel Just
3067942607
Merge pull request #43537 from xxhdx1985126/wip-seastore-iov-max
crimson/os/seastore: deal with scenarios in which bufferlist of do_writev has more than IOV_MAX buffers

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-10-15 15:04:22 -07:00
Sage Weil
c2f82d7228 mgr/orchestrator: clean up 'orch device ls' output
- headings in CAPS
- DEVICE ID, not just serial or model/vendor
- drop the health, LED fields from non-wide view
- sort devices
- 'Yes' or '', for easier visual parsing

Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-15 15:32:31 -04:00
Casey Bodley
e338ae1d7a
Merge pull request #43507 from cbodley/wip-rgw-log-spam
rgw: hide some unnecessary log spam

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-10-15 14:05:05 -04:00
Casey Bodley
2729b829a8
Merge pull request #43488 from cbodley/wip-52873
radosgw-admin: 'reshard list' doesn't log ENOENT errors

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-10-15 14:04:13 -04:00
Casey Bodley
fff75ecc01
Merge pull request #43428 from cfsnyder/wip-52818-random-rgw-req-id
rgw/rgw_rados: make RGW request IDs non-deterministic

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
2021-10-15 14:03:37 -04:00
Casey Bodley
386f04fa7b
Merge pull request #42981 from mkogan1/rgw-d3n-fix-lru-exp
rgw: d3n: fix d3n cached objects on disk lru eviction entering infini…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-15 14:02:26 -04:00
Casey Bodley
9cbf7eb5f2
Merge pull request #42208 from BryceCao/wip-add-return-when-access-key-exist-in-subuser-create
rgw : add return when access key exist in subuser create

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-15 14:01:58 -04:00
Casey Bodley
1ed12cf29c
Merge pull request #40144 from hualongfeng/osd_compression_bypass
rgw: osd compression bypass after rgw compression

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-15 13:59:58 -04:00
Anthony D'Atri
02bec8fba4 common/options: document rgw_lc_debug_interval configuration option
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
2021-10-15 09:39:48 -07:00
Ilya Dryomov
0dcea098cf librbd: honor FUA op flag for write_same() in write-around cache
WriteAroundObjectDispatch::write_same() should pass op_flags through
to dispatch_io() so that it can bypass the cache if needed.

Fixes: https://tracker.ceph.com/issues/52956
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2021-10-15 18:24:58 +02:00
Kefu Chai
45862b4c2a
Merge pull request #43533 from tchaikov/wip-cmake-less-chatty
cmake: only pass --verbose when VERBOSE env variable is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-16 00:10:32 +08:00
Casey Bodley
ca8997acaf rgw: silence error message for nonexistent realm
in a normal vstart cluster without a realm or period, every command
prints this error message:

0 RGWPeriod::init failed to init realm  id  : (2) No such file or directory

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2021-10-15 11:57:15 -04:00
Sage Weil
a95531f37d Merge PR #43558 into master
* refs/pull/43558/head:
	doc/governance.rst: update clt membership

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2021-10-15 11:48:29 -04:00
Patrick Donnelly
e55381c105
Merge PR #43430 into master
* refs/pull/43430/head:
	qa: skip c-v install on pacific start

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-10-15 11:16:25 -04:00
Patrick Donnelly
5af9882f94
Merge PR #43426 into master
* refs/pull/43426/head:
	qa/cephfs: update xfstests_dev for centos stream

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-15 11:15:41 -04:00
Patrick Donnelly
a8e77365b1
Merge PR #43425 into master
* refs/pull/43425/head:
	qa: import CommandFailedError from exceptions not run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
2021-10-15 11:13:55 -04:00
Patrick Donnelly
7aea7f48ba
Merge PR #43420 into master
* refs/pull/43420/head:
	qa: skip internal metadata directory when scanning ceph debugfs directory

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-15 11:13:05 -04:00
Patrick Donnelly
a584ffab83
Merge PR #43382 into master
* refs/pull/43382/head:
	mds: skip journaling blocklisted clients when in `replay` state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-15 11:11:48 -04:00
Sage Weil
be3fd05fc5 mgr/cephadm: make upgrade ls output structured
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-15 11:09:07 -04:00
Daniel Gryniewicz
9b7a208241
Merge pull request #43120 from dang/wip-dang-user-load-attrs
RGW Zipper - load user attributes with user

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-15 11:02:18 -04:00
Sebastian Wagner
8e4f9ea46c
doc/cephadm: Hosts: Add "initial CRUSH location"
Followup on d2a9a35993

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-15 15:59:08 +02:00
Kefu Chai
cae9552863
Merge pull request #31909 from rosinL/wip-fix-dpdk-cond-wait
msg/async/dpdk: exit condition waiting when DPDKStack is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-10-15 19:25:55 +08:00
Joao Eduardo Luis
0a86107dcc
cephadm: allow pulling from insecure registries
Signed-off-by: Joao Eduardo Luis <joao@suse.com>
2021-10-15 10:10:31 +00:00
Xuehan Xu
9f9b021087 crimson/os/seastore: deal with scenarios in which bufferlist of do_writev has more than IOV_MAX buffers
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 18:08:47 +08:00
Chunsong Feng
2c49202027 msg/async/dpdk: exit condition waiting when DPDKStack is destructed
exit() will call pthread_cond_destroy attempting to destroy dpdk::eal::cond
upon which other threads are currently blocked results in undefine
behavior. Link different libc version test, libc-2.17 can exit,
libc-2.27 will deadlock, the call stack is as follows:

Thread 3 (Thread 0xffff7e5749f0 (LWP 62213)):
 #0  0x0000ffff7f3c422c in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0xaaaadc0e30f4 <dpdk::eal::cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>, cond=0xaaaadc0e30c8 <dpdk::eal::cond>)
    at pthread_cond_wait.c:502
 #2  __pthread_cond_wait (cond=0xaaaadc0e30c8 <dpdk::eal::cond>, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>)
    at pthread_cond_wait.c:655
 #3  0x0000ffff7f1f1f80 in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
   from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
 #4  0x0000aaaad37f5078 in dpdk::eal::<lambda()>::operator()(void) const (__closure=<optimized out>, __closure=<optimized out>)
    at ./src/msg/async/dpdk/dpdk_rte.cc:136
 #5  0x0000ffff7f1f7ed4 in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
 #6  0x0000ffff7f3be088 in start_thread (arg=0xffffe73e197f) at pthread_create.c:463
 #7  0x0000ffff7efc74ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffff7ee3b010 (LWP 62200)):
 #0  0x0000ffff7f3c3c38 in futex_wait (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:61
 #1  futex_wait_simple (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>)
    at ../sysdeps/nptl/futex-internal.h:135
 #2  __pthread_cond_destroy (cond=0xaaaadc0e30c8 <dpdk::eal::cond>) at pthread_cond_destroy.c:54
 #3  0x0000ffff7ef2be34 in __run_exit_handlers (status=-6, listp=0xffff7f04a5a0 <__exit_funcs>, run_list_atexit=255,
    run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
 #4  0x0000ffff7ef2bf6c in __GI_exit (status=<optimized out>) at exit.c:139
 #5  0x0000ffff7ef176e4 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:344
 #6  0x0000aaaad2939db0 in _start () at ./src/include/buffer.h:642

Fixes: https://tracker.ceph.com/issues/42890
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
2021-10-15 17:17:52 +08:00
Kefu Chai
40bc172c49
Merge pull request #43555 from tchaikov/wip-dashboard-tsconfig
mgr/dashboard: s/hosts.clickHostTab/hosts.clickTab/

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2021-10-15 16:07:36 +08:00
Kefu Chai
a6bb0e9771 mgr/dashboard: s/hosts.clickHostTab/hosts.clickTab/
this change is a follow-up fix of
3276147049

otherwise we have following test failure:

[lint:tsc     ] cypress/integration/orchestrator/01-hosts.e2e-spec.ts(29,13): error TS2339: Property 'clickHostTab' does not exist on type 'HostsPageHelper'.

also change "host" to "hostname" to be more consistent

Fixes: https://tracker.ceph.com/issues/52645
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-15 14:46:40 +08:00
Xuehan Xu
b93d748d29 common: add prepare_iovs to bufferlist
Enable bufferlist that have more than IOV_MAX buffers to build a vector of iov.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 14:28:04 +08:00
Xuehan Xu
055dfa8d39 crimson/common: extract parallel_for_each into errorator-loop.h
otherwise, crimson::errorator<AllowedErrors...>::parallel_for_each would be
referencing incomplete crimson::parallel_for_each_state type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 14:28:00 +08:00
Feng Hualong
edde516a26 rgw: osd compression bypass after rgw compression
In particular, when rgw is itself doing compression,
rgw should send a incompressible hint to OSD.
Because the OSD has little effect on the data
compressed in the condition. So there bypassing the
osd compression is to avoid repeated compression
calculation.

Signed-off-by: Feng Hualong <hualong.feng@intel.com>
2021-10-15 14:20:17 +08:00
J. Eric Ivancich
6517d942c7 rgw: allow force-check filter to pass through the SAL layer
The zipper updates did not pass a force-check filter from
RGWRados::Bucket::List through to
RGWRados::cls_bucket_list_ordered. This filter is necessary for the
"radosgw-admin bucket check --fix..." functionality.

Declares type RGWBucketListNameFilter to encapsualte the filter
type. Renames some fields so the two filters can be distinguished.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2021-10-15 01:44:42 -04:00
Kefu Chai
4cb57b7539
Merge pull request #43554 from athanatos/sjust/wip-block-segment-manager-sync
crimson/os/seastore/segment_manager/block: open with dsync

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-10-15 10:09:52 +08:00
Josh Durgin
5d80867f2e doc/governance.rst: update clt membership
This now matches the mailing list which Sage audited.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2021-10-14 21:30:26 -04:00
Paul Cuzner
b7782084ac mgr/cephadm:improve validation of orch device zap
Improve the logic in the function to :
1. check that the host exists
2. check the host isn't in maintenance
3. check the disk exists on the host
4. check the disk isn't supporting a known OSD
5. check that the disk doesn't belong to another cluster

In addition the command now returns a completion message so
you know it worked.

Fixes:  https://tracker.ceph.com/issues/52919

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2021-10-15 14:30:15 +13:00
Sage Weil
ec4fbe918b Merge PR #43333 into master
* refs/pull/43333/head:
	.github/CODEOWNERS: tag rook devs on rook related PRs

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 21:00:13 -04:00