Commit Graph

127321 Commits

Author SHA1 Message Date
Kefu Chai
cae9552863
Merge pull request #31909 from rosinL/wip-fix-dpdk-cond-wait
msg/async/dpdk: exit condition waiting when DPDKStack is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-10-15 19:25:55 +08:00
Xuehan Xu
9f9b021087 crimson/os/seastore: deal with scenarios in which bufferlist of do_writev has more than IOV_MAX buffers
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 18:08:47 +08:00
Chunsong Feng
2c49202027 msg/async/dpdk: exit condition waiting when DPDKStack is destructed
exit() will call pthread_cond_destroy attempting to destroy dpdk::eal::cond
upon which other threads are currently blocked results in undefine
behavior. Link different libc version test, libc-2.17 can exit,
libc-2.27 will deadlock, the call stack is as follows:

Thread 3 (Thread 0xffff7e5749f0 (LWP 62213)):
 #0  0x0000ffff7f3c422c in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0xaaaadc0e30f4 <dpdk::eal::cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
 #1  __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>, cond=0xaaaadc0e30c8 <dpdk::eal::cond>)
    at pthread_cond_wait.c:502
 #2  __pthread_cond_wait (cond=0xaaaadc0e30c8 <dpdk::eal::cond>, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>)
    at pthread_cond_wait.c:655
 #3  0x0000ffff7f1f1f80 in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
   from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
 #4  0x0000aaaad37f5078 in dpdk::eal::<lambda()>::operator()(void) const (__closure=<optimized out>, __closure=<optimized out>)
    at ./src/msg/async/dpdk/dpdk_rte.cc:136
 #5  0x0000ffff7f1f7ed4 in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
 #6  0x0000ffff7f3be088 in start_thread (arg=0xffffe73e197f) at pthread_create.c:463
 #7  0x0000ffff7efc74ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffff7ee3b010 (LWP 62200)):
 #0  0x0000ffff7f3c3c38 in futex_wait (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:61
 #1  futex_wait_simple (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>)
    at ../sysdeps/nptl/futex-internal.h:135
 #2  __pthread_cond_destroy (cond=0xaaaadc0e30c8 <dpdk::eal::cond>) at pthread_cond_destroy.c:54
 #3  0x0000ffff7ef2be34 in __run_exit_handlers (status=-6, listp=0xffff7f04a5a0 <__exit_funcs>, run_list_atexit=255,
    run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
 #4  0x0000ffff7ef2bf6c in __GI_exit (status=<optimized out>) at exit.c:139
 #5  0x0000ffff7ef176e4 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:344
 #6  0x0000aaaad2939db0 in _start () at ./src/include/buffer.h:642

Fixes: https://tracker.ceph.com/issues/42890
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
2021-10-15 17:17:52 +08:00
Kefu Chai
40bc172c49
Merge pull request #43555 from tchaikov/wip-dashboard-tsconfig
mgr/dashboard: s/hosts.clickHostTab/hosts.clickTab/

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
2021-10-15 16:07:36 +08:00
Kefu Chai
a6bb0e9771 mgr/dashboard: s/hosts.clickHostTab/hosts.clickTab/
this change is a follow-up fix of
3276147049

otherwise we have following test failure:

[lint:tsc     ] cypress/integration/orchestrator/01-hosts.e2e-spec.ts(29,13): error TS2339: Property 'clickHostTab' does not exist on type 'HostsPageHelper'.

also change "host" to "hostname" to be more consistent

Fixes: https://tracker.ceph.com/issues/52645
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-15 14:46:40 +08:00
Xuehan Xu
b93d748d29 common: add prepare_iovs to bufferlist
Enable bufferlist that have more than IOV_MAX buffers to build a vector of iov.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 14:28:04 +08:00
Xuehan Xu
055dfa8d39 crimson/common: extract parallel_for_each into errorator-loop.h
otherwise, crimson::errorator<AllowedErrors...>::parallel_for_each would be
referencing incomplete crimson::parallel_for_each_state type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2021-10-15 14:28:00 +08:00
Feng Hualong
edde516a26 rgw: osd compression bypass after rgw compression
In particular, when rgw is itself doing compression,
rgw should send a incompressible hint to OSD.
Because the OSD has little effect on the data
compressed in the condition. So there bypassing the
osd compression is to avoid repeated compression
calculation.

Signed-off-by: Feng Hualong <hualong.feng@intel.com>
2021-10-15 14:20:17 +08:00
J. Eric Ivancich
6517d942c7 rgw: allow force-check filter to pass through the SAL layer
The zipper updates did not pass a force-check filter from
RGWRados::Bucket::List through to
RGWRados::cls_bucket_list_ordered. This filter is necessary for the
"radosgw-admin bucket check --fix..." functionality.

Declares type RGWBucketListNameFilter to encapsualte the filter
type. Renames some fields so the two filters can be distinguished.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2021-10-15 01:44:42 -04:00
Kefu Chai
4cb57b7539
Merge pull request #43554 from athanatos/sjust/wip-block-segment-manager-sync
crimson/os/seastore/segment_manager/block: open with dsync

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-10-15 10:09:52 +08:00
Josh Durgin
5d80867f2e doc/governance.rst: update clt membership
This now matches the mailing list which Sage audited.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2021-10-14 21:30:26 -04:00
Sage Weil
ec4fbe918b Merge PR #43333 into master
* refs/pull/43333/head:
	.github/CODEOWNERS: tag rook devs on rook related PRs

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 21:00:13 -04:00
Sage Weil
03ba3ceb5d mgr/nfs: add some test coverage for cluster info, cluster config
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-14 20:57:41 -04:00
Sage Weil
522c184f42 mgr/nfs: add 'nfs cluster config get'
Fixes: https://tracker.ceph.com/issues/52942
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-14 20:57:38 -04:00
Samuel Just
d741334111 crimson/os/seastore/segment_manager/block: open with dsync
67efc4 appears to be simply incorrect, I don't see any calls
to flush(), so we do need to open with dsync until we
implement a smarter flushing scheme.

Also, refactor open_device to remove mode param -- we always
pass the same value.

Reverts: 67efc48332
Signed-off-by: Samuel Just <sjust@redhat.com>
2021-10-14 14:54:11 -07:00
Casey Bodley
51b2b76408
Merge pull request #43477 from anthonyeleven/anthonyeleven/fix-23027
ceph: document rgw_multipart_min_part_size

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-10-14 16:13:28 -04:00
Adam King
3205a835f6 cephadm: do not log commands sent to "call" function
They could potentially log sensitive info such as registry
login passwords

Signed-off-by: Adam King <adking@redhat.com>
2021-10-14 14:56:51 -04:00
Sage Weil
e4e9477dd9 qa/suites/orch/rook/smoke: use nvme-loop
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-14 12:58:38 -05:00
Sage Weil
1020192170 mgr/rook: fix mds service list
Signed-off-by: Sage Weil <sage@newdream.net>
2021-10-14 11:06:24 -05:00
Sebastian Wagner
c19250ce3c
Merge pull request #42997 from josephsawaya/wip-mgr-rook-orch-ls
mgr/rook: orch ls displays info about OSDs

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
2021-10-14 17:33:26 +02:00
Sebastian Wagner
4e28b163a2
Merge pull request #43332 from sebastian-philipp/sphinx-rgw-spec
doc/cephadm: use sphinx autoclass to document RGWSpec

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-10-14 16:54:18 +02:00
Kefu Chai
70b049ffdb
Merge pull request #43239 from trociny/wip-48959
osd: handle inconsistent hash info during backfill and deep scrub gracefully

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-10-14 22:43:16 +08:00
Kefu Chai
7aefc0ff77
Merge pull request #43498 from dheart-joe/bluestore_fix
os/bluestore: _do_write_small fix head_pad

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2021-10-14 22:41:52 +08:00
Kefu Chai
44e89c613c
Merge pull request #42099 from aclamk/wip-bluefs-fine-grain-locking-2
os/bluestore: BlueFS fine grain locking

Reviewed-by: Sage Weil <sage@newdream.net>
2021-10-14 22:40:20 +08:00
Kefu Chai
552ac66577
Merge pull request #43374 from ljflores/wip-remove-pool-name
mgr/telemetry: remove pool name from telemetry perf report

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
2021-10-14 22:39:14 +08:00
Kefu Chai
b5d2548ceb
Merge pull request #43463 from Zhiwei-Dai/wip-enhance-qa-python3-compatibility
qa/tasks: replace iterkeys() with keys() for Python 3

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 22:38:39 +08:00
Kefu Chai
94a9757ff8
Merge pull request #43405 from ljflores/wip-perfcounter-priorities
os/bluestore: update perf counter priorities

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-10-14 22:37:36 +08:00
Kefu Chai
cb40e8fc65
Merge pull request #43422 from ronen-fr/wip-rf-exact-time-sec
common/ceph_time: fix wrong seconds output in exact_timespan_str()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 22:36:30 +08:00
Kefu Chai
84adba7588
Merge pull request #43515 from fengchunsong/dpdk-dev
msgs/async/dpdk: fix dpdk build error

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 22:34:56 +08:00
Kefu Chai
9fb3ded7b5
Merge pull request #43531 from tchaikov/wip-pick-addr
common/pick_address: refactor pick_addresses()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-10-14 22:33:39 +08:00
Mark Kogan
551e0c8f38 rgw: under fips, set flag to allow md5 in select rgw ops - for review
the ovrrides for review and relevant md5 usage:
```
src/rgw/rgw_crypt.cc:975:      unsigned char key_hash_res[CEPH_CRYPTO_MD5_DIGESTSIZE];
    int rgw_s3_prepare_encrypt(...)
        crypt_http_responses["x-amz-server-side-encryption-customer-algorithm"] = "AES256";
        crypt_http_responses["x-amz-server-side-encryption-customer-key-MD5"] = std::string(keymd5);
                                                           ~~~~~~~~~~~~~~~~

src/rgw/rgw_crypt.cc:1225:    uint8_t key_hash_res[CEPH_CRYPTO_MD5_DIGESTSIZE];
    int rgw_s3_prepare_decrypt(...)
        crypt_http_responses["x-amz-server-side-encryption-customer-algorithm"] = "AES256";
        crypt_http_responses["x-amz-server-side-encryption-customer-key-MD5"] = keymd5;
                                                           ~~~~~~~~~~~~~~~~

src/rgw/rgw_keystone.cc:40:  unsigned char m[CEPH_CRYPTO_MD5_DIGESTSIZE];
        void TokenCache::add_admin(...)
	  rgw_get_token_id(token.token.id, admin_token_id);
	                                   ~~~~~~~~~~~~~~ md5
	  add_locked(admin_token_id, token);

        void TokenCache::add_barbican(...)
	  rgw_get_token_id(token.token.id, barbican_token_id);
	                                   ~~~~~~~~~~~~~~~~~ md5
	  add_locked(barbican_token_id, token);
```

Signed-off-by: Mark Kogan <mkogan@redhat.com>
2021-10-14 14:32:31 +00:00
Sebastian Wagner
a3fc496553
Merge pull request #43073 from strenuous-life/wip-cephadm-osd-rm-status
mgr/cephadm: the display of 'orch osd rm status' is incorrect

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 16:29:05 +02:00
Sebastian Wagner
d4783f5a65
Merge pull request #43214 from batrick/i52654
pybind/mgr/cephadm: set allow_standby_replay during CephFS upgrade

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 16:28:30 +02:00
Sebastian Wagner
64c6fa2d84
Merge pull request #43253 from guits/fix_orch_ls_osd
cephadm: fix osd size reported in 'orch ls'

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-10-14 16:28:03 +02:00
Sebastian Wagner
9e68e27ce6
Merge pull request #43339 from adk3798/volume-chunk
mgr/cephadm: use json-pretty for ceph-volume calls

Reviewed-by: Sage Weil <sage@newdream.net>
2021-10-14 16:27:10 +02:00
Mark Kogan
a5df0cfcbe rgw: under fips, set flag to allow md5 in select rgw ops
Fixes: https://tracker.ceph.com/issues/52900

Signed-off-by: Mark Kogan <mkogan@redhat.com>
2021-10-14 14:19:14 +00:00
Patrick Donnelly
6e589e5080
Merge PR #43517 into master
* refs/pull/43517/head:
	doc / cephfs: health message codes should be permalinks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-10-14 10:08:18 -04:00
Kefu Chai
a948322275
Merge pull request #43538 from rzarzynski/wip-crimson-net-actual-bind-failure
crimson/osd: print the actual reason on bind failure.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 22:00:27 +08:00
Ernesto Puerta
90bbcab09f
Merge pull request #42557 from ceph/feature-50336-cluster-creation-wizard
mgr/dashboard: Cluster Creation/Expansion Wizard

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>
2021-10-14 15:12:42 +02:00
Ernesto Puerta
9f2fd685fa
Merge pull request #43453 from rhcs-dashboard/e2e-host-fix-teuth
mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-10-14 13:26:44 +02:00
Chunsong Feng
f696c365a2 src/msg/dpdk: reserve funcs capacity to avoid reallocation
When a new vector is added larger than then current vector capacity,
it reallocates space. lamda function accesses the previous adress will
cause a segment fault. Therefore, reserve sufficient funcs space to
avoid reallocation.

Fixes: https://tracker.ceph.com/issues/52907
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
2021-10-14 09:55:05 +00:00
Chunsong Feng
86ec66a9b5 msgs/async/dpdk: fix dpdk build error
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
2021-10-14 09:55:00 +00:00
Radoslaw Zarzynski
bba537988b crimson/osd: print the actual reason on bind failure.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-10-14 09:53:48 +00:00
Yuval Lifshitz
d009895a6b
Merge pull request #43308 from TRYTOBE8TME/wip-rgw-post-object-upload
src/rgw: HTTP POST object upload
2021-10-14 10:41:15 +03:00
haoyixing
55eae0bd99 rbd: correct image_spec() output when rbd-nbd Detach/Unmap error
Signed-off-by: haoyixing <haoyixing@kuaishou.com>
2021-10-14 15:08:47 +08:00
Kefu Chai
f2477d97c2 cmake: only pass --verbose when VERBOSE env variable is set
so that the setuptools is less chatty by default when building python
bindings.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 14:39:57 +08:00
Kefu Chai
45d9eff5e4 common/pick_address: refactor pick_addresses()
* consolidate the logic handling CEPH_PICK_ADDRESS_PREFER_IPV4 using
  std::sort(). this might be overkill. but it helps to explain
  what CEPH_PICK_ADDRESS_PREFER_IPV4 is for, and helps to dedup
  the code to order the addresses.
* let fill_in_one_address() return an optional<entity_addrvec_t>.
  more readable this way
* early return if the required address is not found, instead of
  checking variables like ipv4_r
* rename fill_in_one_address() to get_one_address() to reflect
  the change of the function's return value's type

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 05:38:27 +00:00
Venky Shankar
3d97d6d98f doc / cephfs: health message codes should be permalinks
... so that such links can be included in alert warnings.

Additionally, document some other health warnings. Credit to @pcuzner
to point out that not all health warnings have been documented.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
2021-10-14 10:21:07 +05:30
Kefu Chai
8661ba1807 common/pick_address: use string_view when appropriate
we don't need a string when string_view is good enough.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2021-10-14 01:23:00 +00:00
adk3798
c7419c8c43
Merge pull request #43526 from adk3798/disable-agent-default
mgr/cephadm: disable agent by default

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-10-13 17:45:42 -04:00