Commit Graph

125095 Commits

Author SHA1 Message Date
Radoslaw Zarzynski
d0b8aeeea4 os: make MemStore suitable for AlienStore.
This boils down into 2 things:

1. building `MemStore` for crimson,
2. implementing the crimson-specific variant of `get_omap_values()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 13:11:16 +00:00
Sebastian Wagner
8e513c6c9b
Merge pull request #42430 from tchaikov/wip-cmake-qa-drop-mypy
cmake: drop "mypy" from tox envlist of "qa"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-21 08:58:18 +02:00
Kefu Chai
fa816c0586
Merge pull request #42418 from tchaikov/wip-crimson-nbd
crimson/tools/store_nbd: pass app.alien() down to FSDriver

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-21 14:49:02 +08:00
Kefu Chai
351e0df4a2
Merge pull request #42367 from tchaikov/wip-cmake-Og
cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG

Reviewed-by:  Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2021-07-21 11:38:22 +08:00
Kefu Chai
c9707a989d
Merge pull request #42427 from tchaikov/wip-seastore-metric-cleanup
crimson/os/seastore: trade a map with a plain array

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-07-21 11:25:40 +08:00
Kefu Chai
056065c7e6
Merge pull request #42429 from neha-ojha/wip-51638-cleanup
qa/*/test_envlibrados_for_rocksdb.sh: remove OS specific configuration

Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 11:24:40 +08:00
Kefu Chai
286e46578d cmake: drop "mypy" from tox envlist of "qa"
this change partially reverts 81305b0da9,
otherwise we have following errors:

tasks/vstart_runner.py:691: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"
tasks/vstart_runner.py:705: error: Definition of "_run_python" in base class "LocalCephFSMount" is incompatible with definition in base class "CephFSMount"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-21 11:09:20 +08:00
Kefu Chai
9e4157b39f crimson/os/seastore: trade a map with a plain array
simpler this way, as we don't need a std::map<> at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:09:41 +08:00
Kefu Chai
eabe3c5cee
Merge pull request #42261 from cyx1231st/wip-seastore-metrics-2
crimson/os/seastore: implement metrics at cache level

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:06:51 +08:00
Kefu Chai
169101e5fe
Merge pull request #42386 from liu-chunmei/seastore-metrics
crimson/seastore: add seastore operation latency metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 01:05:34 +08:00
Kefu Chai
dc1a8a8b0e
Merge pull request #41929 from sebastian-philipp/fix-qa-tox
qa: Various make check fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-21 00:36:59 +08:00
Kefu Chai
50bff2d39d
Merge pull request #42419 from dsavineau/fix_typo_radosgw_admin_max_entries
doc/man/8/radosgw-admin: fix typo on max entries

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-20 23:40:13 +08:00
Sebastian Wagner
6be59759e3
Merge pull request #42289 from mgfritch/cephadm-test-create-daemon-dirs-prom
cephadm: use pyfakefs during test_create_daemon_dirs_prometheus

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
2021-07-20 17:35:09 +02:00
Sebastian Wagner
2cd3fc4c5b
Merge pull request #42028 from mgfritch/cephadm-infer-mismatch-fsid
cephadm: infer fsid from ceph.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-20 17:34:15 +02:00
Sebastian Wagner
cb553909d3
Merge pull request #41280 from sebastian-philipp/test_cephadm-stdin
qa/workunits/test_cephadm: Also test stdin

Reviewed-by: Michael Fritch <mfritch@suse.com>
2021-07-20 17:33:20 +02:00
Dimitri Savineau
99166e1044 doc/man/8/radosgw-admin: fix typo on max entries
This fixes a typo for the --max-entries option.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2021-07-20 10:49:41 -04:00
Kefu Chai
9f70239ef0 crimson/tools/store_nbd: mark padding variable with [[maybe_unused]]
also silences -Wunused-variable warning from GCC

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 22:48:37 +08:00
Kefu Chai
8ec97762c4 crimson/tools/store_nbd: pass app.alien() down to FSDriver
this was an oversight in 8fad523d41

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 22:48:27 +08:00
Kefu Chai
c7930b0a06 cmake: use string(APPEND ...) to concat strings
less repeating this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 20:20:06 +08:00
Kefu Chai
8e742424a7 cmake: add "-Og" to CMAKE_C_FLAGS_DEBUG
without specifying -Og, -O0 is used. as per
https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -O0
>  Reduce compilation time and make debugging produce the expected
>  results. This is the default.

and

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

and Debug is the default built type if .git directory is found under
the root source directory, so by adding "-Og -g" to CMAKE_C_FLAGS_DEBUG,
developers can have better debugging experience when testing Ceph
built from a git repo. but the downside is that it might take longer
to build the tree.

this change should also enable us to link crimson on aarch64.

the same applies to Clang,

> -O0 Means “no optimization”: this level compiles the fastest and generates the most debuggable code.
> -O1 Somewhere between -O0 and -O2.
> -O2 Moderate level of optimization which enables most optimizations.
> -Og Like -O1. In future versions, this option might disable different optimizations in order to improve debuggability.

see
https://clang.llvm.org/docs/CommandGuide/clang.html#cmdoption-o0

see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101448

Fixes: https://tracker.ceph.com/issues/51441
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 20:20:06 +08:00
Casey Bodley
d373d07a9c
Merge pull request #42413 from galsalomon66/fix_s3tests_yaml
RGW:revert force-branch to be ceph-master (s3-tests)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-20 08:04:59 -04:00
Sebastian Wagner
669660e706
Merge pull request #42304 from sebastian-philipp/doc-dameon-add-note
doc/cephadm: add notes to `orch daemon add`

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
2021-07-20 12:09:33 +02:00
Sebastian Wagner
657eacfc41
Merge pull request #42372 from sebastian-philipp/doc-cephadm-rgw-ssl
doc/cephadm: Add RGW ssl

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
2021-07-20 12:09:20 +02:00
galsalomon66
7d4bc5a72c revert force-branch to be ceph-master
Signed-off-by: galsalomon66 <gal.salomon@gmail.com>
2021-07-20 12:48:00 +03:00
Yingxin Cheng
bdadcd56bb crimson/os/seastore/cache: report dirty extents usage
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:50:05 +08:00
Yingxin Cheng
3c8d12217e crimson/os/seastore/cache: report cached extents usage
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:48:27 +08:00
Yingxin Cheng
181e9110d4 crimson/os/seastore: measure read transaction efforts that are successful
In order to calculate the ratio of discarded vs successful effort from
read-only transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:48:21 +08:00
Yingxin Cheng
a981417833 crimson/os/seastore: measure transactional efforts that are discarded or committed
The efforts of a transaction include the number and bytes of its read,
mutate, retire and fresh extents, and the bytes of delta generated.

This helps to understand the following aspects:
* The ratio of discarded efforts vs committed efforts;
* The average efforts of a transaction;
* The distribution of read/mutate/delta/retire/fresh efforts;
* The memory overhead and potential disk overhead of a transaction;
* How early a transaction invalidation happens;
* The average extent length;

It is possible to extend the effort metrics to be labeled by extent
types, in case we want to distinguish and profile the efforts at the
sub-component level.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:44:16 +08:00
Yingxin Cheng
21580723aa crimson/os/seastore: assert the committing delta is not empty
It makes no sense to commit an empty delta. It is mostly an issue that
user forget to generate delta during mutation, or there are futile
copy-on-write operations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:19:44 +08:00
Yingxin Cheng
c21783d4bf crimson/os/seastore/cache: cleanup to use Cache::invalidate(t, conflicting_extent)
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:19:44 +08:00
Yingxin Cheng
6a183356e2 crimson/os/seastore/cache: count transactional cache access/hit
Labeled by source and extent-type.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-20 16:19:35 +08:00
Ronen Friedman
6381704206
Merge pull request #42396 from ronen-fr/wip-ronenf-debug-docker-1
script: fix a bug in naming the source branch in ceph-debug-docker.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-20 10:59:04 +03:00
Yuval Lifshitz
04796910f9
Merge pull request #42355 from yuvalif/wip-yuval-fix-51681
rgw/notifications: send correct size in case of delete marker creation
2021-07-20 10:55:16 +03:00
Yuval Lifshitz
e5a8e17a94
Merge pull request #42358 from yuvalif/wip-yuval-fix-51687
rgw/notification: send current time in complete multipart upload event
2021-07-20 10:54:55 +03:00
Yuval Lifshitz
f108896b5b
Merge pull request #42189 from yuvalif/wip-yuval-fix-51530
rgw/http/notifications: support content type in HTTP POST messages
2021-07-20 10:54:38 +03:00
chunmei-liu
fa656aa1f3 crimson/seastore: add seastore operation latency metrics
Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2021-07-20 00:52:42 -07:00
Kefu Chai
a5d38f0b14 README.md: recommend -Og instead of -O0
per https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

> -Og
>  Optimize debugging experience. -Og should be the optimization level of
>  choice for the standard edit-compile-debug cycle, offering a reasonable
>  level of optimization while maintaining fast compilation and a good
>  debugging experience. It is a better choice than -O0 for producing
>  debuggable code because some compiler passes that collect debug
>  information are disabled at -O0.

so instead of using -O0, for better developer experience, -Og is
recommended.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-20 14:23:59 +08:00
Kefu Chai
40148fc248
Merge pull request #42388 from tchaikov/wip-crimson-alien
crimson/osd: use app.alien() to initialize AlienStore::alien

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-20 08:08:21 +08:00
Kefu Chai
4a52f6199c
Merge pull request #42392 from tchaikov/wip-test-test_ipaddr
test/test_ipaddr: always set ifa_flags

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-20 08:07:45 +08:00
Ernesto Puerta
64dbe17fdb
Merge pull request #42188 from votdev/issue_51408_motd
mgr/dashboard: Add configurable MOTD or wall notification

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2021-07-19 19:56:50 +02:00
Yuval Lifshitz
6fb6a8e3ea
Merge pull request #42102 from Matan-B/lua-request-userId
rgw: Added request field to lua scripting
2021-07-19 18:01:27 +03:00
Yuval Lifshitz
cf0287dbd0
Merge pull request #41927 from Matan-B/rfe-51193-add-luarocks-version
rgw: add package version support to lua scripting
2021-07-19 18:00:51 +03:00
Josh Durgin
adb0454599
Merge pull request #42074 from ljflores/wip-lflores-perf-channel
mgr/telemetry: add new 'perf' channel that shares aggregated perf counter metrics of a cluster

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
2021-07-19 07:59:32 -07:00
Ronen Friedman
de0b7d9437 script: fix a bug in naming the source branch in ceph-debug-docker.sh
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2021-07-19 09:57:33 +00:00
Sebastian Wagner
aeafb8dd46
Merge pull request #42391 from kelubo/master
doc/cephadm: positions of words are interchanged

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
2021-07-19 11:07:34 +02:00
Kefu Chai
c2b9f6e75c
Merge pull request #42393 from tchaikov/wip-cmake-cleanups
cmake: require CMake v3.16 and cleanups

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-19 15:57:45 +08:00
Kefu Chai
0270f73e6e cmake: do not check policies introduced before 3.16
see

- https://cmake.org/cmake/help/latest/policy/CMP0065.html
- https://cmake.org/cmake/help/latest/policy/CMP0074.html
...
- https://cmake.org/cmake/help/latest/policy/CMP0093.html

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-19 14:43:08 +08:00
Kefu Chai
e33dfa7557 cmake: use add_compile_options() to pass compile options
add_definitions() is for adding "-D" define flags, while "-include" does
not define flags, so let's be explicit and use add_compile_options()
instead.

this change partially reverts 5fb90bb796.
back then, we were using ubuntu bionic for building windows packages.
but since we've switched to ubuntu focal, where cmake 3.16 is available,
there is no reason to stick with add_definitions() anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-19 14:43:08 +08:00
wangfei
7f476bea44 doc/cephadm: add missing "ceph"
The word ceph is missing.

Signed-off-by: "Wang,Fei" <wf.ab@126.com>
2021-07-19 13:40:17 +08:00
wangfei
0003abd49c doc/cephadm: correct a transposed word error
The positions of two words are interchanged:
scans each cluster in the host ----> scans each host in the cluster

Signed-off-by: "Wang,Fei" <wf.ab@126.com>
2021-07-19 13:35:37 +08:00