Commit Graph

130229 Commits

Author SHA1 Message Date
chunmei-liu
f4e228a023 crimson: add CEPH_OSD_OP_ASSERT_VER
Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-03-14 19:42:42 -07:00
Kefu Chai
035b55b634
Merge pull request #45378 from tchaikov/wip-docs
doc/dev: s/tchaikov/ceph/

Reviewed-by:Anthony D'Atri <anthony.datri@gmail.com>
2022-03-15 09:37:47 +08:00
Daniel Gryniewicz
3168d1b537
Merge pull request #45343 from dang/wip-dang-bucket-quota
RGW - Get quota on OPs with a bucket

Reviewed-by Casey Bodley <cbodley@redhat.com>
2022-03-14 13:36:02 -04:00
Yuri Weinstein
1572be9b8f qa/tests: added client upgrade suites to the crontab
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2022-03-14 08:28:37 -07:00
Kefu Chai
f63fa1b1e4 doc/dev: s/tchaikov/ceph/
use the repo under ceph orgnization for the sake of better
consistency

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-03-14 23:05:54 +08:00
Casey
e8460cbd5a deb: add build profile for system arrow
Signed-off-by: Casey <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey
223c5e8dc0 ceph.spec.in: add system_arrow and system_utf8proc conditions
Signed-off-by: Casey <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey
5da406a4ee cmake: move Arrow targets into find_package modules
Signed-off-by: Casey <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey
433782dbd5 cmake: use arrow's find_package modules
Signed-off-by: Casey <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey
ed60aeed0b cmake: add WITH_SYSTEM_ARROW to skip submodule build
relies on a hack to find the installed ParquetConfig.cmake

Signed-off-by: Casey <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey Bodley
2d80f0cd25 ceph.spec.in: seastar drops _FORTIFY_SOURCE from CFLAGS also
the arrow submodule builds some C sources that trip up on _FORTIFY_SOURCE in debug builds

[ 79%] Building C object src/arrow/CMakeFiles/arrow_objlib.dir/vendored/musl/strptime.c.o
In file included from /usr/include/time.h:25,
                 from /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10531-gc73e1fda/rpm/el8/BUILD/ceph-17.0.0-10531-gc73e1fda/src/arrow/cpp/src/arrow/vendored/strptime.h:20,
                 from /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-10531-gc73e1fda/rpm/el8/BUILD/ceph-17.0.0-10531-gc73e1fda/src/arrow/cpp/src/arrow/vendored/musl/strptime.c:4:
/usr/include/features.h:381:4: error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Werror=cpp]
  381 | #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
      |    ^~~~~~~
cc1: all warnings being treated as errors
make[5]: *** [src/arrow/CMakeFiles/arrow_objlib.dir/build.make:2543: src/arrow/CMakeFiles/arrow_objlib.dir/vendored/musl/strptime.c.o] Error 1

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey Bodley
b10364dc21 cmake: add submodule for utf8proc at v2.2.0
adds utf8proc submodule, needed by the arrow submodule in centos. add a
WITH_SYSTEM_UTF8PROC option that controls whether or not utf8proc is
built from submodule

non-system utf8proc is built as a static library to avoid conflicts with
system-provided libraries

ceph.spec.in sets WITH_SYSTEM_UTF8PROC=OFF until it's available in
centos

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Casey Bodley
2ca6d75521 cmake: add submodule for Apache Arrow at v6.0.1
adds an arrow submodule. when WITH_RADOSGW_SELECT_PARQUET is enabled,
the submodule is built as an external project and rgw links against its
imported Arrow::Parquet target

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-03-14 10:19:02 -04:00
Ilya Dryomov
0b44878cb2
Merge pull request #45235 from tchaikov/rbd-std-variant
librbd: s/boost::variant/std::variant/

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-03-14 11:36:39 +01:00
Ramana Raja
3c22f6eb71 cephfs/test_nfs: fix _write_to_read_only_export()
... to fail the test if write to read only export succeeded.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2022-03-13 20:14:50 -04:00
Ramana Raja
4df294b7fa mgr/nfs: Don't notify ganesha rados object
... to update export if the ganesha server is going to be
restarted. The ganesha server restart will reload the exports.

Signed-off-by: Ramana Raja <rraja@redhat.com>
2022-03-13 20:14:50 -04:00
Ramana Raja
2415e03217 mgr/nfs: allow dynamic update of cephfs nfs export
mgr/nfs module's apply_export() method is used to update an existing
CephFS NFS export. The method always restarted the ganesha service (
ganesha server cluster) after updating the export object and notifying
the ganesha servers to reload their exports. The restart temporarily
affected the clients connections of all the exports served by the
ganesha servers.

It is not always necessary to restart the ganesha servers. Only
updating the export ID, path, or FSAL block of a CephFS NFS export
requires a restart. So modify apply_export() to only restart the
ganesha servers for such export updates.

The mgr/nfs module creates a FSAL ceph user with read-only or
read-write path restricted MDS caps for each export. To change the
access type of the CephFS NFS export, the MDS caps of the export's FSAL
ceph user must also be changed. Ganesha can dynamically enforce an
export's access type changes, but Ceph server daemons can't dynamically
enforce changes in caps of the Ceph clients. To allow dynamic updates
of CephFS NFS exports, always create a FSAL Ceph user with read-write
path restricted MDS caps per export. Rely on the ganesha servers to
enforce the export access type changes for the NFS clients.

Fixes: https://tracker.ceph.com/issues/54025
Signed-off-by: Ramana Raja <rraja@redhat.com>
2022-03-13 20:14:50 -04:00
Yuri Weinstein
ad7a527c34
Merge pull request #44827 from ceph/wip-yuriw-crontab-master2
qa/tests: added quincy and removed master runs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam King adking@redhat.com
2022-03-13 09:10:27 -07:00
Kefu Chai
974811e819
Merge pull request #45276 from tchaikov/wip-rapidjson
submodule: bump up RapidJSON

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-03-13 20:38:35 +08:00
Xuehan Xu
8480382f09 crimson/os/seastore/btree: move 'with_btree*' from lba manager to fixed kv btree
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-03-13 17:16:53 +08:00
Xuehan Xu
917f45ab89 crimson/os/seastore: extract lba nodes out of lba manager
Change lba nodes to general fixed-size-kv btree nodes

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-03-13 17:16:53 +08:00
Xuehan Xu
bd307d2c6d crimson/os/seastore: extract fixed kv btree implementation out of lba manager
Basically, this pr moves the current LBABtree and lba_range_pin out of lba manager,
and rename LBABtree to FixedKVBtree. This is the preparation for implementing backrefs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-03-13 17:16:53 +08:00
Satoru Takeuchi
729a5b85a6 osd: make osd_fast_shutdown_notify_mon option true by default
osd_fast_shutdown_notify_mon option is false by default. So users suffer
from error log flood, slow ops, and the long I/O timeouts on voluntary OS
shutdown before they are aware of the existence of this option. Let's
make this option true by default.

Fixes: https://tracker.ceph.com/issues/53328

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2022-03-13 07:18:02 +00:00
Matan Breizman
ec7cec6a14 vstart.sh: disable autoscaling for crimson-osd
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
2022-03-12 18:08:17 +00:00
Joseph Sawaya
882fc27718 doc: Add note to osds_per_device description about dual-actuator devices
This commit adds information about using dual-actuator devices with the
osds_per_device drive group option, letting users know they can create
an OSD for each actuator by setting this value to 2 in the drive group
they're using to apply OSDs to the device.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
2022-03-11 15:45:16 -05:00
Ernesto Puerta
206dd9d4a7
mgr/dashboard: fix wrong pg status processing
Fixes: https://tracker.ceph.com/issues/54481
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
2022-03-11 17:37:38 +01:00
Kefu Chai
3a93b40721 librbd: s/boost::variant/std::variant/
boost::variant explicitly prevent it from being compared other types
by marking the return type of, for instance operator==(const T&) as
"void", if T is not identical to the variant type. so address this,
let's use std::variant<> instead.

more standard compliant and simpler this way.

because of https://cplusplus.github.io/LWG/issue3052, libstdc++
only specializes for std::variant<> for std::visit(), while libc++
allows the derived types of std::variant<> to be "visited", hence
we add an adaptor for SnapshotNamespace.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-03-11 20:46:39 +08:00
Aishwarya Mathuria
be9f8a15cd osd/scrub: add scrub duration to pg stats
Addition of a SCRUB_DURATION field that shows how long the scrub/deep-scrub of a pg took.
This field will be displayed in the output of the "ceph pg dump --format=json" and "ceph pg ls-by-pool --format=json" commands.

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
2022-03-11 12:25:27 +05:30
Zack Cerza
e39304dcd6
Merge pull request #45034 from ceph/cephadm-conditional-hostname
qa/tasks/cephadm: Make hostname change idempotent
2022-03-10 19:07:17 -07:00
chunmei-liu
ef166f154a crimson/seastore: fix 4 bytes alignment
runtime error: reference binding to misaligned address 0x610000008395
for type 'device_type_t', which requires 4 byte alignment

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-03-10 16:07:20 -08:00
Yuri Weinstein
21928dd489
Merge pull request #44925 from JoshSalomon/balancer-perf
balancer: Improve performance in rare cases

Reviewed-by: Laura Flores <lflores@redhat.com>
2022-03-10 15:24:47 -08:00
Casey Bodley
a350888bf9 test/cls/rgw: test dir_suggest after successful completion
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-03-10 17:04:10 -05:00
Casey Bodley
aa381b6765 cls/rgw: rgw_dir_suggest_changes detects race with completion
if bucket listing races with a pending index transaction, its suggested
removal may be mistakenly applied if that index transaction completes
before the osd receives this suggestion

in `rgw_dir_suggest_changes()`, the sole condition for applying a
suggested change is that the `cur_disk.pending_map` is empty. this is
true after rgw_bucket_complete_op()

on index completion, `rgw_bucket_dir_entry::index_ver` is updated to match
the new value of `rgw_bucket_dir_header::ver`. because most of `struct
rgw_bucket_dir_entry` makes the round trip through bucket listing ->
dir_suggest, we have access to the index_ver of the suggested entry. by
comparing this against the stored entry, we can ignore any suggestions
that were sent before the most recent completion

Fixes: https://tracker.ceph.com/issues/54528

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2022-03-10 17:04:01 -05:00
chunmei-liu
260b15ab05 crimson/seastore: change seastore_device_size
to be half of the original value.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-03-10 13:46:59 -08:00
Daniel Gryniewicz
8966edd2fb RGW - Get quota on OPs with a bucket
Bucket head OPs should have quota in the output.  However, we were only
fetching quota on OPs that also had an object.  The object itself is not
necessary for quota (although a bucket is).  Change it so that we get
quota on bucket OPs as well.

Fixes https://tracker.ceph.com/issues/54488

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2022-03-10 15:01:38 -05:00
Kamoltat Sirivadhna
5bfc71e85d
Merge pull request #45200 from kamoltat/wip-ksirivad-pg-num-max-reorder
osd/osd_types: pg_num_max reordering
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2022-03-10 12:01:27 -05:00
Nizamudeen A
79f8545deb mgr/dashboard: BDD approach for the dashboard cephadm e2e
Files under the directory cypress/integration/common/* will contain
common specs which can be used on all the .feature files. We can change
the common directory to cypress/integration/* from the package.json, but
if we do that now then we'll need to take care of all the absolute
import in that path. So for now at least that's not a good choice.

The bug in the cypress-browserify-preprocessor which doesn't allow to
take our tsconfig.json file forced me to go with relative imports rather
than the absolute import. We'll need to wait for this to be fixed before
 changing all our tests to BDD.

Fixes: https://tracker.ceph.com/issues/54224
Signed-off-by: Nizamudeen A <nia@redhat.com>
2022-03-10 15:33:16 +05:30
Anthony D'Atri
49c4be1174
Merge pull request #45330 from wxypro/fix-typos
doc/cephfs: fix typos
2022-03-10 00:55:19 -08:00
wangxinyu
b132d6d69b doc/cephfs: fix typos
fix typos

Signed-off-by: wangxinyu <wangxinyu@inspur.com>
2022-03-10 15:44:18 +08:00
Aashish Sharma
4690238130 mgr/cephadm: fix cephadm errors
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
2022-03-10 10:35:57 +05:30
Samuel Just
40610e8fb7
Merge pull request #43474 from xxhdx1985126/wip-epm-strategy
crimson/os/seastore: implement Sprite LFS' gc and extent placement strategy to lower gc cost

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2022-03-09 18:34:00 -08:00
Neha Ojha
f436973332
Merge pull request #45244 from Matan-B/wip-matanb-revert-keyring
Revert "doc/dev: Running workunits locally"

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-03-09 15:54:21 -08:00
Neha Ojha
1187405ead
Merge pull request #45305 from Thingee/update-foundation-mem-202203
docs: Updating Foundation member list for 202203

Reviewed-by: Dan van der Ster <daniel.vanderster@cern.ch>
Reviewed-by:Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-03-09 15:19:35 -08:00
Yuri Weinstein
c6da0c55a2
Merge pull request #44913 from benhanokh/safe_shutdown_v2
OSD::Modify OSD Fast-Shutdown to work safely i.e. quiesce all activit…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2022-03-09 13:53:43 -08:00
Kamoltat Sirivadhna
36b0dc0954
Merge pull request #45318 from kamoltat/wip-fix-autoscale-doc
doc/rados/operations/placement-groups:  ...fix typo...
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by:  Neha Ojha <nojha@redhat.com>
2022-03-09 12:41:01 -05:00
David Galloway
6917edc743
Merge pull request #45272 from petrutlucian94/build_fix
Fix Windows build
2022-03-09 10:42:36 -05:00
Kamoltat
7f1c7637e2 doc/rados/operations/placement-groups: typo two 'the'
typo, should not have two 'the' next to each
other.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
2022-03-09 15:36:20 +00:00
Kamoltat Sirivadhna
791ea7e7e8
Merge pull request #45279 from kamoltat/wip-ksirivad-fix-bulk-doc
doc/rados/operations/placement-groups: fix --bulk commands
Reviewed-by: Neha Ojha <nojha@redhat.com>
2022-03-09 10:13:21 -05:00
Kefu Chai
b04655645b cls/rbd: define SnapshotNamespace's ctor using its parent
simpler this way. also this prevent the compiler from trying to
convert a random value into SnapshotNamespace just because it
has a templated constructor when it tries to lookup a candidate
of operator<<(ostream&, Random value).

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
2022-03-09 21:53:17 +08:00
Kefu Chai
4382062ba3
Merge pull request #45302 from rzarzynski/wip-crimson-fix-noenoent-no-remove
crimson/osd: fix no ENOENT when removing already removed object

Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2022-03-09 21:34:47 +08:00