Commit Graph

125528 Commits

Author SHA1 Message Date
Kefu Chai
89770fe691 common/bloom_filter: return by return values not by input params
it'd be easier for the static analyzer (like GCC), to reason about if
a variable is initialized before being used.

this change also helps to improve the readability, and to silence the
false alarm like:

In file included from ../src/os/bluestore/BlueStore.h:42,
                 from ../src/os/bluestore/BlueStore.cc:26:
../src/common/bloom_filter.hpp: In member function 'void std::vector<_Tp, _Alloc>::_M_fill_insert(std::vector<_Tp, _Alloc>::iterator, std::vector<_Tp, _Alloc>::size_type, const value_type&) [with _Tp = bloom_filter; _Alloc = mempool::pool_allocator<mempool::mempool_bluestore_fsck, bloom_filter>]':
../src/common/bloom_filter.hpp:118:46: warning: '*((void*)(& __tmp)+8).bloom_filter::table_size_' may be used uninitialized in this function [-Wmaybe-uninitialized]
  118 |  mempool::bloom_filter::alloc_byte.deallocate(bit_table_, table_size_);
      |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Alfonso Martínez
65b75000b7 mgr/dashboard: cephadm-e2e script: improvements
Improvements and some adaptations related to the jenkins job.

Fixes: https://tracker.ceph.com/issues/51612
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
2021-07-27 10:51:18 +02:00
Kefu Chai
f0027c05c6
Merge pull request #42457 from cyx1231st/wip-seastore-segment-metrics
crimson/seastore: implement metrics in BlockSegmentManager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-27 14:37:33 +08:00
Yingxin Cheng
3758671ee9 crimson/os/seastore/segment_manager: implement metrics in BlockSegmentManager
* data read/write effort;
* metadata write effort;
* opened/closed/released segments;
* unused bytes from closed segments;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-27 12:43:26 +08:00
jhonxue
1cafa08687 client:make sure only to update dir dist from auth mds
Fixes: https://tracker.ceph.com/issues/51857

Signed-off-by: Xue Yantao <xueyantao2114@163.com>
2021-07-27 12:18:20 +08:00
Kefu Chai
0edc573fca include/denc: always initialize local variable
this change silences following false positive warning:

In file included from ../src/include/encoding.h:41,
                  from ../src/kv/KeyValueDB.h:12,
                  from ../src/os/bluestore/bluestore_common.h:20,
                  from ../src/os/bluestore/BlueFS.cc:5:
 ../src/include/denc.h: In function ‘std::enable_if_t<(is_same_v<T, bluefs_extent_t> || is_same_v<T, const bluefs_extent_t>)> _denc_friend(T&, P&) [with T = bluefs_extent_t; P = ceph::buffer::v15_2_0::p$
 ../src/include/denc.h:639:11: warning: ‘shift’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   639 |     shift += 7;
       |     ~~~~~~^~~~
 ../src/include/denc.h:613:7: note: ‘shift’ was declared here
   613 |   int shift;
       |       ^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 11:43:30 +08:00
Kefu Chai
d7862cb607
Merge pull request #42475 from tchaikov/wip-crimson-cleanup
crimson/osd: cleanups

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-07-27 10:37:07 +08:00
Kefu Chai
03ac43a4b4
Merge pull request #42483 from Astroakanksha24/master
mgr/volumes/fs/operations/group.py: add extra blank lines

Reviewed-by: Varsha Rao <varao@redhat.com>
2021-07-27 10:32:29 +08:00
Kefu Chai
d637dcda22
Merge pull request #42485 from tchaikov/wip-strtoll
common/strtol: replace `const char*` with `std::string_view`

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-27 10:31:41 +08:00
Xiubo Li
a448d1c3ee qa: wait pgs to be clean before using the pools
Or in some use cases, like the mds-full tests, we will hit the
"PG_AVAILABILITY" warning.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:54:03 +08:00
Xiubo Li
3456ff2628 qa: ignore PG_RECOVERY_FULL and PG_DEGRADED for mds-full
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
999c787ac6 qa: wait more time since there have many more pgs than before
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
ba3833a622 qa: do not multiple the full ratio twice
The cluster has already multiple the full ratio before returning
the "max_avail".

Fixes: https://tracker.ceph.com/issues/50984
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
a96ee41908 qa: do not raise for kclient for _fsync test
For kclient, the write() will return -ENOSPC instead of the fsync().

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
c1cea71299 qa: use the pg autoscale mode to calcuate the pg_num
Setting the pg_num to 8 is too small that some osds maybe not covered by the
pools, some osds maybe overloaded. Remove the hardcodeing pg_num here and let
the pg autoscale mode to calculate it as needed, and at the same time set the
pg_num_min to 64 to avoid the pg_num to small.

If ec pool is used, for the test cases most datas will go to the ec pool and
the primary replicated pool will store a small amount of metadata for all the
files only, so set the target size ratio to 0.05 should be enough.

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
c7837484d9 qa: set the object_size to 1M
Set the object_size to 1MB to make the objects destributed more even
among the OSDs.

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
f4288f2a9b qa: move the is_full() to parent class
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Deepika Upadhyay
ba52429897 doc/rados/op: ceph/df show DIRTY field N/A
DIRTY field should only be available if cache tiering is in use.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:29 +05:30
Deepika Upadhyay
e80a80c7d9 test: update test for when cache tier is not available
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:29 +05:30
Deepika Upadhyay
fafebb1824 mon/PGMap: DIRTY field as N/A in df detail when cache tier not in use
'ceph df detail' reports a column for DIRTY objects under POOLS even
    though cache tiers not being used.  In replicated or EC pool all objects
    in the pool are reported as logically DIRTY as they have never been
    flushed .
    we display N/A for DIRTY objects if the pool is not a cache tier.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:20 +05:30
Sage Weil
f8f7b86571 Merge PR #42292 into master
* refs/pull/42292/head:
	qa/suites/upgrade: log_to_journald=false

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-26 19:23:26 -04:00
Sage Weil
ac63ab6125 Merge PR #42489 into master
* refs/pull/42489/head:
	qa/suites/upgrade/pacific-x/stress-split: do not avoid_pacific_features

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-26 19:23:03 -04:00
Sage Weil
8d9db910f7 doc/mgr/nfs: document ingress in more detail
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
0448a8d39d doc/mgr/nfs: typo
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
684c6a2dba doc/mgr/nfs: add note about incomplete ingress
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
cd089ee74e qa/suites/orch/cephadm: add rgw nfs export test
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
bae406f746 mgr/cephadm: ingress: tolerate no daemons
This doesn't normally happen, but did before the daemon inventory breakage
(see previous patches) was fixed.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
ad5fe4452d mgr/nfs: add --squash option to 'nfs export create rgw ...'
The cephfs variant already has it.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
7d2f7efe3f mgr/nfs: use bucket owner creds for rgw bucket export
The bucket owner can always read/write to the bucket, so use those creds
for the export.  This is less complicated than setting up a dedicated
user anyway.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
deaff0c42c mgr/cephadm: use new CEPH_IMAGE_TYPES for all daemons using ceph container image
We were using CEPH_TYPES + GATEWAY_TYPES, but that isn't really accurate.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
45737fe95a qa/tasks/python: simple task to run python code
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
8b96d9c22b doc/mgr/nfs: revisions
- clean up language
- move config hierarchy to the bottom (this is an implementation detail
that is only useful if managing ganesha externally)

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
b73f730749 mgr/nfs/export: nicer exceptions on cap update
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:21:24 -04:00
Guillaume Abrioux
4bfa87ee55
Merge pull request #42181 from lyind/51526-fix-ceph-volume-zero-division-error
ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError
2021-07-26 21:58:07 +02:00
Zac Dover
d27bb21cc1 doc/rados: rewrite index.rst
This PR makes minor alterations to the
text at the beginning of the RADOS Guide.

Most notably, the monitor daemon has been
added to the list of types of daemons that
constitute a Ceph cluster.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2021-07-27 05:16:41 +10:00
Michael Fritch
d9198d8668
cephadm: ceph-volume should raise fsid mismatch
raise an fsid mismatch error when passed differing fsids via `--fsid` and `--config`:

```
self = <tests.test_cephadm.TestCephVolume object at 0x7f1c711961f0>, cephadm_fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f1c713addc0>

    def test_fsid(self, cephadm_fs):
        cv_cmd = ['--', 'inventory', '--format', 'json']
        fsid = '00000000-0000-0000-0000-0000deadbeef'

        cmd = ['ceph-volume', '--fsid', fsid] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        s = get_ceph_conf(fsid=fsid)
        f = cephadm_fs.create_file('ceph.conf', contents=s)

        cmd = ['ceph-volume', '--fsid', fsid, '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            cd.command_ceph_volume(ctx)
            assert ctx.fsid == fsid

        cmd = ['ceph-volume', '--fsid', '10000000-0000-0000-0000-0000deadbeef', '--config', f.path] + cv_cmd
        with with_cephadm_ctx(cmd) as ctx:
            err = 'fsid does not match ceph.conf'
            with pytest.raises(cd.Error, match=err):
                cd.command_ceph_volume(ctx)
>               assert ctx.fsid == None
E               AssertionError: assert '10000000-0000-0000-0000-0000deadbeef' == None
E                +  where '10000000-0000-0000-0000-0000deadbeef' = <cephadm.CephadmContext object at 0x7f1c7121c1c0>.fsid
```

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Michael Fritch
2d2bb9f96d
cephadm: add ceph-volume tests
add basic ceph-volume tests for `--fsid`, `--config`, and `--keyring`

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Michael Fritch
b589ae68fa
cephadm: remove get_parm mock
fixture does not need to patch the `get_parm` func

Signed-off-by: Michael Fritch <mfritch@suse.com>
2021-07-26 10:29:39 -06:00
Kalpesh Pandya
3be2b4bcc5 rgw/notifications: Changing the Multipart Upload notification behavior
Changing the notification behavior in case of Multipart Upload, updating
the related test cases and adding the documentation changes for the same.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>
2021-07-26 21:22:33 +05:30
Guillaume Abrioux
d69ba6ce58
Merge pull request #42443 from dsavineau/ceph_volume_tests_functional_ssh_args
ceph-volume/tests: update ansible environment variables in tox
2021-07-26 17:08:10 +02:00
Patrick Donnelly
83d252cc30 qa: fold frag confs into conf/mds.yaml
These overrides are standard for all configurations. The config to
enable fragmentation is also long removed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-07-26 07:14:38 -07:00
Kefu Chai
54c3e1b688 *: drop strict_iecstrtoll(const char *str,..)
replace strict_iecstrtoll(const char *str,..) with
strict_iecstrtoll(std::string_view, ..). which is more convenient.
and both of them share the same implementation:

strict_iec_cast<uint64_t>(str, err);

so they are interchangeable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
5d22af9316 common,mon: drop strict_sistrtoll()
strict_sistrtoll() is but an alias of strict_si_cast<uint64_t>(..),
let's just drop the former. there are way too many thin wrappers in
strtol.{h,cc}. they don't offer lots of benefit to us.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
650a0c9363 common/strtol: drop strict_strto*(const char*,..)
they are but proxies to strict_strto*(string_view, ...). and their
callers can just call the latter without any change. so drop these
proxies.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
55ed86bb98 common, mon: use strict_si_cast<>(std::string_view,..)
simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
b6f29f4c1b common/strtol: expose strict_si_cast(std::string_view, ..)
this variant is better than strict_si_cast(const char*), because:

* we can just pass std::string to it, as std::string_view can be
  constructed from a std::string implicitly
* strict_si_cast(std::string_view, ..) is the underlying
  implementation of strict_si_cast(const char*,..), so less
  indirection helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
58c4ece8f3 rgw,mon,common/strtol: use strict_iec_cast(std::string_view, ..)
this variant is better than strict_iec_cast(const char*), because:

* we can just pass std::string to it, as std::string_view can be
  constructed from a std::string implicitly
* strict_iec_cast(std::string_view, ..) is the underlying
  implementation of strict_iec_cast(const char*,..), so less
  indirection helps with the readability.
* rgw,mon: use strict_iec_cast(std::string_view, ..) instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-26 20:12:17 +08:00
Kefu Chai
f608782797
Merge pull request #42403 from BlaineEXE/ceph-volume-use-safer-check-for-has-bluestore-label
ceph-volume: use safer check for bluestore label

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
2021-07-26 18:31:13 +08:00
Kefu Chai
0dfab38c01
Merge pull request #42474 from tchaikov/wip-kv-cleanup
kv/RocksDBStore: always initialize local variable

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-26 18:29:01 +08:00
Igor Fedotov
8c49d56b93
Merge pull request #42456 from ifed01/wip-ifed-alloc-histogram2
os/bluestore: implement allocation perf histogram.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-26 13:21:28 +03:00