Commit Graph

125544 Commits

Author SHA1 Message Date
Casey Bodley
b3cffa193d
Merge pull request #42222 from flipkart-incubator/aws_sse_s3
rgw: Introduce BucketEncryption APIs to support SSE-S3 feature

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-07-27 10:52:23 -04:00
Kefu Chai
587a8c9c22 doc/dev/crimson: document cyanstore and alienstore
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 22:50:15 +08:00
Sage Weil
2db3cfabc3 mgr/nfs: remove unused 'realm' arg for 'nfs export create rgw'
This argument is not used.  The rgw exports currently only work on the
default realm.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-27 10:47:55 -04:00
Sage Weil
9aeefbc666 doc/mgr/rook: update title
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-27 10:47:55 -04:00
Sage Weil
7d3443412c doc/mgr/nfs: reference customizing ingress
Link to the cephadm docs on modifying the service directly.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-27 10:47:55 -04:00
Sage Weil
b5a32d632d doc/mgr/nfs: add section for manual ganesha config; reframe
This documentation is incomplete because this mode of operation is not
tested/validated.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-27 10:47:55 -04:00
Satoru Takeuchi
25757d28cf kv/RocksDBStore: enrich debug message
It's better to print why ListColumnFamilies() failed.

Signed-off-by: Toshikuni Fukaya <toshiq2@gmail.com>
Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>
2021-07-27 14:02:46 +00:00
Will Smith
5dfda932b2 rbd: Fix mirror_image_get_status in rbd python bindings
When retrieving the status of a mirrored image from the Python rbd
library, a TypeError is raised.

*To Reproduce:*

Set up two Ceph clusters for block storage, and configure image
mirroring between their pools.  Create a least one image with mirroring
enabled, then run the following script on either cluster (once the image
exists everywhere):

```python
import rados, rbd

CONF_PATH = "YOUR-CONF-PATH"
POOL_NAME = "YOUR-POOL-NAME"
IMAGE_LABEL = "YOUR-IMAGE-LABEL"

with rados.Rados(conffile=CONF_PATH) as cluster:
  with cluster.open_ioctx(POOL_NAME) as ioctx:
    with rbd.Image(ioctx, IMAGE_LABEL) as image:
      image.mirror_image_get_status()
```

This will result in the following stack trace:

```
Traceback (most recent call last):
  File "repo-bug.py", line 10, in <module>
    image.mirror_image_get_status()
  File "rbd.pyx", line 3363, in rbd.requires_not_closed.wrapper
  File "rbd.pyx", line 5209, in rbd.Image.mirror_image_get_status
TypeError: list indices must be integers or slices, not str
```

Fixes: https://tracker.ceph.com/issues/51867

Signed-off-by: Will Smith <wsmith@linode.com>
2021-07-27 08:15:25 -04:00
wangyingbin
7aced9275f tools/rados: improve the usage message of {get,set}omapaheader
Signed-off-by: wangyingbin <ybwang0211@163.com>
2021-07-27 20:06:59 +08:00
Kefu Chai
26e7034752 common/bloom_filter: use popcount() to implement density()
instead of counting the bits using a loop, it'd be more efficient to do
this using the SIMD instruction.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
f8b3d9dcb3 common/bloom_filter: s/bits_per_char/CHAR_BIT/
use macro defined by standard libary instead of inventing our own.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
197368cfb0 include/intarith: add popcount()
for counting "1" in an integer.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
943b91a11b test/test_intarith: adapt to the change in intarith.h
back in 2623fec1cd, the vaiants of, for
instance, ctz() are consolidated to a single template. so the
ctz<>() dispatches by the size of argument after that change.
but the tests were not updated accordingly.

in this change:

* the tests are updated to use the template.
* instead of using integer literal postfix, use the macros like
  UINT64_C to define integer constants for better portability on
  different architectures where the lengths of integer *might* be
  different from amd64. also, it's more readable than postfixes
  like ULL in this context, as we really care about the exact
  length of an integer in this context when counting the leading
  zeros.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
49bd620f75 cmake: add unittest_intarith back
this unit test was missed when we migrated from autotools to CMake.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
5d68abe194 common/bloom_filter: use vector for holding bits
to be more consistent with the upstream at https://github.com/ArashPartow/bloom/blob/master/bloom_filter.hpp
and for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
5079e56b0c common/bloom_filter: use "using" for better readability
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Kefu Chai
89770fe691 common/bloom_filter: return by return values not by input params
it'd be easier for the static analyzer (like GCC), to reason about if
a variable is initialized before being used.

this change also helps to improve the readability, and to silence the
false alarm like:

In file included from ../src/os/bluestore/BlueStore.h:42,
                 from ../src/os/bluestore/BlueStore.cc:26:
../src/common/bloom_filter.hpp: In member function 'void std::vector<_Tp, _Alloc>::_M_fill_insert(std::vector<_Tp, _Alloc>::iterator, std::vector<_Tp, _Alloc>::size_type, const value_type&) [with _Tp = bloom_filter; _Alloc = mempool::pool_allocator<mempool::mempool_bluestore_fsck, bloom_filter>]':
../src/common/bloom_filter.hpp:118:46: warning: '*((void*)(& __tmp)+8).bloom_filter::table_size_' may be used uninitialized in this function [-Wmaybe-uninitialized]
  118 |  mempool::bloom_filter::alloc_byte.deallocate(bit_table_, table_size_);
      |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 20:03:32 +08:00
Alfonso Martínez
65b75000b7 mgr/dashboard: cephadm-e2e script: improvements
Improvements and some adaptations related to the jenkins job.

Fixes: https://tracker.ceph.com/issues/51612
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
2021-07-27 10:51:18 +02:00
Kefu Chai
f0027c05c6
Merge pull request #42457 from cyx1231st/wip-seastore-segment-metrics
crimson/seastore: implement metrics in BlockSegmentManager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-27 14:37:33 +08:00
Yingxin Cheng
3758671ee9 crimson/os/seastore/segment_manager: implement metrics in BlockSegmentManager
* data read/write effort;
* metadata write effort;
* opened/closed/released segments;
* unused bytes from closed segments;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2021-07-27 12:43:26 +08:00
jhonxue
1cafa08687 client:make sure only to update dir dist from auth mds
Fixes: https://tracker.ceph.com/issues/51857

Signed-off-by: Xue Yantao <xueyantao2114@163.com>
2021-07-27 12:18:20 +08:00
Kefu Chai
0edc573fca include/denc: always initialize local variable
this change silences following false positive warning:

In file included from ../src/include/encoding.h:41,
                  from ../src/kv/KeyValueDB.h:12,
                  from ../src/os/bluestore/bluestore_common.h:20,
                  from ../src/os/bluestore/BlueFS.cc:5:
 ../src/include/denc.h: In function ‘std::enable_if_t<(is_same_v<T, bluefs_extent_t> || is_same_v<T, const bluefs_extent_t>)> _denc_friend(T&, P&) [with T = bluefs_extent_t; P = ceph::buffer::v15_2_0::p$
 ../src/include/denc.h:639:11: warning: ‘shift’ may be used uninitialized in this function [-Wmaybe-uninitialized]
   639 |     shift += 7;
       |     ~~~~~~^~~~
 ../src/include/denc.h:613:7: note: ‘shift’ was declared here
   613 |   int shift;
       |       ^~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-07-27 11:43:30 +08:00
Kefu Chai
d7862cb607
Merge pull request #42475 from tchaikov/wip-crimson-cleanup
crimson/osd: cleanups

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-07-27 10:37:07 +08:00
Kefu Chai
03ac43a4b4
Merge pull request #42483 from Astroakanksha24/master
mgr/volumes/fs/operations/group.py: add extra blank lines

Reviewed-by: Varsha Rao <varao@redhat.com>
2021-07-27 10:32:29 +08:00
Kefu Chai
d637dcda22
Merge pull request #42485 from tchaikov/wip-strtoll
common/strtol: replace `const char*` with `std::string_view`

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-07-27 10:31:41 +08:00
Xiubo Li
a448d1c3ee qa: wait pgs to be clean before using the pools
Or in some use cases, like the mds-full tests, we will hit the
"PG_AVAILABILITY" warning.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:54:03 +08:00
Xiubo Li
3456ff2628 qa: ignore PG_RECOVERY_FULL and PG_DEGRADED for mds-full
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
999c787ac6 qa: wait more time since there have many more pgs than before
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
ba3833a622 qa: do not multiple the full ratio twice
The cluster has already multiple the full ratio before returning
the "max_avail".

Fixes: https://tracker.ceph.com/issues/50984
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
a96ee41908 qa: do not raise for kclient for _fsync test
For kclient, the write() will return -ENOSPC instead of the fsync().

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
c1cea71299 qa: use the pg autoscale mode to calcuate the pg_num
Setting the pg_num to 8 is too small that some osds maybe not covered by the
pools, some osds maybe overloaded. Remove the hardcodeing pg_num here and let
the pg autoscale mode to calculate it as needed, and at the same time set the
pg_num_min to 64 to avoid the pg_num to small.

If ec pool is used, for the test cases most datas will go to the ec pool and
the primary replicated pool will store a small amount of metadata for all the
files only, so set the target size ratio to 0.05 should be enough.

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
c7837484d9 qa: set the object_size to 1M
Set the object_size to 1MB to make the objects destributed more even
among the OSDs.

Fixes: https://tracker.ceph.com/issues/45434
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Xiubo Li
f4288f2a9b qa: move the is_full() to parent class
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2021-07-27 09:48:56 +08:00
Deepika Upadhyay
ba52429897 doc/rados/op: ceph/df show DIRTY field N/A
DIRTY field should only be available if cache tiering is in use.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:29 +05:30
Deepika Upadhyay
e80a80c7d9 test: update test for when cache tier is not available
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:29 +05:30
Deepika Upadhyay
fafebb1824 mon/PGMap: DIRTY field as N/A in df detail when cache tier not in use
'ceph df detail' reports a column for DIRTY objects under POOLS even
    though cache tiers not being used.  In replicated or EC pool all objects
    in the pool are reported as logically DIRTY as they have never been
    flushed .
    we display N/A for DIRTY objects if the pool is not a cache tier.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2021-07-27 07:06:20 +05:30
Sage Weil
f8f7b86571 Merge PR #42292 into master
* refs/pull/42292/head:
	qa/suites/upgrade: log_to_journald=false

Reviewed-by: Neha Ojha <nojha@redhat.com>
2021-07-26 19:23:26 -04:00
Sage Weil
ac63ab6125 Merge PR #42489 into master
* refs/pull/42489/head:
	qa/suites/upgrade/pacific-x/stress-split: do not avoid_pacific_features

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-07-26 19:23:03 -04:00
Sage Weil
8d9db910f7 doc/mgr/nfs: document ingress in more detail
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
0448a8d39d doc/mgr/nfs: typo
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
684c6a2dba doc/mgr/nfs: add note about incomplete ingress
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
cd089ee74e qa/suites/orch/cephadm: add rgw nfs export test
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
bae406f746 mgr/cephadm: ingress: tolerate no daemons
This doesn't normally happen, but did before the daemon inventory breakage
(see previous patches) was fixed.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
ad5fe4452d mgr/nfs: add --squash option to 'nfs export create rgw ...'
The cephfs variant already has it.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:17 -04:00
Sage Weil
7d2f7efe3f mgr/nfs: use bucket owner creds for rgw bucket export
The bucket owner can always read/write to the bucket, so use those creds
for the export.  This is less complicated than setting up a dedicated
user anyway.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
deaff0c42c mgr/cephadm: use new CEPH_IMAGE_TYPES for all daemons using ceph container image
We were using CEPH_TYPES + GATEWAY_TYPES, but that isn't really accurate.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
45737fe95a qa/tasks/python: simple task to run python code
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
8b96d9c22b doc/mgr/nfs: revisions
- clean up language
- move config hierarchy to the bottom (this is an implementation detail
that is only useful if managing ganesha externally)

Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:23:06 -04:00
Sage Weil
b73f730749 mgr/nfs/export: nicer exceptions on cap update
Signed-off-by: Sage Weil <sage@newdream.net>
2021-07-26 16:21:24 -04:00
Guillaume Abrioux
4bfa87ee55
Merge pull request #42181 from lyind/51526-fix-ceph-volume-zero-division-error
ceph-volume: lvm batch: fast_allocations(): avoid ZeroDivisionError
2021-07-26 21:58:07 +02:00