Commit Graph

132083 Commits

Author SHA1 Message Date
Huber-ming
a892525c71 SubmittingPatches.rst: PRs should target "main"
Signed-off-by: Huber-ming <zhangsm01@inspur.com>
2022-06-13 09:38:40 +08:00
Zac Dover
4f6edb92b9 doc/start: make OSD and MDS structures parallel
This PR makes the "Ceph OSDs" and "MDSs" bullet points
parallel by naming "object storage daemon" before referring
to the (admittedly more common and colloquial, but surely
unknown to people who genuinely require a document called
'Intro') acronym "OSD".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-06-13 10:12:57 +10:00
Zac Dover
ba1a85b292 doc/start: rewrite CRUSH para
This PR supersedes https://github.com/ceph/ceph/pull/46584
and makes changes suggested by Anthony D'Atri that improve
the coherence and consistency of the paragraph that explains
the basics of the CRUSH algorithm.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-06-13 09:41:28 +10:00
Nikhilkumar Shelke
9957a036df qa: remove incorrect 'size' from output of 'snapshot info'
The 'size' shown in the output of snapshot info command relies on
rstats which is incorrect snapshot size. It tracks size of the
subvolume from the snapshot has been taken instead of the snapshot
itself. Hence having the 'size' field in the output of 'snapshot info'
doesn't make sense until the rstats is fixed.

Fixes: https://tracker.ceph.com/issues/55822
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 17:17:03 +05:30
Nikhilkumar Shelke
86f4cd3dca docs: remove incorrect 'size' from output of 'snapshot info'
The 'size' shown in the output of snapshot info command relies on
rstats which is incorrect snapshot size. It tracks size of the
subvolume from the snapshot has been taken instead of the snapshot
itself. Hence having the 'size' field in the output of 'snapshot info'
doesn't make sense until the rstats is fixed.

Fixes: https://tracker.ceph.com/issues/55822
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 17:16:49 +05:30
Nikhilkumar Shelke
9f514a0bd1 mgr/volumes: remove incorrect 'size' from output of 'snapshot info'
The 'size' shown in the output of snapshot info command relies on
rstats which is incorrect snapshot size. It tracks size of the
subvolume from the snapshot has been taken instead of the snapshot
itself. Hence having the 'size' field in the output of 'snapshot info'
doesn't make sense until the rstats is fixed.

Fixes: https://tracker.ceph.com/issues/55822
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 17:16:29 +05:30
Nikhilkumar Shelke
2b0ffbb36d docs: add details about all options used in 'ceph fs new' command
Fixes: https://tracker.ceph.com/issues/54111
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 16:25:40 +05:30
Nikhilkumar Shelke
0f4add67eb qa: verify command status if data or metadata pool already in use
Fixes: https://tracker.ceph.com/issues/54111
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 16:25:34 +05:30
Nikhilkumar Shelke
55b7e285cb mon: verify pool is already not in use by any other app or fs
pool should not be shared between multiple file system.
Hence adding check to verify pool is already not in use.

Fixes: https://tracker.ceph.com/issues/54111
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
2022-06-12 16:24:09 +05:30
Anthony D'Atri
b3cc593774 src/ceph-volume/ceph_volume/activate: Improve usage message text
Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
2022-06-11 18:23:28 -07:00
Radosław Zarzyński
a2190f901a tools: ceph-objectstore-tool is able to trim pg log dups' entries.
The main assumption is trimming just dups doesn't need any update
to the corresponding pg_info_t.

Testing:

1. cluster without the autoscaler
```
rzarz@ubulap:~/dev/ceph/build$ MON=1 MGR=1 OSD=3 MGR=1 MDS=0 ../src/vstart.sh -l -b -n -o "osd_pg_log_dups_tracked=3000000" -o "osd_pool_default_pg_autoscale_mode=off"
```

2. 8 PGs in the testing pool.
```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph osd pool create test-pool 8 8
```

3. Provisioning dups with rados bench
```
bin/rados bench -p test-pool 300 write -b 4096  --no-cleanup
...
Total time run:         300.034
Total writes made:      103413
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.34637
Stddev Bandwidth:       0.589071
Max bandwidth (MB/sec): 2.4375
Min bandwidth (MB/sec): 0.902344
Average IOPS:           344
Stddev IOPS:            150.802
Max IOPS:               624
Min IOPS:               231
Average Latency(s):     0.0464151
Stddev Latency(s):      0.0183627
Max latency(s):         0.0928424
Min latency(s):         0.0131932
```

4. Killing osd.0
```
rzarz@ubulap:~/dev/ceph/build$ kill 2572129 # pid of osd.0
```

5. Listing PGs on osd.0 and calculating number of pg log's entries and
dups:

```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph-objectstore-tool --data-path dev/osd0 --op list-pgs --pgid 2.c > osd0_pgs.txt
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
3100
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

6. Trimming dups
```
rzarz@ubulap:~/dev/ceph/build$ CEPH_ARGS="--osd_pg_log_dups_tracked 2500 --osd_pg_log_trim_max=100" bin/ceph-objectstore-tool --data-path dev/osd0 --op trim-pg-log-dups --pgid 2.7
max_dup_entries=2500 max_chunk_size=100
Removing keys dup_0000000020.00000000000000000001 - dup_0000000020.00000000000000000100
Removing keys dup_0000000020.00000000000000000101 - dup_0000000020.00000000000000000200
Removing keys dup_0000000020.00000000000000000201 - dup_0000000020.00000000000000000300
Removing keys dup_0000000020.00000000000000000301 - dup_0000000020.00000000000000000400
Removing keys dup_0000000020.00000000000000000401 - dup_0000000020.00000000000000000500
Removing keys dup_0000000020.00000000000000000501 - dup_0000000020.00000000000000000600
Finished trimming, now compacting...
Finished trimming pg log dups
```

7. Checking number of pg log's entries and dups
```
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
2500
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-06-12 00:44:29 +02:00
Radosław Zarzyński
e312733598 Revert "tools/ceph_objectstore_took: Add duplicate entry trimming"
This reverts commit 9fb7ec61ba.

Although the chunking in off-line `dups` trimming (via COT) seems
fine, the `ceph-objectstore-tool` is a client of `trim()` of
`PGLog::IndexedLog` which means than a partial revert is not
possible without extensive changes. Moreover, trimming pg log
is not enough without modifying pg_info_t accordingly which
the reverted patch lacks.

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-06-12 00:44:29 +02:00
Radosław Zarzyński
9bf0053bc9 Revert "osd/PGLog.cc: Trim duplicates by number of entries"
This reverts commit 0d253bcc09.
which is the in-OSD part of the fix for accumulation of `dup`
entries in a PG Log. Brainstorming it has brought questions
on the OSD's behaviour during an upgrade if there are tons of
dups in the log. What must be double-checked before bringing
it back is ensuring we chunk the deletions properly to not
impose OOMs / stalls in, to exemplify, RocksDB.

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
2022-06-12 00:44:29 +02:00
Anthony D'Atri
3e24921adc doc/man/8: Tweak formatting and wording in ceph.rst
Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
2022-06-11 14:21:50 -07:00
Anthony D'Atri
784f5bb9bf
Merge pull request #46200 from elacunza/doc-man-ceph-add-enable_stretch_mode
doc/man/8: Add enable_stretch_mode docs
2022-06-11 14:02:32 -07:00
Anthony D'Atri
de85ee65ed
Merge pull request #46462 from Thingee/update-foundation-mems-202205
doc: Updating Ceph Foundation members for May
2022-06-11 13:43:48 -07:00
Anthony D'Atri
5af1f5f3cc
Merge pull request #46195 from snosratiershad/fix-docs-double-dash-convertion-to-em-dash
doc: Disable double dashes "--" smartquotes conversion to en-dashes
2022-06-11 13:32:15 -07:00
Salar Nosrati-Ershad
26d44bcae8 doc: Disable double dashes "--" smartquotes conversion to en-dashes 2022-06-11 23:34:06 +04:30
Laura Flores
4c9fd0c272
Merge pull request #46604 from ljflores/wip-librados-test-fix
test/librados: modify LibRadosMiscConnectFailure.ConnectFailure to comply with new seconds unit
2022-06-10 11:56:19 -05:00
Yuri Weinstein
986146b0a6
Merge pull request #46606 from rzarzynski/wip-55982
osd: log the number of 'dups' entries in a PG Log

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
2022-06-10 09:39:24 -07:00
Ilya Dryomov
5d18e44511
Merge pull request #46600 from idryomov/wip-pmdk-wno-error
cmake: pass -Wno-error when building PMDK

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-06-10 18:28:18 +02:00
Casey Bodley
991e56eab5
Merge pull request #46617 from yuvalif/wip-yuval-fix-55978
rgw/lua: pass correct value to "toplevel" param

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-06-10 09:36:38 -04:00
Casey Bodley
ffda5e50ee
Merge pull request #44648 from Huber-ming/index_cancel
rgw: delete redundant brackets

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2022-06-10 09:36:00 -04:00
Xuehan Xu
f2b911711c crimson/os/seastore/lba_manager: correct btree node sizes
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-06-10 15:41:50 +08:00
Xuehan Xu
7f1b2514ba crimson/os/seastore/segment_cleaner: fix journal seq leak issue
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-06-10 11:31:31 +08:00
Xuehan Xu
cbaca29185 crimson/os/seastore/cache: Cache::get_(del)_backrefs_in_range return extents in half-open range
Change Cache::get_(del)_backrefs_in_range's return from extents in [start,end] to extents in
[start, end); this is because BackrefManager::get_mappings return extents in [start,end), and
Cache's related methods have to be consistent with BackrefManager::get_mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-06-10 11:26:52 +08:00
Patrick Donnelly
d0b69a3cdf
Merge PR #46421 into main
* refs/pull/46421/head:
	doc/dev: move option -R to a different section of doc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
2022-06-09 20:12:49 -04:00
Radoslaw Zarzynski
8f1c8a7309 osd: log the number of 'dups' entries in a PG Log
We really want to have the ability to know how many
entries `PGLog::IndexedLog::dups` has inside.
The current ways are either invasive (stopping an OSD)
or indirect (examination of `dump_mempools`).

The code comes from Nitzan Mordechai (part of
ede37edd79a9d5560dfb417ec176327edfc0e4a3).

Fixes: https://tracker.ceph.com/issues/55982
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-06-09 20:01:40 +00:00
yuval Lifshitz
c1b7bcb2c0 rgw/lua: pass correct value to "toplevel" param
Fixes: https://tracker.ceph.com/issues/55978

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>
2022-06-09 22:51:50 +03:00
Laura Flores
f357459e6b test/librados: modify LibRadosMiscConnectFailure.ConnectFailure to comply with new seconds unit
The unit type for `client_mount_timeout` was changed from "float" to "secs" in
983b10506d. To make this test comply with the new
seconds unit change, we need to change the value to an integer, as seconds
does not accept float values.

Fixes: https://tracker.ceph.com/issues/55971
Signed-off-by: Laura Flores <lflores@redhat.com>
2022-06-09 18:55:48 +00:00
yuval Lifshitz
0afa30f056 rgw/lua: replace int64_t with "long long int" for the background table
lua integer is "long long int" which works with GCC 11.2.1 (fedora 36)
but fails to compile on centos8 and ubuntu focal

Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>
2022-06-09 21:43:51 +03:00
yuval Lifshitz
c6065f733e rgw/lua: use constants for upvalue indexes
Signed-off-by: yuval Lifshitz <ylifshit@redhat.com>
2022-06-09 21:43:51 +03:00
Yuval Lifshitz
2a6039c2f6 rgw/lua: add atomic increment/decrement to RGW table
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2022-06-09 21:43:51 +03:00
Yuval Lifshitz
402c35cf97 rgw/lua: support multiple types in lua background table
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
2022-06-09 21:43:51 +03:00
Casey Bodley
16b73f58c6
Merge pull request #45806 from hualongfeng/qat_error_fix1
common/compressor: fix the issue that read more data

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-06-09 10:50:24 -04:00
Ilya Dryomov
03d27945a6 cmake: pass -Wno-error when building PMDK
It's hitting pacific with a nuisance -Werror=array-parameter= const
char * vs const char[37] mismatch.  Follow commit 91a616b26e ("cmake:
pass RTE_DEVEL_BUILD=n when building dpdk") and just disable -Werror.

Fixes: https://tracker.ceph.com/issues/55977
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-06-09 14:37:10 +02:00
Guillaume Abrioux
7ffea89992 ceph-volume: make is_valid() optional
There are cases where `ceph-volume` doesn't have to require
a `ceph.conf` file that has the `fsid` parameter.

See: https://github.com/rook/rook/pull/10333#pullrequestreview-994958873

Fixes: https://tracker.ceph.com/issues/55970

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2022-06-09 10:50:58 +02:00
Sarthak0702
fa8a62fb8f mgr/dashboard: stop polling when page is not visible
Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>
2022-06-09 13:24:36 +05:30
Yuval Lifshitz
98bfe85f1f
Merge pull request #45937 from faithuniterh/adding_go_examples
examples: Add Golang code showing object upload to Ceph RGW bucket 

Reviwed-by: anthonyeleven
Reviwed-by: yuvalif
2022-06-09 10:36:17 +03:00
Yuval Lifshitz
80ffccaa17
Merge pull request #45970 from faithuniterh/adding_java_examples
examples: Add Java code showing object upload to Ceph RGW bucket

Reviewed-by: anthonyeleven
Reviewed-by: yuvalif
2022-06-09 10:33:18 +03:00
Tim Serong
265d24f09e ceph.spec.in: turn jaeger off by default for SUSE distros
Building with jaeger by default pulls in opentelemetry, and
cmake/modules/BuildOpentelemetry.cmake tries to go get
https://github.com/ideepika/opentelemetry-cpp.git at build
time, which doesn't work on SUSE's build service (no internet
access at build time).  Also, since WITH_JAEGER now defaults
to ON in CMakeLists.txt, we need to flip the logic when
setting -DWITH_JAEGER.

Fixes: 644c99826d
Fixes: 7be8be6350
Signed-off-by: Tim Serong <tserong@suse.com>
2022-06-09 16:23:50 +10:00
Kefu Chai
fdb198da30
Merge pull request #46564 from liu-chunmei/crimson-return-direct
crimson/osd: return directly when obc not exists.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
2022-06-09 14:08:59 +08:00
David Galloway
7fa69401af
Merge pull request #46585 from ceph/schedule
schedule_subset.sh: Default to ceph.git
2022-06-08 18:19:15 -04:00
chunmei-liu
96aabf16aa crimson/osd: return directly when obc not exists.
if goest into do_osd_ops, will set osd_op.rval, but it
is not reuqired in some tests, this follow classic ceph do_op logic.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2022-06-08 15:03:23 -07:00
faithuniterh
7d1ec30e51 examples: Add Java code showing object upload to Ceph RGW bucket
Signed-off-by: faithuniterh <faithuniterh@tutanota.com>
2022-06-08 23:21:03 +03:00
David Galloway
c6acb44ff7 schedule_subset.sh: Default to ceph.git
https://github.com/ceph/teuthology/pull/999 never got overridden in ceph.git.  We've been using a years-old checkout of teuthology for the `teuthology` user.

With the master->main change, that checkout needed to go.  Then when trying to schedule new nightlies, teuthology-suite was defaulting to ceph-ci.git which either has very old versions of the release branches (octopus, pacific, etc.) or they don't exist at all.

Signed-off-by: David Galloway <dgallowa@redhat.com>
2022-06-08 16:13:29 -04:00
faithuniterh
1acd53e373 examples: Add Golang code showing object upload to Ceph RGW bucket
Signed-off-by: faithuniterh <faithuniterh@tutanota.com>
2022-06-08 23:02:15 +03:00
Rishabh Dave
bbe71266de doc/dev: move option -R to a different section of doc
It's incorrect to pass option "-R fail" to the teuthology-suite command
meant for triggering tests for first time. teuthology-suite command will
fail if "-R" is passed without "-r". Therefore, move this option and its
description from the section meant for triggering tests for first time
to the section dedicated to re-running of tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2022-06-09 00:55:51 +05:30
Samuel Just
fbd7c77497
Merge pull request #46526 from cyx1231st/wip-seastore-fix-cache
crimson/os/seastore/cache: fix a potential leak when replace placeholder

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
2022-06-08 12:03:12 -07:00
Casey Bodley
b985f3a2f8
Merge pull request #46172 from cbodley/wip-54405
rgw: add 'rgw_access' log subsys for frontend http access log

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
2022-06-08 14:41:20 -04:00