Commit Graph

140162 Commits

Author SHA1 Message Date
Casey Bodley
9b46b9b0a0
Merge pull request #53478 from EdwardVitor/cuiming_chinamobile
auth:rectify a cmake compilation warning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-18 14:46:55 +01:00
Patrick Donnelly
6c993ee3bb
Merge PR #53175 into main
* refs/pull/53175/head:
	qa: increase the http postBuffer size and disable sslVerify

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-18 08:37:16 -04:00
cuiming_yewu
800c148d2e auth:rectify a cmake compilation warning
rectify src/auth/cephx/CephxProtocol.cc 1 warning
with the variable 'ch' Used before initialized
auth/cephx/CephxProtocol.cc:595:57: warning: '*((void*)& ch +8)' may be used uninitialized in this function [-Wmaybe-uninitialized]
     msg.server_challenge_plus_one = ch.server_challenge + 1;
                                     ~~~~~~~~~~~~~~~~~~~~^~~

Signed-off-by: cuiming <cuiming_yewu@cmss.chinamobile.com>
2023-09-18 15:30:06 +08:00
Anthony D'Atri
245f8af8c4
Merge pull request #53490 from zdover23/wip-doc-2023-09-17-architecture-6-of-x
doc/architecture: "Edit HA Auth" (one of several)
2023-09-17 10:40:26 -04:00
Zac Dover
e50c3e1c08 doc/architecture: "Edit HA Auth" (one of several)
Edit "High Availability Authentication" in doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-17 18:56:40 +10:00
Anthony D'Atri
a75afe5215
Merge pull request #53487 from zdover23/wip-doc-2023-09-16-architecture-5-of-x
doc/architecture: Edit "HA Auth"
2023-09-16 12:46:58 -04:00
Zac Dover
4cbef54bba doc/architecture: Edit "HA Auth"
Edit "High Availability Authentication" in doc/architecture.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-16 22:27:29 +10:00
Patrick Donnelly
6c387554d8
Merge PR #52638 into main
* refs/pull/52638/head:
	mds: flush monc log before abort
	qa: check for expected cluster log message
	qa: ignore expected cluster warning from damage tests

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
2023-09-15 14:38:02 -04:00
Patrick Donnelly
01ef9e5e91
Merge PR #52199 into main
* refs/pull/52199/head:
	mds: continue linking if targeti is temporarily located in stray dir
	Revert "mds: wait unlink to finish to avoid conflict when creating same dentries"
	Revert "mds: clear the STATE_UNLINKING state when the unlink fails"
	Revert "mds: wait reintegrate to finish when unlinking"
	Revert "mds: notify the waiters in replica MDSs"
	Revert "mds: wait the linkmerge/migrate to finish after unlink"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2023-09-15 12:11:37 -04:00
Daniel Gryniewicz
a4b19133f1
Merge pull request #52933 from dang/wip-dang-posix-driver
RGW - Add POSIX Driver

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2023-09-15 08:58:52 -04:00
Adam King
980a0a4f84
Merge pull request #53296 from phlogistonjohn/jjm-fix-git-ls-files
cephadm: remove duplicates when listing files in scan

Reviewed-by: Adam King <adking@redhat.com>
2023-09-14 13:10:16 -04:00
Daniel Gryniewicz
5258bcbd73 RGW - Add POSIX Driver
This is the MVP for a driver for RGW that operates on top of a POSIX
filesystem.  It supports get, put, list, copy, multipart, external
access via the filesystem itself, and ordered bucket listings via an
LRU-based cache.

Note that this is currently a Filter, indended to run on top of dbstore.
This is because it currently doesn't have any User implementation, so it
depends on dbstore's User.  Everything else is implemented in
POSIXDriver.  Once there is a User implementation, this will become a
Store, instead of a Filter.

Commit messages from bucket listing cache:

  rgw/posixdriver: recycle lmdb database handles as required

    While LMDB workflows often do not close/return database handles,
    ours continually reuses them.  This requires us to close each
    handle (atomically) when a cache entry is recycled.

  rgw/posixdriver: don't instantiate bucket cache entries from notify events

  rgw/posixdriver: incorporate lmdb-safe for now

    The current inclusion is based on https://github.com/Martchus/lmdb-safe,
    which is actively maintained but currently has some packaging issues the
    author has agreed to accept fixes for.

    For now, skip the submodule to save time and remove an external dependency.

  rgw/posixdriver: fix listing of cached, empty bucket

    * check lmdb enumeration result in all cases and w/better style
    * add unit test for enumeration of an empty cached directory

  rgw/posixdriver: nest lmdbs in a directory under the dbroot path to avoid cleanup issues

  rgw/posixdriver: refactor for posix integration

    * Derive BucketCache types as templates on a SAL driver and SAL
      bucket pair.

    * Integrate cache fills as callbacks into SAL layer (or mock, for
      tests)

    * Renaming and cleanups

  rgw/posixdriver: add bucket cache implementation and tests

    Adds free-standing cache of buckets and object names, with
    bucket names (and listing attributes, upcoming) managed in
    a hashed set of lmdb databases, which provides ordering and
    a high-performance listing cache.

    An framework for notification on new object creation (e.g.,
    outside S3 workflow) is provided, and a Linux implementation
    using inotify.

    FindLMDB.cmake taken with attribution and license.

  rgw/posixdriver: add zpp_bits serialization (FAST)

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2023-09-14 12:09:40 -04:00
Adam King
1df11733c4
Merge pull request #52690 from phlogistonjohn/jjm-cephadm-daemon-ident
cephadm: refactor code to make use of DaemonIdentity type

Reviewed-by: Adam King <adking@redhat.com>
2023-09-14 11:23:38 -04:00
zdover23
0bf6819b12
Merge pull request #53450 from zdover23/wip-doc-2023-09-14-man-ceph-monstore-tool
doc/man: edit ceph-monstore-tool.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-14 21:44:15 +10:00
Ronen Friedman
a6419735eb
Merge pull request #53426 from ronen-fr/wip-rf-fshut
osd: cleaning stop_for_fast_shutdown()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
2023-09-14 13:33:46 +03:00
Venky Shankar
98758eee89 Merge PR #53070 into main
* refs/pull/53070/head:
	PendingReleaseNotes: add note about 'm' and 'M' periods
	doc/cephfs: add note about monthly scheduling and retention period
	qa: update minute period multiplier to m
	qa: change minute duration specifier from M to m
	mgr/snap_schedule: change minute duration specifier from M to m

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-09-14 15:39:07 +05:30
Yingxin
2ea51761cd
Merge pull request #53433 from myoungwon/fix-53232
crimson/tools/perf_staged_fltree: fix compile error

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Aravind Ramesh <aravind.ramesh@wdc.com>
2023-09-14 15:15:24 +08:00
Venky Shankar
99adb599e7 Merge PR #51891 into main
* refs/pull/51891/head:
	qa: fix cephfs-top tests for the new fields
	cephfs-top: include the missing fields in --dump output

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
2023-09-14 09:29:07 +05:30
zdover23
2089353a10
Merge pull request #53444 from zdover23/wip-doc-2023-09-14-architecture-4-of-x
doc/architecture: edit "High Avail. Monitors"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-14 13:27:52 +10:00
Zac Dover
02177f6368 doc/man: edit ceph-monstore-tool.rst
Clean up the English in doc/man/8/ceph-monstore-tool.rst. Put all
commands in the imperative mood; make sentences more direct.

Follows https://github.com/ceph/ceph/pull/52860.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-14 13:27:10 +10:00
Zac Dover
57019c3469 doc/architecture: edit "High Avail. Monitors"
Improve the sentence structure in the "High Availability Monitors"
section of doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-14 13:12:44 +10:00
Radoslaw Zarzynski
f417f38bee
Merge pull request #53178 from aclamk/wip-aclamk-bs-esb-5-enabled
bluestore: Elastic Shared Blobs 5 - enabled, final

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com
2023-09-13 21:45:47 +02:00
John Mulligan
b811205048 cephadm: fix an error running iscsi services
Signed-off-by: John Mulligan <jmulligan@redhat.com>
2023-09-13 15:44:40 -04:00
Matt Benjamin
7377eb1e05
Merge pull request #52793 from linuxbox2/wip-fix-test-librgw-xattr
rgw/file:  fix build of librgw_file_xattr.cc unit test
2023-09-13 12:57:52 -04:00
Daniel Gryniewicz
e19b3d9590
Merge pull request #51368 from dang/wip-dang-zipper-motr
Fix MOTR build, and move MOTR and DAOS to proper location
2023-09-13 09:35:39 -04:00
Ali Maredia
011bd37a36
Merge pull request #53391 from shreyanshjain7174/bucket_quota
RadosGW API: incorrect bucket quota in response to HEAD /{bucket}/?usage

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2023-09-13 07:46:19 -04:00
Myoungwon Oh
251288410c crimson/tools/perf_staged_fltree: fix compile error
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2023-09-13 18:57:06 +09:00
zdover23
2bc9a17696
Merge pull request #53423 from zdover23/wip-doc-2023-09-12-architecture-3-of-x
doc/architecture: edit "Cluster Map"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2023-09-13 17:07:26 +10:00
Venky Shankar
59d1f7655c Merge PR #53153 into main
* refs/pull/53153/head:
	Revert "mds: align quota.max_bytes to 4MB or 4KB"

Reviewed-by: Xiubo Li <xiubli@redhat.com>
2023-09-13 11:10:59 +05:30
Casey Bodley
2cbd795745
Merge pull request #53370 from wowly/rgw-cache-nginx-if-match-header
examples: fixed errors with the if-match header in the examples of rgw-cache nginx configs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2023-09-12 19:27:00 +01:00
Zac Dover
cc8dce2531 doc/architecture: edit "Cluster Map"
Edit the section "Cluster Map" in doc/architecture.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-13 03:15:38 +10:00
Anthony D'Atri
c1ddf1d2fb
Merge pull request #53422 from zdover23/wip-doc-2023-09-12-glossary-primary-affinity
doc/glossary: add "primary affinity" to glossary
2023-09-12 12:00:08 -04:00
Ronen Friedman
210dbd4ff1 osd: cleaning stop_for_fast_shutdown()
Removed unsued variables to prevent compiler warnings.
Protected the shard lock.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
2023-09-12 10:55:48 -05:00
Adam Kupczyk
58636b20e9
Merge pull request #52407 from rhcs-dashboard/osd-op-scraper
os/bluestore: osd scrape read/write operations
2023-09-12 17:47:52 +02:00
Pere Diaz Bou
f3cf1bf5bc os/bluestore: osd scrape read/write operations
To run:
```bash
../src/test/objectstore/allocsim/ops_scraper.py --ceph_bin_path bin/ceph
```

Example:
```bash
$ ../src/test/objectstore/allocsim/ops_scraper.py --ceph_bin_path bin/ceph                                                                           [13:06:39]
collecting from osd. 0
op_type | offset/extent | name | pgid | initiated_at | who
-------------------------------------------------------
write 0~4194304 4:c4b97d2c:::be151868021c187609fb97e4c22ae1f9:head 4.3 2023-07-12T13:06:34.959082+0200 client.4831.0:351
write 0~4194304 4:d392075e:::b068dce954dca8ca56d72e02e87f39c8:head 4.b 2023-07-12T13:06:35.093439+0200 client.4831.0:358
write 0~4194304 4:bc1ac643:::7b815bd25a40695c891dc06bdf2b65be:head 4.1d 2023-07-12T13:06:35.128097+0200 client.4831.0:360
write 0~4194304 4:d0d21bd5:::cc531199dd2707370bb461cce82bdef6:head 4.b 2023-07-12T13:06:36.023082+0200 client.4831.0:367
write 0~4194304 4:a9abc92c:::0703d6f43d14bc33d69f94ecb96c522c:head 4.15 2023-07-12T13:06:36.367886+0200 client.4831.0:369
write 0~4194304 4:9b4684ea:::d5856dc0ecad96f811012e0937d5e0ac:head 4.19 2023-07-12T13:06:36.654584+0200 client.4831.0:370
write 0~4194304 4:d15b8c87:::c3344e5de8c8936876965d9fb056ad89:head 4.b 2023-07-12T13:06:36.835690+0200 client.4831.0:371
write 0~4194304 4:ad41de2a:::8a1e2743b577a67d6bc8a1d514391b53:head 4.15 2023-07-12T13:06:37.102741+0200 client.4831.0:373
write 0~4194304 4:fc35d54f:::39cd961e7e33680f4bfd1ca0fe43dda4:head 4.1f 2023-07-12T13:06:38.047612+0200 client.4831.0:377

collecting from osd. 1
op_type | offset/extent | name | pgid | initiated_at | who
-------------------------------------------------------
write 0~4194304 4:4c9ed2e8:::86206ccc35d8aceaf78670a751a6b550:head 4.12 2023-07-12T13:06:34.771423+0200 client.4831.0:340
write 0~4194304 4:106d9524:::fe7534cc604b0eb50498cb23aaca9161:head 4.8 2023-07-12T13:06:35.128105+0200 client.4831.0:359
write 0~4194304 4:a6e3fdb1:::2692491bfc4c584ca5b1efcb896936c2:head 4.5 2023-07-12T13:06:35.567070+0200 client.4831.0:363
write 0~4194304 4:2d9a471a:::ba51255646d00c496c2638bad03e0869:head 4.14 2023-07-12T13:06:35.568290+0200 client.4831.0:364
write 0~4194304 4:2b1381b1:::c58fa31e5d3a4f5550ee143bcb07a49f:head 4.14 2023-07-12T13:06:35.764699+0200 client.4831.0:365
write 0~4194304 4:a22e158f:::b2116e08700da18595b96d58120fe81c:head 4.5 2023-07-12T13:06:36.023101+0200 client.4831.0:366
```

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
2023-09-12 15:51:49 +02:00
Ali Masarwa
0ecc5f1262
Merge pull request #53419 from AliMasarweh/wip-alimasa-persistent-q-test
RGW | bucket notification: adding more sleeping time to prevent test from failing

Reviewed-by: yuvalif <ylifshit@redhat.com>
2023-09-12 16:51:25 +03:00
Jos Collin
0241a82b4d
qa: fix cephfs-top tests for the new fields
Fixes: https://tracker.ceph.com/issues/61397
Signed-off-by: Jos Collin <jcollin@redhat.com>
2023-09-12 17:35:30 +05:30
Jos Collin
ce3e61eeb7
cephfs-top: include the missing fields in --dump output
Fixes: https://tracker.ceph.com/issues/61397
Signed-off-by: Jos Collin <jcollin@redhat.com>
2023-09-12 17:35:21 +05:30
Zac Dover
f3a3f10e9c doc/glossary: add "primary affinity" to glossary
Add "Primary Affinity" to the glossary and link to the information in
the RADOS documentation that (more or less) explains it.

Signed-off-by: Zac Dover <zac.dover@proton.me>
2023-09-12 20:56:03 +10:00
Ali Masarwa
1f8f0a9e66 RGW | bucket notification: adding more sleeping time to prevent test from failing
Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
2023-09-12 13:31:31 +03:00
Milind Changire
7e2546f945
PendingReleaseNotes: add note about 'm' and 'M' periods
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 14:11:03 +05:30
Milind Changire
7431b813a5
doc/cephfs: add note about monthly scheduling and retention period
Fixes: https://tracker.ceph.com/issues/62494
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 14:11:03 +05:30
Milind Changire
9c88234791
qa: update minute period multiplier to m
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 14:11:03 +05:30
Milind Changire
e7218e03ae
qa: change minute duration specifier from M to m
Fixes: https://tracker.ceph.com/issues/62494
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 13:58:26 +05:30
Milind Changire
2f6d87d181
mgr/snap_schedule: change minute duration specifier from M to m
Problem:
As per the issue tracker, the period spec specifier 'M' is not
consistent with what is used elsewhere, like the period specifiers
displayed in the 'ceph status' command output.
The 'M' period specifier is used as a 'minute' level period specifier by
the cephfs team.
The issue reporter suggests to use 'M' as a 'month' period specifier.

Solution:
Since the 'minute' level period specifer, 'M', is used internally by
the development team, it is failrly easy to swap the 'minute' ('M')
level and 'month' ('m') level period specifers to finally mean that 'm'
implies 'minute' level period and 'M' implies 'month' level period.
Also, since this is the first time that somebody has ever reported that
neither the 'M' nor the 'm' level specifiers work in production, it is a
good idea to fix them once and for all.

Fixes: https://tracker.ceph.com/issues/62494
Signed-off-by: Milind Changire <mchangir@redhat.com>
2023-09-12 13:58:21 +05:30
Roy Sahar
b690343128
Merge pull request #53408 from roysahar/nvmeof_fix_omap_state_object_contains_None
nvmeof-gw: omap object name contains None as string due to generated template to ceph-nvmeof.conf
2023-09-12 10:43:24 +03:00
Liu-Chunmei
17df35e9a7
Merge pull request #53232 from myoungwon/wip-enable-rbm-tests
test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
2023-09-11 23:25:24 -07:00
Myoungwon Oh
73de8937f6 crimson/os/seastore/object_data_handler: consider a RBM case when checking if write can be merged
RBM's paddr always indicates physical address, which means it doesn't have the dealayed.
So, this commit adds a condition that checks if given paddr is used for ongoing write.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
2023-09-12 12:17:47 +09:00
Xiubo Li
5b9df823a2 mds: continue linking if targeti is temporarily located in stray dir
If one inode has more than one hardlink and after the primary dentry
is unlinked it will located the inode in the stray dir temporarily,
which is pending reintegration.

Just before the linkmerge/migrate is triggered a link request comes
it will fail with -EXDEV.

Just skip it and continue the linking.

Fixes: https://tracker.ceph.com/issues/56695
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2023-09-12 11:10:18 +08:00
Xiubo Li
8e240303f8 Revert "mds: wait unlink to finish to avoid conflict when creating same dentries"
This reverts commit d4b9431dfe.

Fixes: https://tracker.ceph.com/issues/61818
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2023-09-12 11:10:18 +08:00