Commit Graph

135417 Commits

Author SHA1 Message Date
Laura Flores
e315c5d9a3
Merge pull request #48223 from myoungwon/wip-fix-tier-evict-range
osd: mark modified region as dirty when tier-evict is called
2022-12-05 18:13:21 -06:00
Igor Fedotov
84bae39a08
Merge pull request #48467 from baergj/fix-read-error-code
blk/kernel: Fix error code mapping in KernelDevice::read.

Reviewed-by: Robin H. Johnson <robbat2@orbis-terrarum.net>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>
2022-12-06 02:46:16 +03:00
Laura Flores
e75eb834e7
.github: add core label to files under src/test/librados
In response to https://tracker.ceph.com/issues/58173, it would have helped to have the PR label librados test files with core, so the PR would have been tagged for rados qa.

Signed-off-by: Laura Flores <lflores@redhat.com>
2022-12-05 15:07:42 -06:00
Adam King
3ffa7dfa9c
Merge pull request #48817 from adk3798/device-refresh-message
mgr/cephadm: fix check for if devices have changed

Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2022-12-05 14:38:48 -05:00
Adam King
2bf65fff53
Merge pull request #48874 from senax/ingress_backend_ssl
mgr/cephadm: add ingress support for ssl rgw service

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2022-12-05 14:35:07 -05:00
J. Eric Ivancich
22075100f9
Merge pull request #48899 from ivancich/wip-fix-old-shard
rgw: address bug where object puts could write to decommissioned shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2022-12-05 13:51:05 -05:00
Yuri Weinstein
5fb55e91cc
Merge pull request #48585 from cbodley/wip-test-librados-asio-spawn
test/librados: use spawn submodule instead of boost::asio::spawn

Reviewed-by: Adam Emerson <aemerson@redhat.com>
2022-12-05 09:16:03 -08:00
Yuri Weinstein
313c874c05
Merge pull request #48708 from sseshasa/wip-fix-osd-capacity-display
osd: Reset mClock's OSD capacity config option for inactive device type

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
2022-12-05 09:15:01 -08:00
Yuri Weinstein
3d3b96a092
Merge pull request #48623 from batrick/i57923
log: use non-blocking atomic writes to stderr fifos

Reviewed-by: Sage Weil <sage@redhat.com>
2022-12-05 09:13:49 -08:00
Yuri Weinstein
af33104b04
Merge pull request #48513 from ideepika/wip-crush-debug
src/crush: extra logging to debug CPU burn in test_with_fork()


Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2022-12-05 09:12:26 -08:00
Anthony D'Atri
4defc22e33
Merge pull request #49253 from kvanals/update-board-softiron
doc: Update Foundation member for SoftIron
2022-12-05 12:06:16 -05:00
Ilya Dryomov
8780f602a9
Merge pull request #48618 from idryomov/rbd-clone-encryption-part2
librbd: add encryption format support for clones (part 2/2)

Reviewed-by: Mykola Golub <mgolub@suse.com>
Acked-by: Or Ozeri <oro@il.ibm.com>
2022-12-05 17:47:19 +01:00
zdover23
35d70ea256
Merge pull request #49252 from zdover23/wip-doc-2022-12-05-glossary-das
doc/glossary: add DAS

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
2022-12-06 02:38:53 +10:00
Zac Dover
a1f53a0fc0 doc/glossary: add DAS
Add a definition for "DAS" (directly attached storage).

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-12-06 02:07:15 +10:00
kvanals
e2423ec0c5 doc: Update Foundation member for SoftIron
Signed-off-by: kvanals <Kenny.VanAlstyne@softiron.com>
2022-12-05 10:59:41 -05:00
Ilya Dryomov
86b60ebd70
Merge pull request #49235 from idryomov/wip-rbd-groups-bash
qa: rbd_groups.sh: change interpreter to bash

Reviewed-by: Ramana Raja <rraja@redhat.com>
2022-12-05 15:59:29 +01:00
Anthony D'Atri
d0c86e4e5e
Merge pull request #49204 from Tooa/adapt_keepalived_reference
containers: use quay.io/ceph/keepalived in docs
2022-12-05 09:43:31 -05:00
Anthony D'Atri
f08bf506ff
Merge pull request #49249 from krambrod/main
doc/cephadm: fix initial-ceph.conf example
2022-12-05 09:05:01 -05:00
shreyanshjain7174
5008953617 rados: fix extra tabs on warning for pool copy
There seems to be some extra tabs in rados cppool error message which might confuse users.In reality there should not be any extra spaces on warning messages.

Fixes: https://tracker.ceph.com/issues/58165
Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>
2022-12-05 06:34:46 -06:00
Matan
b583ed6ede
Merge pull request #49076 from Matan-B/wip-matanb-crimson-suite-adjust
crimson: Suite testing minor adjustment

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-05 12:19:36 +02:00
krambrod
930fa9150c
doc/cephadm: fix initial-ceph.conf example
Signed-off-by: krambrod <114069812+krambrod@users.noreply.github.com>
2022-12-05 09:36:10 +01:00
colemitchell
a2bc2ab72d
Merge pull request #49241 from zdover23/wip-doc-2022-12-05-rados-ops-monitoring-prompts
doc/rados: add prompts to monitoring.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
2022-12-04 22:07:29 -05:00
colemitchell
9c29927521
Merge pull request #49243 from zdover23/wip-doc-2022-12-05-rados-pg-repair-prompts
doc/rados: add prompts to pg-repair.rst

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>
2022-12-04 22:07:05 -05:00
Zac Dover
e09007c106 doc/rados: add prompts to pg-repair.rst
Add unselectable prompts to doc/rados/operations/pg-repair.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-12-05 12:44:01 +10:00
Zac Dover
7f2872b776 doc/rados: add prompts to monitoring.rst
Add unselectable prompts to doc/rados/operations/monitoring.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-12-05 12:30:35 +10:00
zdover23
b606f8afeb
Merge pull request #49238 from zdover23/wip-doc-2022-12-05-rados-ops-monitoring-osd-pg-prompts
doc/rados: add prompts to monitoring-osd-pg.rst

Reviewed-by: Cole Mtichell <cole.mitchell@gmail.com>
2022-12-05 05:30:28 +10:00
Zac Dover
cd8e903c9c doc/rados: add prompts to monitoring-osd-pg.rst
Add unselectable prompts to doc/rados/operations/monitoring-osd-pg.rst.

https://tracker.ceph.com/issues/57108

Signed-off-by: Zac Dover <zac.dover@gmail.com>
2022-12-05 05:18:36 +10:00
Ilya Dryomov
8d5d478532 qa/workunits/rbd: add encryption-aware resize test
Note that we are hitting https://tracker.ceph.com/issues/58160 here
because by the time we get to "rbd resize" RAW_DEV mapping owns the
lock (due to a write to /dev/mapper/cryptsetupdev being last).

While at it, resurrect the ability to easily run this script on
vstart clusters -- see commit f737c2855a ("qa/workunits/rbd: make
luks-encryption test work on vstart cluster").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:24:10 +01:00
Ilya Dryomov
51d8c526f0 doc/rbd: add clone encryption details and examples
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:24:06 +01:00
Ilya Dryomov
a27ee2bdf8 rbd, rbd-nbd: make --encryption-format optional
If no --encryption-format specified at all, default to "luks" for each
specified --encryption-passphrase-file.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
e62e3b6613 rbd, rbd-nbd: accept "luks", "luks1" and "luks2" formats
Since RBD_ENCRYPTION_FORMAT_LUKS1, RBD_ENCRYPTION_FORMAT_LUKS2
and RBD_ENCRYPTION_FORMAT_LUKS aren't treated the same when loading
encryption anymore, "luks1" and "luks2" formats need to be accepted
in addition to "luks" format.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
75acf7b6f5 librbd: don't decay LUKS{1,2}EncryptionFormat into LUKSEncryptionFormat
Commit 9892ead7fc ("librbd/crypto: allow loading luks format
without specifying version") introduced RBD_ENCRYPTION_FORMAT_LUKS
format identifier, matching cryptsetup's CRYPT_LUKS ("load any LUKS
version happens to be there").  However, in an effort to enable an
obscure "layered encryption with the same passphrase + old QEMU" use
case, it also introduced decaying of RBD_ENCRYPTION_FORMAT_LUKS1 and
RBD_ENCRYPTION_FORMAT_LUKS2 format identifiers, making it impossible
to assert on the format that is being loaded.  This new behavior was
then extended to standalone images.

Treating RBD_ENCRYPTION_FORMAT_LUKS1, RBD_ENCRYPTION_FORMAT_LUKS2
and RBD_ENCRYPTION_FORMAT_LUKS the same when loading encryption can
be construed as an opening for a format downgrade attack.  Let's
resurrect the previous standalone images behavior and extend it to
layered encryption instead.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
0230c1782e rbd: fix passphrase zeroing in "rbd encryption format" handler
"rbd encryption format" handler sets up a scope guard to zero out
the passphrase string on return but also makes a copy of same which
isn't zeroed out.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
d642f7804b rbd, rbd-nbd: don't strip trailing newline in passphrase files
One of the stated goals is compatibility with standard LUKS tools,
in particular being able to load encryption on images formatted with
cryptsetup.  cryptsetup doesn't do this and this really interferes
with randomly generated (binary) passphrases.

While at it, open passphrase files as binary -- it communicates the
intent if nothing else on POSIX.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
adea1ab805 librbd: constify specs array in rbd_encryption_load2()
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
5f816f9b2c librbd: remove unused trim_image() method
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
15c248243f librbd: non-pruning parent overlap handling fixes
Apply similar "reduce overlap and respect area" logic to places
that don't use prune_parent_extents().  Changes to FlattenRequest
and TrimRequest here should complete the long tail of encrypted
I/O path and flatten fixes.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
d47bb2a2db librbd: reduce overlap and respect area when pruning parent extents
DATA area in the parent may be smaller than the part of DATA area in
the clone that is still within the overlap.  This would occur e.g. in
LUKS2-formatted parent + LUKS1-formatted clone case, due to LUKS2
header usually being bigger than LUKS1 header:

parent: raw size = 64M
        LUKS2 header area = 16M
        data area = 48M

clone:  raw size = 64M (raw overlap 64M)
        LUKS1 header area = 4M
        data area = 60M

Currently, because parent extents are pruned only according to raw
overlap (64M), the clone ends up attempting to reach the parent for all
of its data area (60M < 64M) even though the parent only has 48M worth
of data.  All kinds of bugs ensue for 48M..60M offsets and this range
basically becomes inaccessible to the user.

A related issue is that prune_parent_extents() ignores area.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
0e318952a5 librbd: clip extents to their area instead of DATA area
This fixes cases where CRYPTO HEADER area is larger than DATA area.
In particular, it was effectively impossible to flatten unformatted
clones of such images.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
c97b7c4e8b librbd: introduce reduce_parent_overlap() and switch overlap API
When encryption is loaded, rbd_get_overlap() and Image::overlap() now
return "effective" overlap, similar to rbd_get_size() and Image::size().
Previously, returned overlap could have been bigger than "effective"
size.

Note that get_effective_image_size() successor doesn't take snap_id
because passing anything but ictx->snap_id was broken.  Since the size
of the crypto header is stored in the crypto header itself, image areas
are defined only for the "opened at" snap_id.  Getting "effective" size
for an arbitrary snapshot requires actually opening it and loading
encryption on it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
50070a78a8 librbd: tweak get_parent_overlap() signature
Make it clear that get_parent_overlap() returns the raw parent overlap
value (i.e. physical offset into the parent image).  Also drop redundant
ceph_mutex_is_locked assert -- get_parent_info() already has one.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
2279ae08f1 librbd: remap resize target size if encryption is loaded
When encryption is loaded, rbd_get_size() and Image::size() return
"effective" size, but rbd_resize() and Image::resize() continue to take
raw size.  The user has to constantly keep these domains in mind.

Saying that resize must be done without encryption loaded is not an
answer because shrinking a clone that has snapshots involves copying up
objects in the affected range (identical to flattening).  In addition,
even if a clone doesn't have snapshots, shrinking it to a size that
isn't an object boundary is going to involve a copyup for the victim
object as well.

To avoid subtle data corruption on shrink, treat resize operation the
same as flatten operation (including on the CLI).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
b3deeb152a librbd: check stripe pattern when loading encryption
Currently it's done in FormatRequest but not in LoadRequest.  However
an image can be deep copied or exported and imported with a different
stripe pattern such that an area boundary would fall in the middle of
an object.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
36c2d589af librbd: check image size when loading encryption
Currently it's done in FormatRequest but not in LoadRequest.  However
an image can be shrunk to a size such that encryption can loaded (i.e.
enough of the header is still present) but nothing else can, breaking
implicit assumptions all around.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
203560999d librbd: relax image size check in luks::FormatRequest
Proceed with formatting an image even if all space would be consumed by
the crypto header.  There is no reason to be strict here since we allow
creating zero-sized images as well as shrinking any image to 0.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
4f5b735c57 librbd: no need to return format from ShutDownCryptoRequest
luks::FlattenRequest was the only user of this interface.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
abded6eeda librbd: don't temporarily shut down crypto when flattening
(Temporarily) shutting down crypto can lead to data corruption in the
face of concurrent I/O, especially when flatten operation is proxied to
the remote lock owner.  This was added to be able to read, optionally
modify and write crypto header without it being subjected to remapping
and encryption itself.  read_header() and write_header() now achieve
that by specifying CRYPTO_HEADER area explicitly.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
744379b8c6 librbd: move get_file_offset() into CryptoObjectDispatch
This method doesn't propagate area.  Since its only user is
CryptoObjectDispatch which is now applied only to DATA area,
move get_file_offset() there to avoid misuse in the future.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
a54f1a25fa librbd: apply CryptoObjectDispatch layer only to DATA area
Objects in CRYPTO_HEADER area should not be subjected to encryption.
Unit tests needed adjustment because MockCryptoInterface is configured
with DATA_OFFSET = 4 * 1024 * 1024, thus disqualifying object 0.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00
Ilya Dryomov
b9a2384cdc librbd: propagate area down to file_to_extents()
- readahead and PWL cache are limited to DATA area as explained in
  the previous commit
- DATA area is assumed for the journal as encryption can't be used
  with journaling anyway

To postpone the churn associated with passing area through
ImageDispatchInterface (where only WriteLogImageDispatch and
ImageDispatch care), add a new image dispatch flag.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2022-12-04 18:19:19 +01:00