Commit Graph

136919 Commits

Author SHA1 Message Date
Luís Henriques
b70c752e4a PendingReleaseNotes: add reference to the new mdsmap max_xattr_size field
Signed-off-by: Luís Henriques <lhenriques@suse.de>
2023-03-01 10:38:11 +00:00
Luís Henriques
bf7ddd8e45 qa/tasks/mgr/dashboard/test_health: Add 'max_xattr_size' to the mdsmap schema
Let the mdsmap schema know about the new field 'max_xattr_size'.  This prevents
the following error:

tasks.mgr.dashboard.helper._ValError: \
  In `input['fs_map']['filesystems'][0]['mdsmap']`: unknown keys: {'max_xattr_size'}

URL: https://tracker.ceph.com/issues/55725
Signed-off-by: Luís Henriques <lhenriques@suse.de>
2023-03-01 10:35:28 +00:00
Luís Henriques
ca1a397cca mds: prevent exceeding xattrs limits on initial set
When doing a sync setxattr (MDS_OP_SETXATTR) to set the first xattr in an
inode it is possible for the client to set a huge xattr key/value that would
exceed the limit.  Prevent this from from happening by checking the size
against the maximum allowed size.

URL: https://tracker.ceph.com/issues/55725
Signed-off-by: Luís Henriques <lhenriques@suse.de>
2023-03-01 10:35:28 +00:00
Luís Henriques
13e07ff3e9 mds: prevent clients from exceeding the xattrs key/value limits
Commit eb915d0eec ("cephfs: fix write_buf's _len overflow problem") added
a limit to the total size of xattrs.  This limit is respected by clients
doing a "sync" operation, i.e. MDS_OP_SETXATTR.  However, clients with
CAP_XATTR_EXCL can still buffer these operations and ignore these limits.

This patch prevents clients from crashing the MDSs by also imposing the
xattr limits even when they have the Xx caps.  Replaces the per-MDS knob
"max_xattr_pairs_size" by the new mdsmap setting that the clients can
access.

Unfortunately, clients that misbehave, i.e. old clients that don't respect
this xattrs limit and buffer their xattrs, will see them vanishing.

URL: https://tracker.ceph.com/issues/55725
Signed-off-by: Luís Henriques <lhenriques@suse.de>
2023-03-01 10:35:28 +00:00
Luís Henriques
7b8def5c30 mon: add new configuration to limit filesystem xattrs size
This new configuration option will allow to define the maximum size for a
filesystem xattrs blob.  This is a filesystem-wide knob that will replace
the per-MDS mds_max_xattr_pairs_size option.

Note:

The kernel client patch to handle this new configuration was merged before
the corresponding ceph-side pull-request.  This was unfortunate because in
the meantime PR #43284 was merged and the encoding/decoding of
'bal_rank_mask' got in between.  Hence the 'max_xattr_size' is being
encoding/decoded before 'bal_rank_mask'.

URL: https://tracker.ceph.com/issues/55725
Signed-off-by: Luís Henriques <lhenriques@suse.de>
2023-03-01 10:35:25 +00:00
Rongqi Sun
ece7ccccc9 cephadm: eliminate duplication of [global] section when applying set-extra-ceph-conf
Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
2023-03-01 16:15:26 +08:00
Radoslaw Zarzynski
0ad603862b
Merge pull request #49286 from rzarzynski/wip-crimson-snaptrimmer
crimson/osd, osd: bring snap trimming to crimson

Reviewed-by: Samuel Just <sjust@redhat.com>
2023-03-01 08:57:27 +01:00
Xuehan Xu
1923f8db92 crimson/os/seastore/epm: also do prepare_write for rbm
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2023-03-01 07:42:16 +00:00
Venky Shankar
cca84e653d Merge PR #48720 into main
* refs/pull/48720/head:
	qa: fix api failure issue caused during a dashboard test
	PendingReleaseNotes: noted new MDSMap field refuse_client_session
	qa: added two testcases
	client: do not initiate session if flag refuse_client_session is set
	mds: do not reconnect when refuse_client_session is set
	mds: add new feature to block clients from establishing sessions

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2023-03-01 07:54:18 +05:30
Neha Ojha
707af78f9d
Merge pull request #49006 from yanghonggang/yhg-upgrade-rocksdb-7.7.3
rocksdb: Upgrade rocksdb to latest facebook's v7.9.2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
2023-02-28 15:50:21 -08:00
Pete Zaitcev
07f947684c RGW: Add a reader feature
This feature is prompted by a desire to audit contents of
a cluster safely. Currently, such task is done by a privileged
account, which can damage the data by accident. In this
situation, using an account that can only read is an application
of the least privilege.

Note that the reader is a role in RBAC. So, an account can have
both a reader role and a normal user role. The latter allows it to
write the audit report into the cluster, if the operator wants that.

In OpenStack terms, this reader is known as "system reader persona".

Signed-off-by: Pete Zaitcev <zaitcev@redhat.com>
2023-02-28 16:35:27 -06:00
Radoslaw Zarzynski
d90d81585b qa/valgrind.supp: suppress the ColumnFamily ctor called on init
```
    <frame>
      <ip>0x1712F1E</ip>
      <obj>/home/rzarzynski/ceph3/build/bin/ceph-mon</obj>
      <fn>rocksdb::BytewiseComparator()</fn>
      <dir>/home/rzarzynski/ceph3/src/rocksdb/util</dir>
      <file>comparator.cc</file>
      <line>304</line>
    </frame>
    <frame>
      <ip>0x162361C</ip>
      <obj>/home/rzarzynski/ceph3/build/bin/ceph-mon</obj>
      <fn>rocksdb::ColumnFamilyOptions::ColumnFamilyOptions()</fn>
      <dir>/home/rzarzynski/ceph3/src/rocksdb/options</dir>
      <file>options.cc</file>
      <line>123</line>
    </frame>
    <frame>
      <ip>0x12CA21E</ip>
      <obj>/home/rzarzynski/ceph3/build/bin/ceph-mon</obj>
      <fn>Options</fn>
      <dir>/home/rzarzynski/ceph3/build/../src/rocksdb/include/rocksdb</dir>
      <file>options.h</file>
      <line>1409</line>
    </frame>
    <frame>
      <ip>0x12CA21E</ip>
      <obj>/home/rzarzynski/ceph3/build/bin/ceph-mon</obj>
      <fn>RocksDBStore::init(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;)</fn>
      <dir>/home/rzarzynski/ceph3/build/../src/kv</dir>
      <file>RocksDBStore.cc</file>
      <line>400</line>
    </frame>
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 20:58:27 +00:00
Radoslaw Zarzynski
0de4d99c72 qa/valgrind.supp: suppress the get-or-create CacheItemHelper
```
// Get an CacheItemHelper pointer for value type T and role R.
template <typename T, CacheEntryRole R>
Cache::CacheItemHelper* GetCacheItemHelperForRole() {
  static Cache::CacheItemHelper cache_helper(
      BlocklikeTraits<T>::SizeCallback, BlocklikeTraits<T>::SaveToCallback,
      GetCacheEntryDeleterForRole<T, R>());
  return &cache_helper;
}
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 20:43:48 +00:00
Radoslaw Zarzynski
9949151620 qa/valgrind.supp: even broader suppression for rocksdb's init
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 20:43:41 +00:00
Adam King
0aef3d9741
Merge pull request #50308 from adk3798/mock-fqdn-secure-alertmanager
mgr/cephadm: mock get_fqdn in secure alertmanager config test

Reviewed-by: Redouane Kachach <rkachach@redhat.com>
2023-02-28 14:57:44 -05:00
Radoslaw Zarzynski
a3aa624db4 qa/valgrind.supp: ingore rocksdb leak in option parsing
This suppression is needed since upgrading to v7.9.2.
See PR https://github.com/ceph/ceph/pull/49006.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 19:13:39 +00:00
Adam King
3011d95498 qa/distros: pass --allowerasing --nobest when installing container-tools
One of the tests in the orch suite is running distro install
commands from multiple distros, causing it to first install
container-tools 3.0 and then later install container-tools,
which fails, causing the test to fail. This is sort of a bandaid
fix to getthe test to work. It will cause whatever the last
version of the package to be installed to end up being installed
(and will do so without error) which is what we want in the tests.

Fixes: https://tracker.ceph.com/issues/57771

Signed-off-by: Adam King <adking@redhat.com>
2023-02-28 13:23:30 -05:00
Mykola Golub
4d049880a0 mgr/cephadm: try to avoid pull when getting container image info
only if use_repo_digest is not set.

The commit ac88200 introduced this possibility to skip pulling, but
doing this unconditionally broke a use case when one was able to have
a ceph image on a floating tag, and was able to upgrade to a new image
pushed to that tag. As using a floating tag is possible only when
use_repo_digest is enabled (the default), now skipping the pull
if use_repo_digest is disabled will not break it anymore.

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>
2023-02-28 19:27:39 +02:00
Patrick Donnelly
778501178b
client: remove unused method
/home/pdonnell/ceph/src/client/Client.cc:8575:13: warning: ‘void attr_set_atime_and_mtime(stat*, const utime_t&, const utime_t&)’ defined but not used [-Wunused-function]
     8575 | static void attr_set_atime_and_mtime(struct stat *attr,
          |             ^~~~~~~~~~~~~~~~~~~~~~~~

Fixes: 8abccedd12
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2023-02-28 11:26:14 -05:00
Radoslaw Zarzynski
4dbe8f56b6 crimson/osd: fix compilation guard around specialization for libfmt => 9
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
b03ab045e9 crimson/osd: handle snap trim's interruption as any other error
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
38c030e2cb crimson/osd: handle PG_STATE_SNAPTRIM_ERROR when trimming snaps
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
f091a70e3c crimson/osd: errorate SnapTrimEvent
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
e0bb021796 crimson/osd: manage PG_STATE_SNAPTRIM when trimming snaps
After a few tries:

```
$ bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2022-12-19T20:18:21.856+0000 7ff87b46f700 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-19T20:18:21.857+0000 7ff87b46f700  1 build_initial for_mkfs: 0
2022-12-19T20:18:21.860+0000 7ff87b46f700 -1 WARNING: all dangerous and experimental features are enabled.
2022-12-19T20:18:21.861+0000 7ff87b46f700  1 build_initial for_mkfs: 0
  cluster:
    id:     347aedc9-d72f-4a34-98fc-26cb04b0c2fc
    health: HEALTH_WARN
            11 mgr modules have failed dependencies
            1 pool(s) do not have an application enabled
            1 pool(s) have no replicas configured

  services:
    mon: 1 daemons, quorum a (age 3m)
    mgr: x(active, since 3m)
    osd: 1 osds: 1 up (since 3m), 1 in (since 3m)

  data:
    pools:   1 pools, 1 pgs
    objects: 95 objects, 6 B
    usage:   1.0 GiB used, 100 GiB / 101 GiB avail
    pgs:     1 active+clean+snaptrim

  io:
    client:   1.2 KiB/s wr, 0 op/s rd, 1 op/s wr
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
6923b05d70 crimson/osd: drop C++'s ranges to workaround unbuildability on Clang
```
/home/jenkins-build/build/workspace/ceph-pull-requests/src/crimson/osd/osd_operations/snaptrim_event.cc:38:33: note: while checking constraint satisfaction for template 'operator|<const std::ranges::views::_Elements<0> &, const std::vector<std::pair<unsigned long, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<>>>> &>' required here
    for (const auto &i : subops | std::views::keys) {
                                ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/crimson/osd/osd_operations/snaptrim_event.cc:38:33: note: in instantiation of function template specialization 'std::ranges::views::__adaptor::operator|<const std::ranges::views::_Elements<0> &, const std::vector<std::pair<unsigned long, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<>>>> &>' requested here
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/iterator_concepts.h:963:7: note: candidate template ignored: constraints not satisfied [with _Tp = std::ranges::ref_view<const std
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
6dce7df440 crimson/osd: simplify SnapTrimObjSubEvent::start()
Throwing away the NOP-looking-like `maybe_delay` concept became
possible after eradication of the `with_interruption` call nested
within `SnapTrimSubEvent`. Before, on Ubuntu and GCC 11.2.1, lack
of it was the reson behind assertion failures on resetting the
`thread_local` intteruption condition.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
877c272c2f crimson/osd: drop the nested interruptors in snap trimming
`crimson::interruptible` does not support that. See the
`DISABLED_nested_interruptors` unit test.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
c08d06fb0d crimson/test: verify nesting of interruptors
It doesn't work.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
9c719406cc crimson/osd: errorate SnapTrimObjSubEvent::remove_clone()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
912a940f46 crimson/osd: convert SnapTrimObjSubEvent::remove_or_update to ertr
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
30db140344 crimson/osd: dissect snapset update into SnapTrimObjSubEvent::update_head()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
dda6d453d1 crimson/osd: dissect head whiteout removal into SnapTrimObjSubEvent::remove_head_whiteout()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
37f2bda50c crimson/osd: drop data aliases from SnapTrimObjSubEvent::adjust_snaps()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
355b7f7ebd crimson/osd: dissect clone removal into SnapTrimObjSubEvent::remove_clone()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
bb756b5983 crimson/osd: drop data aliases from SnapTrimObjSubEvent::remove_or_update()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
bc5d15e7fc crimson/osd: simplify informing SnapMapper about new mappings
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
b9d81029ae crimson/osd: implement basic osd_snap_trim_sleep support
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
6df8e3965c crimson/osd: introduce WaitSubop stage to SnapTrimEvent
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
9245c7650d crimson/osd: snap trimming informs OpsExecuter about altered objects
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
ad22a9a228 common: add debugs to crimson's green_condition_variable
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
9ce59d7648 crimson/osd: drop unused declaration of PG::fill_op_params_bump_pg_version()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:05 +00:00
Radoslaw Zarzynski
5d6dd2cba8 osd: SnapMapper::OSDriver uses crimson's interruptor::green_get().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
301b2c6b8a crimson/common: bring green_get() to interruptor
The rationale
=============
Calling `get()` on an unavailable future can result in yielding
the reactor, so -- from the interruptor's POV -- the interrupt
condition must be handled exactly like it is done for `yield()`.

Interruptor takes care about that but, at the moment, only for the
interruptible futures; there is mechanism for plain `seastar::future`.
This patch paves the way for handling them as well.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
dccca6beb9 crimson/common: trace management of interrupt_cond in interruptor::async()
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
b9b4101571 crimson/common: fix method name in interrupt_cond_t::reset()'s debugs
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
ab9bd2430f osd: fix intersecion of green threads and interuptible
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
3e5c64a92d crimson/osd: wire SnapMapper::add_oid() with OpsExecuter
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
f6059ed582 crimson/osd: settle snap_trimq in PG and wire it up
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
cb2f8325e8 crimson/osd: settle SnapMapper within crimson'o PG
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00
Radoslaw Zarzynski
e57d3bb082 crimson/osd: PerShardState::start_operation() doesn't assume Ret is seastar::future<>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2023-02-28 16:22:04 +00:00