Commit Graph

114951 Commits

Author SHA1 Message Date
Rishabh Dave
9a36a0abd0 qa/cephfs: allow not aborting execution when mount command fails
This commit adds a new argument check_status to mount methods of
KernelMount, FuseMount, LocalKernelMount and LocalFuseMount. When value
of this argument is False, these methods would catch the
CommandFailedError exception and would return a tuple consisting of the
exception itself, and stdout and stderr of the mount command. This
allows reusing these mount methods while running negative tests for
commands.

The name "check_status" is selected so since teuthology's run() and
vstart_runner's run() use a variable with same name for the very same
purpose.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
07e493ffb5 qa/cephfs: allow reusing mount objects and add remount method
This commit introduces following two set of changes -

First, make client keyring path, mountpoint on host FS and CephFS and
CephFS's name attributes of the object representing the mount
and update all the mount object creation calls accordingly. Also,
rewrite all the mount object creation to use keyword arguments instead
of positional arguments to avoid mistakes, especially since a new
argument was added in this commit.

Second, add remount method to mount.py so that it's possible to unmount
safely, modify the attributes of the object representing the mount and
mount again based on new state of the object *in a single call*. The
method is placed in mount.py to avoid duplication.

This change has two leads to two more changes: upgrading interface of
mount() and mount_wait() and upgrading testsuites to adapt to these
change.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
8301c44c70 doc/cephfs: add minor fixes for client-auth doc
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
7de4de1b71 mon/AuthMonitor: make fs authorize assign FS specific MON cap
So far running "fs authorize" subcommand sets "allow r" as MON cap;
change that to "allow <perms> fsname=<fsname>".

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Rishabh Dave
f93c4c7188 src/mds: allow passing fs names and path in same cap
Allow passing path along with fs names and capspec while creating an MDS
cap. The new syntax looks as follows -

allow rw fsname=<fsname> path=<path>

To provide caps for multiple file systems, pass the same phrase multiple
times separated by commas -

allow rw fsname=<fsname1> path=<path1>, allow rw fsname=<fsname2>
path=<path2>, ...

This commit also makes sure that the old syntax 'allow rw path=<path>'
is supported for backwards compatibility. The old syntax would imply
'allow rw fsname=* path=<path>' and would grant read-write permission for
all FSs containing the path <path>.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Douglas Fuller
91d5715be6 cephfs: add auth caps based on fs names
Add new auth caps to restrict access to clients based on fsnames. To
specify this, for example:

mds 'allow rw fsname=cephfs1'

This will restrict client access to fs name "cephfs1" only. Messages to
active MDS assigned to any other FSMap will be dropped. Standby MDS not
associated with an FSMap will accept messages from clients. To allow
multiple file systems, create MDS cap as follows -

mds 'allow rw fsname=cephfs1, allow rw fsname=cephfs2'

Fixes: http://tracker.ceph.com/issues/15070
Signed-off-by: Douglas Fuller <dfuller@redhat.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Douglas Fuller
51af2346fd cephfs: store fs names associated with local MDSMap in MDSRank
Modify MMDSMap to include the name of the file system from the FSMap
when bringing up a new MDS. Store the name in the MDSRank structure for
use in security validation.

Fixes: http://tracker.ceph.com/issues/15070
Signed-off-by: Douglas Fuller <dfuller@redhat.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Douglas Fuller
811e48d6ff mon/MDSMonitor: add mon auth caps for CephFS names
Add a 'fsname' clause to mon auth caps to restrict a client's view
of the FSMap. Example:

mon 'allow rw fsname=cephfs2'

This would restrict the client's view of the FSMap to the MDSMap for
cephfs2. Any MDS allocated to a different filesystem will be invisible.
Global standby daemons are always visible. To allow multiple CephFSs,
add multiple caps:

mon 'allow rw fsname=cephfs2, allow rw fsname=cephfs2'

Fixes: http://tracker.ceph.com/issues/15070
Signed-off-by: Douglas Fuller <dfuller@redhat.com>
Signed-off-by: Rishabh Dave <ridave@redhat.com>
2020-09-10 17:10:51 +05:30
Jan Fajerski
4ade3e84ad
Merge pull request #37039 from jan--f/c-v-no-fstrings
ceph-volume: avoid format strings for now
2020-09-10 13:34:44 +02:00
Kefu Chai
576528d08a
Merge pull request #36855 from yaarith/fix-devid-split
mgr/telemetry: fix device id splitting when anonymizing serial

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-10 15:51:20 +08:00
Jan Fajerski
5baeff5ca5 ceph-volume: avoid format strings for now
Fixes: https://tracker.ceph.com/issues/47354

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2020-09-10 09:50:55 +02:00
Kefu Chai
d2ba22a2d1
Merge pull request #37005 from tchaikov/wip-buffer-hyprcombine
common/buffer: check for null before deref

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-09-10 15:49:55 +08:00
Ilya Dryomov
a1f8589679
Merge pull request #37050 from idryomov/wip-krbd-namespace-symlinks
rbd: include RADOS namespace in krbd symlinks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-09-10 08:04:10 +02:00
Kefu Chai
0f2e67a616
Merge pull request #37073 from tchaikov/wip-crimson-test-config
test/crimson: fix racing in testing

Reviewed-by: Samuel Just <sjust@redhat.com>
2020-09-10 13:43:13 +08:00
Josh Durgin
ce33c21054
Merge pull request #36961 from jdurgin/wip-auth-perf
mon/AuthMonitor: speed up caps updates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-09 22:12:52 -07:00
Kefu Chai
864364547f cmake: add unittest_seastar_config as a ceph test
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-10 11:53:43 +08:00
Kefu Chai
fe0f662a93 test/crimson: fix racing in testing
before this change, unittest_seastar_config set the specified option to
the shard id in parallel and expects that the values of the option are
consistent across all shards after setting the value on all shards. but
sharded::invoke_on_all() is executed in parallel, and so does
ConfigProxy::do_change(), so there are actually "n x n" continuations
racing each other. the order of the continuations storing the setting
cannot be determined. so we cannot expected that the last batch of
continuations hitting shareds store the same value.

in this change, only a single `conf.set_val()` call is performed with
a known value. and this value is checked with the values stored on all
shards.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-10 11:53:43 +08:00
Kefu Chai
457fe7202f test/crimson: use seastar::make_ready_future()
instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

this change is a leftover of aa28b51f3c

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-10 11:53:43 +08:00
Kefu Chai
e5b9b08cc4
Merge pull request #36962 from tchaikov/wip-qa-py3-cleanup
qa: py3 cleanups

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-10 09:39:20 +08:00
Casey Bodley
d03a1cec24
Merge pull request #37054 from cbodley/wip-47306
rgw: ObjectCache::put() clears stale objv

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-09-09 17:11:43 -04:00
Matt Benjamin
ce1a1db9fe
Merge pull request #36619 from cbodley/wip-46734
dmclock: update submodule to fix hang on thread shutdown
2020-09-09 15:57:24 -04:00
Matt Benjamin
4c48e03848
Merge pull request #35548 from adamemerson/wip-datalog-fifo
FIFO the Datalog!
2020-09-09 15:56:26 -04:00
Jason Dillaman
fa53f1032f
Merge pull request #36964 from trociny/wip-47275
mgr: don't update pending service map epoch on receiving map from mon

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2020-09-09 11:32:06 -04:00
Lenz Grimmer
e0b2970bca
Merge pull request #34777 from nSedrickm/grafana-iframe
mgr/dashboard: increase Grafana iframe height to avoid scroll bar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2020-09-09 11:10:20 +02:00
Kefu Chai
6dea032040
Merge pull request #37057 from tchaikov/wip-bump-seastar
seastar,crimson: bump seastar to use upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-09-09 15:36:04 +08:00
Ilya Dryomov
7ccd2c0dce qa: add test for krbd symlinks created by udev
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2020-09-09 09:21:54 +02:00
Kefu Chai
ae2b187470
Merge pull request #37042 from xxhdx1985126/wip-47357
crimson/osd: avoid accessing WaitForObjectRecovery::pi when no pulling occurred

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:48:46 +08:00
Kefu Chai
168f23f9c2 seastar: bump seastar to use upstream master
before this change we are using a fork which includes a fix to address
the FTBFS with GnuTLS < 3.5.6

after this change, we are using the lastest upstream which already
contains this fix.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
4234c3f6c0 crimson/osd: print notify_id instead of its address
this addresses the FTBFS like

    /usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
      902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
          |                   ^~~~~~~~~~

also, i think it is more helpful in this context to have the
notify id instead of the address, as we can use the id to reference
the notification printed by other logging messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
2fa982eabc crimson/os: cast pointers to void* before printing them
to address the FTBFS like:

../src/crimson/os/seastore/lba_manager/btree/btree_range_pin.cc:110:14:   required from here
/usr/include/fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
ebfed2b1ac crimson/os: instantiate future_type<> with future_stored_type
to accommodate the change on seastar's side, which started to use
monostate for representing `void` return type, and hide this convention
using future_stored_type_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
5563e0584b crimson/common: move continuation_base_from_future out of internal namespace
otherwise we have

../src/crimson/common/errorator.h:1090:8: error: specialization of ‘template<class Future> struct seastar::continuation_base_from_future’ in different namespace [-fpermissive]
 1090 | struct continuation_base_from_future<Container<::crimson::errorated_future_marker<Values...>>> {
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

since seastar has moved its continuation_base_from_future() out
of seastar::internal into seastar namespace

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
ce8b544b98 crimson/net: print entity_addr_t not the in4_addr in it
for two reasons:

* we don't have operator<<(ostream&, in4_addr&) defined.
* we don't have any fmt::format() facility specialized for in4_addr
* to print entity_addr_t instead of in4_addr is more correct in
  the sense that we should support IPv6

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
6e7cc5b2c4 test/crimson: include seastar/util/later.hh for seastar::now()
seastar::now() is extracted into seastar/util/later.hh in a recent
change in seastar, so update accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:27:25 +08:00
Kefu Chai
8d814cc09a crimson: include seastar/core/when_all.hh for when_all()
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 13:26:11 +08:00
Xuehan Xu
d5fc5708fa crimson/osd: use temporary RecoveryInfo for recovering object delete ops
Right now, recover_delete use WaitForObjectRecovery::pi, which may not be available
if there were no pulling, to pass recovery_info, so use temporary recovery_info instead

Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-09-09 13:15:49 +08:00
Xuehan Xu
652ecffaf1 crimson/osd: set push future an exception when there are errors during pushing
Fixes: https://tracker.ceph.com/issues/47357
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-09-09 13:15:44 +08:00
Kefu Chai
86e9b6887f crimson/net: do not print non-void* pointer
to address the failure like

fmt/core.h:902:19: error: static assertion failed: formatting of non-void pointers is disallowed
  902 |     static_assert(!sizeof(T), "formatting of non-void pointers is disallowed");
      |                   ^~~~~~~~~~

and i searched around, couldn't find any other
statement printing the address of message, so
i think it's not of much use to print their
addresses in a single place, unless we can cross
check them in different places.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 12:03:37 +08:00
Kefu Chai
aa28b51f3c crimson: use seastar::make_ready_future()
instead of seastar::now(). as seastar::now() was moved into
seastar/util/laster.hh.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-09 12:03:37 +08:00
Adam C. Emerson
dbd2ee5819 cls/fifo: Add 'exclusive' option to Trim
To support RGW's MetadataLog, add a flag to FIFO::trim that when true
trims up to but not including the given entry.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
0819f57474 rgw: Datalog with selectable backends, FIFO and Omap
By default, use whatever is present (and use FIFO if nothing is
present.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
af522e06f9 rgw: Fix async caller into Datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
56aa586f9c rgw: Remove superfluous wrapper around datalog
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
ebc1eed202 rgw: Normalize DataLog function signatures
Pull out the stuff we aren't using so FIFO and Timelog Datalog can use
the same interface.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
50c8646a75 common: Make error conversion functions nodiscard
As suggested by Casey, so the compiler can catch failures to return
errors.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
186b22175c rgw: Get rid of BucketFilter class, use a std::function
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
ae5660fbb6 rgw: Remove date from REST log trim/list
Also only accept one marker.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
26f5b2f58e rgw: Remove some TimeLog options from radosgw-admin
Remove date-oriented trim/list functionality from MDLOG, DataLog, and
Sync-Error-Log.

Don't provide start/end markers. Support only a start marker for list
and an end marker for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
6110ba6058 gitignore: Add vstart_environment.sh
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
2d2a90c15a rgw: FIFO for legacy RADOS client
Use optional_yield for most operations and provide an AioCompletion*
overload for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00