Commit Graph

114063 Commits

Author SHA1 Message Date
Kefu Chai
a53a2e6e37 crimson/osd: support getxattrs op
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-07 19:15:24 +08:00
Kefu Chai
96fba5e6b2 crimson/osd: consume indata
we should chop off the consumed chunk from indata, otherwise the op
consuming data from indata will always read from the beginning of
indata.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-07 19:15:24 +08:00
Kefu Chai
36c45bfbcb crimson/osd: support checksum op
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-07 19:15:24 +08:00
Sebastian Wagner
296ef28952
Merge pull request #36373 from varshar16/wip-update-orch-doc-str
mgr/orchestrator: Update docstring of service_action method

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2020-08-07 11:30:49 +02:00
Sebastian Wagner
60eaecefd4
Merge pull request #36447 from sebastian-philipp/mgr-cephadm-kill--refresh
mgr/cephadm: Make `ceph orch * --refresh` asynchronous

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-08-07 11:29:23 +02:00
Sebastian Wagner
6105d1ba4c mgr/cephadm: _get_config_and_keyring: host is required now
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-07 11:24:52 +02:00
Varsha Rao
4d6365b6a9 mgr/orchestrator: Update service_action() docstring
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-08-07 11:27:28 +05:30
Patrick Donnelly
4c52403367
Merge PR #36501 into master
* refs/pull/36501/head:
	qa: add tests for mds_min_caps_working_set
	mds: add working set minimum for caps
	qa: use config_set/config_get
	qa: do not append file names to dirname
	qa: add exception for test timeouts

Reviewed-by: Douglas Fuller <dfuller@redhat.com>
2020-08-06 22:12:42 -07:00
Patrick Donnelly
63b963e899
Merge PR #36504 into master
* refs/pull/36504/head:
	qa: log entrance/exit into the volumes plugin
	qa: add debug client for volumes plugin qa

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2020-08-06 22:02:19 -07:00
Patrick Donnelly
47b1319df3
Merge PR #36503 into master
* refs/pull/36503/head:
	client: use non-static dirent for thread-safety

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2020-08-06 21:51:59 -07:00
Kefu Chai
dc52575429
Merge pull request #36361 from tchaikov/wip-common-config-refactor
common/config: refactor md_config_t::parse_config_files()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-08-07 10:32:51 +08:00
Patrick Donnelly
f4fc138849
qa: add tests for mds_min_caps_working_set
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:03 -07:00
Patrick Donnelly
63392e1b65
mds: add working set minimum for caps
A client may hold many inodes pinned in its cache for open files. That
client may be unable to release those caps to respond to cache pressure
from the MDS (or quiescent client cap recall). We should not complain if
that number of capabilities is reasonable (< 10k by default).

Fixes: https://tracker.ceph.com/issues/46830
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:03 -07:00
Patrick Donnelly
edc5c14d1c
qa: use config_set/config_get
It's simpler and does not require MDS restarts.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:03 -07:00
Patrick Donnelly
ac6c150eb0
qa: do not append file names to dirname
Otherwise the files generated are not actually under the sub-directory!
This is correcting a confusing aspect of the test infrastructure but
doesn't actually require any changes to the tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 19:17:02 -07:00
Patrick Donnelly
a89b1633e1
qa: log entrance/exit into the volumes plugin
Fixes: https://tracker.ceph.com/issues/46851
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 17:25:07 -07:00
Radoslaw Zarzynski
dd3cab64a3 crimson: request the newest Seastar's API.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 18:00:07 -04:00
Radoslaw Zarzynski
ed08faf249 crimson/test: adapt to the new interface of seastar::when_all_succeed().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 18:00:07 -04:00
Radoslaw Zarzynski
c155464f6a crimson/osd: adapt to the new interface of seastar::make_file_output_stream().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 18:00:07 -04:00
Radoslaw Zarzynski
d483c52375 crimson/osd: adapt to the new interface of seastar::when_all_succeed().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 18:00:07 -04:00
Radoslaw Zarzynski
6538b34ea4 crimson/common: implement satisfy_with_result_of() in errorator's futurize.
This is needed due to the recent changes in `seastar::future`'s
implementation.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 17:59:06 -04:00
Radoslaw Zarzynski
7acef5765b crimson/osd: use std::filesystem without S*'s compatibility layer.
This is needed as newer versions of the framework have
the `seastar::compat` compatibility layer stripped.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 17:59:05 -04:00
Radoslaw Zarzynski
d3eb060edc crimson/common: switch errorator to use std::apply directly.
This is needed as newer versions of the framework have
the `seastar::apply` compatibility layer stripped.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 17:59:05 -04:00
Patrick Donnelly
9bcda7ca42
qa: add debug client for volumes plugin qa
Fixes: https://tracker.ceph.com/issues/46851
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 14:27:09 -07:00
Patrick Donnelly
9b00083489
client: use non-static dirent for thread-safety
Fixes: https://tracker.ceph.com/issues/46832
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 14:04:57 -07:00
Patrick Donnelly
dbc44e5e34
qa: add exception for test timeouts
To make this easier to catch. It is still a RuntimeError so it should
not affect current tests by default.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2020-08-06 11:21:54 -07:00
Kefu Chai
e0620eefbd qa/tasks/ceph: redirect stderr for crimson flavor
we should redirect stderr for crimson instead for default flavor. this
change addresses a regression introduced by
da76f46461

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-07 01:11:57 +08:00
Sebastian Wagner
b7caeed9b6 mgr/orch: _interface.py: PEP8tify
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-06 18:32:57 +02:00
Sebastian Wagner
0bb09324e7 mgr/orch: _interface.py: Convert type annotations to py3
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-06 18:32:57 +02:00
Willem Jan Withagen
678a3154c8 librbd/cache: Fix scoping issue with lambda capture renaming
Clang complains:
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:97:66: error: 'object_off' in capture list does not name a variable
   ([this, read_data, dispatch_result, on_dispatched, object_no, object_off,
                                                                 ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:98:6: error: 'object_len' in capture list does not name a variable
     object_len, snap_id, &parent_trace](ObjectCacheRequest* ack) {
     ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:99:41: error: reference to local binding 'object_off' declared in enclosing function 'librbd::cache::ParentCacheObjectDispatch::read'
      handle_read_cache(ack, object_no, object_off, object_len, snap_id,
                                        ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:81:9: note: 'object_off' declared here
  auto [object_off, object_len] = extents.front();
        ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:99:53: error: reference to local binding 'object_len' declared in enclosing function 'librbd::cache::ParentCacheObjectDispatch::read'
      handle_read_cache(ack, object_no, object_off, object_len, snap_id,
                                                    ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:81:21: note: 'object_len' declared here
  auto [object_off, object_len] = extents.front();
                    ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:99:41: error: reference to local binding 'object_off' declared in enclosing function 'librbd::cache::ParentCacheObjectDispatch<librbd::ImageCtx>::read'
      handle_read_cache(ack, object_no, object_off, object_len, snap_id,
                                        ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:242:31: note: in instantiation of member function 'librbd::cache::ParentCacheObjectDispatch<librbd::ImageCtx>::read' requested here
template class librbd::cache::ParentCacheObjectDispatch<librbd::ImageCtx>;
                              ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:81:9: note: 'object_off' declared here
  auto [object_off, object_len] = extents.front();
        ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:99:53: error: reference to local binding 'object_len' declared in enclosing function 'librbd::cache::ParentCacheObjectDispatch<librbd::ImageCtx>::read'
      handle_read_cache(ack, object_no, object_off, object_len, snap_id,
                                                    ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/cache/ParentCacheObjectDispatch.cc:81:21: note: 'object_len' declared here
  auto [object_off, object_len] = extents.front();
                    ^
6 errors generated.

fixes: https://github.com/ceph/ceph/pull/36145
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2020-08-06 16:16:18 +00:00
Sebastian Wagner
f95259251c
Merge pull request #36426 from sebastian-philipp/cephadm-redeploy-image
mgr/cephadm, Add --image to `ceph orch redeploy <what>` 

Reviewed-by: Joshua Schmid <jschmid@suse.de>
2020-08-06 18:00:35 +02:00
Ricardo Marques
4cbf587b68 mgr/cephadm: Prevent unnecessary SSH reconfiguration
When 'set-user', 'set-pub-key' or 'set-priv-key' are called two
times with the same value, we can ignore the second call.

Signed-off-by: Ricardo Marques <rimarques@suse.com>
2020-08-06 16:52:47 +01:00
Sebastian Wagner
01e2dafb6e mgr/cephadm: Some more typing
Make sure we don't mix daemon names, config entities and
auth entities. Cause all of them don't mix well together.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-06 17:49:19 +02:00
Xiubo Li
63cb60b525 librados: add LIBRADOS_SUPPORTS_GETADDRS support
This will be very helpful when release ceph package(like RPM) with
backporting rados_getaddrs() api to previous version.

In tcmu-runner we need to check whether the rados_getaddrs() is
support or not.

Fixes: https://tracker.ceph.com/issues/46842
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 21:58:19 +08:00
Lenz Grimmer
9809c18e11
Merge pull request #36445 from rhcs-dashboard/crushmap_fix
mgr/dashboard: Fix for CrushMap viewer items getting compressed vertically

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
2020-08-06 13:48:54 +02:00
Sebastian Wagner
289f37d296 mgr/cephadm: Refactor upgrade to call _daemon_action with new image
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-06 13:20:04 +02:00
Sebastian Wagner
9d0480d18b mgr/cephadm, Add --image to ceph orch daemon redeploy <what>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-08-06 13:18:58 +02:00
Kefu Chai
8d2401c191
Merge pull request #36490 from tchaikov/wip-crimson-zero-op
crimson/osd: support CEPH_OSD_OP_ZERO op and cls_cxx_write_zero()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 18:16:58 +08:00
Kefu Chai
3637b27084 crimson/osd: support cls_cxx_write_zero()
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-06 17:54:53 +08:00
Kefu Chai
d47055575a crimson/osd: support CEPH_OSD_OP_ZERO op
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-06 17:54:53 +08:00
Kefu Chai
a122f57377 crimson/admin: drop stale comment
the UNIX domain socket file was removed in
71c08cb02e

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-08-06 17:54:53 +08:00
Kefu Chai
6447e20f78
Merge pull request #36455 from rzarzynski/wip-crimson-ops_executer-cleanups
crimson/osd: don't reset osd_op_params.clean_regions. Clean-up OpsExecuter

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-08-06 17:39:52 +08:00
Xiubo Li
1b1af63327 client: switch lock_guard to scoped_lock
Fixes: https://tracker.ceph.com/issues/46649
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 16:23:35 +08:00
Xiubo Li
b47374177c client: remove useless unsafe_sync_write
With the RWRef supporting, before _unmount() could continue all the
the sync write(the "readers") must have finished.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 16:23:35 +08:00
Xiubo Li
d7904c29a4 client: make the root member under the client_lock
Fixes: https://tracker.ceph.com/issues/46649
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 16:23:35 +08:00
Xiubo Li
1198a47b97 client: add mount/initialize states support and convert to RWRef
This will help us to break the big client_lock. Such as if we want
to add one separate lock for each inode, it will work like:

inode_lock.lock();
...
client_lock.lock();

client_lock.unlock();
...
inode_lock.unlock();

So we must get rid of the client_lock for all the "readers", which
must hold the client_lock from the beginning until the end.

Fixes: https://tracker.ceph.com/issues/46649
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 16:23:29 +08:00
Xiubo Li
96514a11ff client: add RWRef support
This is a common read/write reference framework, which will work
like a rw lock, for the readers/writers it won't hold the lock but
will increase the reference instead.

With this we can get rid of big locks in some use case, such as for
the Client.cc, in which such as for the file read()/write() it must
hold the client_lock at the begining until they finish, this will
make sure the _unmount() won't release the resources the read()/write()
are still using. With this it maks breaking the big client_lock to
be possible.

The usage: such as in Client.cc

Readers:

For the read()/writer(), etc, they will be work as "readers", in
the beginning they just need to check the state whether it satisfies
what the "readers" need to be, if not it will return directly, or
will increase the reference and continue.

Writers:

And for the _unmount(), as the "writer" in the beginning it will
just update the state to next stage first and when wait for all
the "readers" to finish. After the "writer" updating the state then
all the new comming "readers" will fail and return directly.

Fixes: https://tracker.ceph.com/issues/46649
Signed-off-by: Xiubo Li <xiubli@redhat.com>
2020-08-06 16:23:16 +08:00
Radoslaw Zarzynski
e77292dc9c crimson/osd: spawn osd_op_params in do_write_op(). Fix overriding.
This commit deduplicates the `OpsExexuter::do_osd_op()` by moving
`std::optional`-typed `osd_op_params` instantiation to `do_write_op()`.
It also fixes an issue where `clean_regions` of `osd_op_params` were
reset on every write, writefull or truncate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 09:52:37 +02:00
Radoslaw Zarzynski
a8733e227d crimson/osd: instantiate osd_op_params for every write.
This simplifies the logic of `OpsExecuter::flush_changes()` and
enables for moving the instantiation code to `do_write_op()` in
the future.

The performance impact seems negligible as the affected ops are
not the ones dominating in our expected workload (read/write, omap).

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 09:52:37 +02:00
Radoslaw Zarzynski
13ef6dcead crimson/osd: delegate the txn.empty() dispatch to OpsExecuter.
The benefit is no need to instantiate `osd_op_params` when only
non-modifying operation had been handled.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-08-06 09:52:36 +02:00