Commit Graph

114894 Commits

Author SHA1 Message Date
Adam C. Emerson
50c8646a75 common: Make error conversion functions nodiscard
As suggested by Casey, so the compiler can catch failures to return
errors.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
186b22175c rgw: Get rid of BucketFilter class, use a std::function
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
ae5660fbb6 rgw: Remove date from REST log trim/list
Also only accept one marker.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
26f5b2f58e rgw: Remove some TimeLog options from radosgw-admin
Remove date-oriented trim/list functionality from MDLOG, DataLog, and
Sync-Error-Log.

Don't provide start/end markers. Support only a start marker for list
and an end marker for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
6110ba6058 gitignore: Add vstart_environment.sh
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
2d2a90c15a rgw: FIFO for legacy RADOS client
Use optional_yield for most operations and provide an AioCompletion*
overload for trim.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
6c950a2c2d cls/fifo: Use neorados interface
Rewrite around the asynchronous ASIO-based RADOS interface, and
include support for calls from multiple threads.

`allocate_unique` pattern contributed by
Casey Bodley <cbodley@redhat.com>

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Yehuda Sadeh
55255343f7 cls/fifo: FIFO over RADOS
This is an implementation of fifo queue over rados. Data is appended
to rados object until it's full. At that point data will be written to
a new object. Data is read from the tail sequentially (can be iterated
using marker). Data can be trimmed (up to and including marker).

Queue has a header object (meta), and zero or more data objects (parts).

The software has two layers: the higher level client operations side
that deals with the application apis, and manages the meta and parts,
and there’s the objclass level that deals with the rados layout of
meta and part objects. There are different objclass methods that deal
with reading and modifying each of these entities.

A single part has max possible size, however, it may become full once
a certain smaller size is reached (full_size_threshold). It is
imperative that once a part has reached its capacity, it will not
allow any more writes into it. For this reason, it is important that
data being written to the queue does not exceed max_entry_size . This
is enforced, by the higher level client api.

Written entries go to the current head object, and when it’s full, a
new head part is created. When listing entries, data is iterated from
tail to the current head. Trim can either change the pointer within
the current tail object, and if needed it removes tail objects.

A unitest has been created to test functionality.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Squashed-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
a29695e82e strtol: Add parse/consume for string_view friendly interface
Also these don't have the stringstream overhead.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
4cec019d5e common/Thread: Don't store pointer to thread_name
Having Thread::create store a pointer to a string that is passed to
ceph_pthread_setname in Thread::entry_wrapper can lead to using a
pointer in the calling thread's stack that gets freed before use.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
2d4c9acc98 common/ceph_json: Support en/decoding Boost's flat containers
Specifically boost::container::flat_map and boost::container::flat_set

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
813acd5d86 common/async: Add handler for librados::AioCompletion
Support using a librados::AioCompletion as a completion handler in
Boost.Asio when the arguments are void or only an error code.

(To support more arguments we'd have to wrap up the AioCompletion with
pointers to hold them.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Casey Bodley
3e5ecffceb common: add ceph::allocate_unique()
adds an allocator-aware version of std::make_unique(). this is similar to
std::allocate_shared(), though it's slightly less convenient because,
unlike std::shared_ptr<T>, the Deleter has to be specified as a template
parameter in std::unique_ptr<T, Deleter>

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
fba1845211 common: bind_like
A template to ensure that one completion has the same executor and
allocator as another.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
9bab68c0f4 common: Add bind_allocator template
Bind an allocator to a completion, analogously to
boost::async::bind_executor.

Based on bind_executor.hpp which is copyright 2003-2019
Christopher M. Kohlhoff <chris@kohlhoff.com> and under
the Boost Software License, Version 1.0

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Adam C. Emerson
da7835947e include: Import (modified) uses_allocator.h
From https://github.com/phalpern/uses-allocator

To handle uses_allocator construction

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2020-09-08 22:09:40 -04:00
Yuri Weinstein
9849b2eb8b
Merge pull request #37049 from yuriw/wip-yuriw-crontab-master
qa/tests: removed extra line

Reviewed-by: Neha Ojha <nojha@redhat.com>
2020-09-08 15:57:20 -07:00
Yuri Weinstein
74f0a8b37a qa/tests: removed extra line
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2020-09-08 10:10:14 -07:00
Kefu Chai
b057b16215
Merge pull request #36230 from trociny/wip-43174-2
os/bluestore: fix collection_list ordering

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-08 21:34:11 +08:00
Kefu Chai
23e3358e3d
Merge pull request #36955 from agayev/zbc-to-zbd
os/bluestore: Switch from libzbc library to libzbd library

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-08 21:24:11 +08:00
Jan Fajerski
7b10f3a18b
Merge pull request #36950 from ceph/wip-bz1872983
ceph-volume: simple scan should ignore tmpfs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
2020-09-08 14:04:00 +02:00
Lenz Grimmer
f4bafb2ada
Merge pull request #36976 from rhcs-dashboard/fix-47285-master
mgr/dashboard: fix duplicate logs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
2020-09-08 12:34:30 +02:00
Kefu Chai
056d8776ce
Merge pull request #37003 from xxhdx1985126/wip-47310
crimson/osd: drop repop if osdmap has increased

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-08 13:38:51 +08:00
Venky Shankar
7251caf7e1
Merge pull request #36974 from jan--f/snap-schedule-keep-with-empty-retention
snap-schedule: don't prune all on empty retention
2020-09-08 10:15:28 +05:30
Xuehan Xu
2c17bc1933 crimson/osd: add can_discard_op method for detecting "discardable" client requests
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-09-08 12:36:41 +08:00
Xuehan Xu
e9ac113f4f crimson/osd: drop repop if it can be discarded
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-09-08 12:36:06 +08:00
Xuehan Xu
ab849a1378 crimson/osd: rename is_valid_rep_op_reply to can_discard_replica_op
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
2020-09-08 12:35:14 +08:00
Sebastian Wagner
3627cd1e5c
Merge pull request #36903 from sebastian-philipp/python-common-simplify-hsp-to-json
python-common: Make HostPlacementSpec.to_json() more friendly

Reviewed-by: Adam King <adking@redhat.com>
2020-09-07 17:58:29 +02:00
Nathan Cutler
839c8ec6ba
Merge pull request #36994 from ktdreyer/boost-context-s390x
rpm: enable WITH_BOOST_CONTEXT for s390x

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2020-09-07 12:39:12 +02:00
Sebastian Wagner
75d9369a02
Merge pull request #36432 from sebastian-philipp/cephadm-repo_digest
mgr/cephadm: convert tags to repo_digest

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
2020-09-07 11:57:51 +02:00
Kefu Chai
425b2167dc
Merge pull request #37015 from tchaikov/wip-seastar-packet
common/buffer_seastar: reserve the packet slots in ctor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-09-07 17:44:05 +08:00
Sebastian Wagner
428125f8a4
Merge pull request #36946 from zdover23/wip-docs-36796-fix
doc/cephadm: Document mgr/cephadm/manage_etc_ceph_ceph_conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-09-07 10:48:27 +02:00
Sebastian Wagner
b54e270930
Merge pull request #36995 from smithfarm/wip-47305
rpm: on SUSE, podman is required for cephadm to work

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-09-07 10:39:25 +02:00
Kefu Chai
1ef86021ca
Merge pull request #36992 from uweigand/fix-encode-float
include/encoding: Fix encode/decode of float types on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:50:37 +08:00
Kefu Chai
b13c14b6e8
Merge pull request #36829 from changchengx/osdmap
crush,osd: refine crushmap comment and implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:49:08 +08:00
Kefu Chai
e3e6d30d37
Merge pull request #36947 from pdvian/wip-mgr-get-rule-root
mgr/mgr_module : Fix keyerror for op if rule steps first element is opcode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:46:52 +08:00
Kefu Chai
bc4255ba2c
Merge pull request #36972 from smithfarm/wip-47112
rpm,deb: drop /etc/sudoers.d/cephadm

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:44:44 +08:00
Kefu Chai
7b23ddf13b
Merge pull request #36937 from tchaikov/wip-mgr-cleanup
mgr: use range-based loop and cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2020-09-07 14:42:01 +08:00
Kefu Chai
f236ead710
Merge pull request #36999 from neha-ojha/wip-47309
qa/*/mon/mon-last-epoch-clean.sh: mark osd out instead of down

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:40:51 +08:00
Kefu Chai
11ca0cdad0
Merge pull request #37008 from tchaikov/wip-crimson-pg-tell
crimson/osd: add "ceph tell <pgid> <command>" support

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
2020-09-07 14:39:44 +08:00
Kefu Chai
94ca7736da common/buffer_seastar: reserve the packet slots in ctor
for better performance, and simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-07 14:07:29 +08:00
Kefu Chai
2425c4b4de
Merge pull request #37010 from tchaikov/wip-crimson-osd-net
crimson/net: move implementation into .cc file

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2020-09-07 13:00:14 +08:00
Kefu Chai
df10f54f71
Merge pull request #37001 from tchaikov/wip-crimson-basic
qa/suites: add crimson-rados/basic tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-09-07 09:52:45 +08:00
Kefu Chai
d536386306 cmake: add "WITH_ZBD" option
for enabling Zoned Block Device support using libzbd

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-06 12:35:05 -04:00
Abutalib Aghayev
bb488b06c9 os/bluestore: Switch from libzbc library to libzbd library
Libzbc maintainers recommend switching to libzbd, which is lighter and supports
both ZNS SSDs and HM-SMR HDDs.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>
2020-09-06 12:35:05 -04:00
Patrick Donnelly
c0d5d348e5
Merge PR #36600 into master
* refs/pull/36600/head:
	mds: make caps dump more readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2020-09-06 08:54:38 -07:00
Kefu Chai
dfe6dd96c2 crimson/osd: add "ceph tell <pgid> <command>" support
* add an abstract class of `PGCommand` for `ceph tell <pgid> <command>`
* add two sample implementations for the pg tell commands.
  - "query"
  - "mark_unfound_lost"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-09-06 21:49:34 +08:00
Changcheng Liu
57873ca779 crush/CrushWrapper: refine type_map/name_map comment
type_map is the item(bucket & device) type id and its type name.
name_map is the item(bucket & device) id and its name.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
2020-09-06 21:01:52 +08:00
Changcheng Liu
92d43dabbd crush/CrushWrapper: use "new_bucket_id" as var name of the added bucket
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
2020-09-06 21:01:52 +08:00
Changcheng Liu
0d87e967dd crush/CrushWrapper: align variable name with comment
In the prolog of "for loop" statement, it writes "add child in the
location with 0 weight"

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
2020-09-06 21:01:52 +08:00