Commit Graph

98004 Commits

Author SHA1 Message Date
Kefu Chai
1b414e8281
Merge pull request #27538 from iotcg/radosgw
init-radosgw: use ceph-conf to get cluster configuration value

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-05-06 13:29:46 +08:00
Kefu Chai
ba970314fe
Merge pull request #27885 from b-ranto/wip-restful-perf-counters
restful: Expose perf counters

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-05-04 13:28:13 +08:00
Kefu Chai
382a3efddf
Merge pull request #27891 from tchaikov/wip-mgr-pgp-num-change-throttling
mgr/DaemonServer: refactor pgp_num changes throttling

Reviewed-by: Sage Weil <sage@redhat.com>
2019-05-04 13:26:39 +08:00
Yuri Weinstein
bb4bceb3b2
Merge pull request #27960 from yuriw/wip-yuriw-crontab-master
qa/tests: removed all runs on ovh
2019-05-03 13:36:56 -07:00
J. Eric Ivancich
a7b4667954
Merge pull request #27896 from ivancich/wip-add-rgw-period-comments
rgw: add some comments to rgw code to help explain functionality

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-05-03 16:12:48 -04:00
Yuri Weinstein
e96f61a205
Merge pull request #27956 from yuriw/wip-yuriw-smoke-master
qa/tests - cleaned up distro settings
2019-05-03 13:06:11 -07:00
Yuri Weinstein
e699d9026d qa/tests: removed all runs on ovh
and consolidated `smoke` runs re: https://github.com/ceph/ceph/pull/27956

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-05-03 12:40:59 -07:00
J. Eric Ivancich
8111dd712b rgw: add some comments to rgw code to help explain functionality
Comments to explain the role of RGWPeriod and the use of prefixes in
gc log entries are added. Some other comments are reformatted for 80
columns.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2019-05-03 15:29:59 -04:00
Casey Bodley
10e9fa3df0
Merge pull request #27838 from yuvalif/wip-yuval-pubsub_teuthology-new
rgw/pubsub: add multisite pubsub tests to teuthology

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-05-03 14:49:25 -04:00
Yuri Weinstein
52fdf23026 qa/tests - cleaned up distro settings
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-05-03 08:21:42 -07:00
Lenz Grimmer
9b99ed9157
Merge pull request #27448 from ricardoasmarques/upgrade-ceph-iscsi-version
mgr/dashboard: Support ceph-iscsi config v9

Reviewed-by: Tiago Melo <tmelo@suse.com>
2019-05-03 13:10:23 +02:00
Nathan Cutler
9b5815f505
Merge pull request #27887 from smithfarm/wip-bci-len255plus
script/backport-create-issue: handle long Redmine issue names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-05-03 13:06:08 +02:00
Mykola Golub
7f80660068
Merge pull request #27895 from dillaman/wip-librbd-zero-copy
librbd: support zero-copy writes via the C API

Reviewed-by: Mykola Golub <mgolub@suse.com>
2019-05-03 10:19:06 +03:00
Kefu Chai
dfdb1d0810
Merge pull request #16929 from amitkumar50/cov-rgw-1
rgw: Initialize member variables in rgw_sync.h, rgw_rados.h

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-05-03 13:25:42 +08:00
Kefu Chai
88d2f975af
Merge pull request #27927 from tchaikov/wip-util.collect_sys_info-is-optional
test/common/test_util: skip it if /etc/os-release does not exist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-05-03 10:34:49 +08:00
Casey Bodley
7a28272c34
Merge pull request #27897 from ivancich/wip-add-editor-comments-rgw-services
rgw: add editor directive comments to rgw services source files

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-05-02 14:23:33 -04:00
Abhishek L
ae8fe24b6f
Merge pull request #27820 from smithfarm/wip-39361
rgw: drop cloud sync module logs attrs from the log

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2019-05-02 19:52:57 +02:00
Kefu Chai
66de7bd8cc test: do not conditionize on GTEST_HAS_PARAM_TEST
value-parameterized is supported on all platforms, so drop this check

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-03 01:38:27 +08:00
Kefu Chai
71f5f3e8da test: s/INSTANTIATE_TEST_CASE_P/INSTANTIATE_TEST_SUITE_P/
the former is deprecated

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-03 01:35:04 +08:00
Jason Dillaman
ade792ee03 qa/valgrind.supp: added false-positive in boost::queue::lockfree
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 12:59:57 -04:00
Kefu Chai
45629975ac test/rbd_mirror: add "using ::testing::DoAll"
we should `using` it before calling it using `DoAll`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-03 00:53:17 +08:00
Kefu Chai
61e4a521b2 test/common/test_util: skip it if /etc/os-release does not exist
some GNU/Linux distros do not ship this file, and we should not fail the
test on them.

inspired by
http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/ceph-skip-collect-sys-info-test.patch?id=48f19e60c4677e392ee2c23f28098cfcaf9d1710

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-03 00:09:15 +08:00
Kefu Chai
d4387d33ef cmake: do not install googletest
the new googletest added an option to allow its consumer to skip this,
so instead of patching googletest, let's use this option.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-02 23:52:45 +08:00
Kefu Chai
9acf335464 googletest: pick up GTEST_SKIP() changes
so we can use it to disable test at runtime

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-05-02 23:52:25 +08:00
Yuval Lifshitz
51fc8a40e3 rgw/pubsub: fix more test issues with teuthology failures
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-05-02 18:51:59 +03:00
Jason Dillaman
4876450faf librbd: use zero-copy on write via the C API when cache is disabled
It will also be disabled when using synchronous write API methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
308fcc3456 common/deleter: added missing headers
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
2856777613 librbd: allow AioCompletion objects to be blocked
This will be used when user-provided memory is wrapped into a
ceph::buffer::raw pointer to prevent its release prior to the
drop of its last reference internally.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
e2ee3cd50c librbd: AioCompletion fail path should complete in clean thread
Ensure that no internal librbd locks are being held when the
callbacks are invoked for AioCompletion::fail.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
4a2e2e08c0 librbd: simplify AioCompletion reference counting for 'fail' case
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
ef9d74720f librbd: switch to lock-free queue for event poll IO interface
'perf' shows several percent of CPU being wasted on lock contention
in the event poll interface. The 'fio' RBD engine uses this poll
IO interface by default when available.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
002afa0fe3 librbd: avoid using lock within AIO completion where possible
'perf' shows several percent of CPU is being utilized handling the
heavyweight locking semantics of AIO completion. With these changes,
the lock contention disappears.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
b5fc7ecaf7 librbd: remove special case for starting AioCompletion ops
All ops can be immediately started now that flush ops won't
accidentally block themselves.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Jason Dillaman
09e4127d5d librbd: simplify IO flush handling through AsyncOperation
Allow ImageFlushRequest to directly execute a flush call through
AsyncOperation. This will allow the flush to be directly linked
to its preceeding IOs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-05-02 09:30:45 -04:00
Sage Weil
6ad73b2d0f Merge PR #27871 into master
* refs/pull/27871/head:
	ceph_test_objectstore: add very_large_write test
	os/bluestore: fix aio pwritev lost data problem.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
2019-05-02 08:22:04 -05:00
Yuval Lifshitz
83e5571c87 rgw/pubsub: fix test issue with 3 zones
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-05-02 10:15:00 +03:00
Jason Dillaman
d712aac916
Merge pull request #27844 from wjwithagen/wjw-fix-src_common_bit_vector.hpp.diff
common: Clang requires a default constructor, but it can be empty

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-05-01 18:56:00 -04:00
Samuel Just
0147ac2221
Merge pull request #27874 from athanatos/sjust/wip-peering-refactor-forreview
Extract peering logic into a module for use in crimson

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-05-01 14:33:38 -07:00
Patrick Donnelly
012809cf58
Merge PR #27763 into master
* refs/pull/27763/head:
	common/PriorityCache: fix over-aggressive assert when mem limited

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-05-01 13:22:58 -07:00
Mark Nelson
75f60f3776 common/PriorityCache: fix over-aggressive assert when mem limited
Fixes: https://tracker.ceph.com/issues/39437
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2019-05-01 15:09:47 -05:00
Samuel Just
7508fffece PGStateUtils: improvements for PGStateHistory
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:29 -07:00
sjust@redhat.com
f3a22feace PeeringState: don't zero backfill target num_bytes on activation
834d3c19a7 preserves num_bytes
on backfill targets in order to estimate space required to complete
backill.  However, from activation until backfill reservation,
info.stats.stats.sum.num_bytes is persisted to disk as 0 messing
up future intervals.  Instead, preserve it in the info sent during
recovery and leave it alone in RequestBackfillPrio.

Additionally, it's possible for backfill to be preempted between
last_backfill=MAX being sent to the replica and Backfilled being
queued occuring.  In that case, the stats get on reservation
and the replica ends up with invalid stats.

Fixes: https://tracker.ceph.com/issues/39401
Signed-off-by: sjust@redhat.com <sjust@redhat.com>
2019-05-01 11:22:29 -07:00
Samuel Just
25904513c2 PeeringState: use ceph_assert
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:29 -07:00
Samuel Just
7915c89e19 admin/build-doc: use PeeringState* for gen_state_diagram.py
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:29 -07:00
Samuel Just
f78bd13f24 PeeringState: add explanations for public interface methods
Also rearranges the methods a little for clarity.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:29 -07:00
sjust@redhat.com
6858ec29ca PG,PeeringState: Fix initialization order
PeeringState needs to be initialized last and destructed
first.

Signed-off-by: sjust@redhat.com <sjust@redhat.com>
2019-05-01 11:22:28 -07:00
Samuel Just
4a0f770d6e PeeringState: mark state and helpers private
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:28 -07:00
Samuel Just
e3fe19cd64 osd/: clean up remaining info mutators
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:28 -07:00
Samuel Just
d33a8b8ab1 osd/: condense missing mutations for recovery/repair/errors
At a high level, this patch attempts to unify the various
sites at which some combination of
- mark object missing in one or more pg_missing_t
- mark object needs_recovery in missing_loc
- manipulate the locations map based on external information
occur.  It seems to me that the pg_missing_t and missing_loc
should be in sync except for the mark_unfound_lost_revert
case and the case where we are about to do a backfill push.

This patch also cleans up repair_object.  It sort of worked by accident
for non-ec non-primary bad peers.  It didn't update missing_loc, so
needs_recovery() returns the wrong answer.  However, is_unfound() does
as well, so ReplicatedBackend is nevertheless happy as the object would
be present on the primary.  This patch updates the behavior to be
uniform as in the other force_obejct_missing cases.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:28 -07:00
Samuel Just
5ea5c47152 test-erasure-eio: first eio may be fixed during recovery
The changes to the way EC/ReplicatedBackend communicate read
t showerrors had a side effect of making first eio on the object in
TEST_rados_get_subread_eio_shard_[01] repair itself depending
on the timing of the killed osd recovering.  The test should
be improved to actually test that behavior at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-01 11:22:28 -07:00