Commit Graph

91120 Commits

Author SHA1 Message Date
Casey Bodley
594dc4cceb rgw: add rgw::putobj::ChunkProcessor and test
ChunkProcessor turns the input stream into a series of discrete chunks
before forwarding to the wrapped DataProcessor

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:01:36 -04:00
Casey Bodley
504b7d9c21 rgw: add rgw::putobj::DataProcessor interface and Pipe
adds an abstract DataProcessor interface (analogous to
RGWPutObjDataProcessor) that allows processors to be composed into
pipelines, and a Pipe class to support the existing filters for
compression and encryption

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-10-15 17:00:08 -04:00
Sage Weil
f22e55ad7a Merge PR #24473 into master
* refs/pull/24473/head:
	common: drop get_contiguous() from ceph::bufferlist.

Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-15 08:42:50 -05:00
Sage Weil
36c3bce219 Merge PR #24493 into master
* refs/pull/24493/head:
	mgr/DaemonState: clean up device life_expectancy values
	mgr/devicehealth: warn based on life_expectancy_max
	mgr/devicehealth: warn on failing devices at 6 weeks

Reviewed-by: John Spray <john.spray@redhat.com>
2018-10-15 08:36:18 -05:00
Lenz Grimmer
66db97eb6b
Merge pull request #24523 from s0nea/wip-dashboard-configs-table-cleanup
mgr/dashboard: config options table cleanup

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2018-10-15 15:14:52 +02:00
Tatjana Dehler
f4b2f4d91d mgr/dashboard: config options table cleanup
Remove columns 'tags', 'enum_values', 'long_desc', 'type', 'flags',
'daemon_default', 'desc', 'level', 'can_update_at_runtime', 'services',
'max', 'see_also', 'min' and 'source' from table view and add them to
the details.
The table contains 'name', 'value' and 'default' only.

Fixes: http://tracker.ceph.com/issues/34533

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2018-10-15 11:36:43 +02:00
John Spray
cf31e6f8a4
Merge pull request #24560 from sebastian-philipp/orchestrator-fix-rook-cluster-in-name
mgr/rook: Fix Rook cluster name detection

Reviewed-by: John Spray <john.spray@redhat.com>
2018-10-15 09:23:39 +01:00
Sage Weil
7ac6ab4b2f Merge PR #24494 into master
* refs/pull/24494/head:
	ceph-kvstore-tool: rename repair -> destructive-repair

Reviewed-by: Neha Ojha <nojha@redhat.com>
2018-10-14 13:11:11 -05:00
Sage Weil
59ab2aa6ab Merge PR #24146 into master
* refs/pull/24146/head:
	mgr/orchestrator: Fix Python 3 issues

Reviewed-by: John Spray <john.spray@redhat.com>
2018-10-14 11:42:17 -05:00
Sage Weil
8cc6369511 ceph-kvstore-tool: rename repair -> destructive-repair
This is shown to corrupt otherwise healthy rocksdb databases.  Rename to
make it clear that it is generally not safe to run and shoud only be used
as a last resort.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-14 11:41:24 -05:00
Sage Weil
b5ba057240 Merge PR #24247 into master
* refs/pull/24247/head:
	PendingReleaseNotes: add note about increased mon memory footprint
	doc/start/hardware-recommendations: refresh recommendations for RAM
	rocksdb: increase default cache size to 512 MB
	mon: mon_osd_cache_size = 500 (from 10)

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2018-10-14 11:40:15 -05:00
Sage Weil
b43dd18fc0 Merge PR #24495 into master
* refs/pull/24495/head:
	kv: apply the move semantics in KeyValueDB::get().

Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-14 08:06:18 -05:00
Sage Weil
bc52bc2f6d Merge PR #24535 into master
* refs/pull/24535/head:
	mon: consider AUTH_NONE clients which "authenticate" to be finished

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-14 08:05:10 -05:00
Mykola Golub
1d92788f71
Merge pull request #24563 from dillaman/wip-36410
test: move OpenStack devstack test to rocky release

Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-10-14 10:40:05 +03:00
Neha Ojha
05fea962f1
Merge pull request #24564 from neha-ojha/wip-21931
osd: do not overestimate the size of the object for reads with trimtrunc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>
2018-10-12 19:15:39 -07:00
Sage Weil
9db328f2ab Merge PR #24204 into master
* refs/pull/24204/head:
	qa/suites/rgw/tempest: valgrind on centos only

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
2018-10-12 16:29:44 -05:00
Sage Weil
b29e4470ac Merge PR #24249 into master
* refs/pull/24249/head:
	doc/rados/troubleshooting-mon: update mondb recovery script

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-10-12 16:29:28 -05:00
Sage Weil
b624cea60a Merge PR #24296 into master
* refs/pull/24296/head:
	osd: Handle is_stopping() by discarding item so it doesn't loop
	osd: Simplify _process() logic recently added

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-12 16:16:39 -05:00
Sage Weil
ec3a825db2 Merge PR #24131 into master
* refs/pull/24131/head:
	tests, common: introduce bufferlist::apennd benchmark.
	test, common: add BufferListIterator.BenchDeref benchmark.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2018-10-12 16:14:55 -05:00
Sage Weil
4aba9cc02f Merge PR #24270 into master
* refs/pull/24270/head:
	osd: make 'cache drop' command require 'executable' permission
	osd: rename 'drop cache' and 'get cache stats' to group them by component
	doc: add documentation for 'drop cache' and 'get cache stats'
	osd: don't print osdmap cache stats in 'get cache stats' command
	osd: do not clear osdmap cache on 'drop cache' command
	osd: offload dumping cache stats to the object store
	osd: pass a stream to flush_cache commands for more verbosity
	osd: implement flush_cache() method for Filestore
	osd: add clear_cache and get_cache_object_count commands

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-12 16:14:37 -05:00
Sage Weil
07e74f803f Merge PR #24312 into master
* refs/pull/24312/head:
	osd: kill the std::stringstream in ReplicatedBackend::do_repop_reply.
	osd: bump-up the dout level in PGLog::write_log_and_missing.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-12 16:14:19 -05:00
Sage Weil
4ae369119d Merge PR #24520 into master
* refs/pull/24520/head:
	install-deps.sh: dashboard frontend needs git

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-10-12 16:13:57 -05:00
Sage Weil
90c342b521 Merge PR #24521 into master
* refs/pull/24521/head:
	ceph.in: reinstate interactive mode output

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2018-10-12 16:13:37 -05:00
Neha Ojha
76c57810ee osd/PrimaryLogPG.cc: reassign size only when object size > truncate_size
Before setting size equal to op.extent.truncate_size, we need to check
if the size of the object is greater than the truncate_size. We do not
need to set size to op.extent.truncate_size, in the case where the size of
the object is less than op.extent.truncate_size.

Without this change, we were always setting size =
op.extent.truncate_size, when (seq < op.extent.truncate_seq) and
(op.extent.offset + op.extent.length > op.extent.truncate_size), were both
true. This ended up in:

1. overestimating the size of the object
2. not considering the correct size of the object, for
   the later checks, which calculate op.extent.length for the read ops
3. causing crashes when trying to read more data than what was present

Fixes: http://tracker.ceph.com/issues/21931
Fixes: http://tracker.ceph.com/issues/22330

Signed-off-by: Neha Ojha <nojha@redhat.com>
2018-10-12 12:49:07 -07:00
Neha Ojha
07f7bd6942 osd: print offset and length to track trimtrunc
Signed-off-by: Neha Ojha <nojha@redhat.com>
2018-10-12 12:47:39 -07:00
Matt Benjamin
9e58faf4a5
Merge pull request #24341 from joke-lee/WIP_FIX_RGWPOSTOBJ_VERSIONING
rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning
2018-10-12 13:02:33 -04:00
Jason Dillaman
1e3dc02604 qa/tasks/workunit: use suite branch/SHA1 when cloning workunits
Right now it's using the Ceph branch/SHA1 but it's using the suite
Git URL.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-10-12 12:41:58 -04:00
John Spray
ea75b20ce9
Merge branch 'master' into orchestrator-fix-rook-cluster-in-name 2018-10-12 16:56:51 +01:00
Sage Weil
a9b0490036 mgr/DaemonState: clean up device life_expectancy values
Instead of "0.000000" for empty time values, use "".

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-12 08:40:30 -05:00
Sage Weil
11bae94c43 mgr/devicehealth: warn based on life_expectancy_max
The failure interval needs to be sufficiently precise that it establishes
an approximate upper bound on the device life expectancy.

Also, deal with the fact that the max value may be '0.000000' intead of ''.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-12 08:38:05 -05:00
Sebastian Wagner
d66afa918f mgr/rook: Fix cluster name detection
`ROOK_CLUSTER_NAME` vanished from environment. Change to
'POD_NAMESPACE'.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2018-10-12 14:44:12 +02:00
Jason Dillaman
2a1dca3fca qa/workunits/rbd: switch devstack to rocky branch and tempest to 19.0.0 tag
Fixes: http://tracker.ceph.com/issues/36410
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-10-12 08:40:44 -04:00
Jason Dillaman
ba32395165
Merge pull request #24550 from dillaman/wip-36409
qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-10-12 08:39:37 -04:00
John Spray
27ea9ca821
Merge pull request #24517 from jcsp/wip-23970-test
mgr/selftest: fix disabled module selection

Reviewed-by: John Spray <john.spray@redhat.com>
2018-10-12 13:05:40 +01:00
Jan Fajerski
54499147ba
Merge pull request #24426 from Miouge1/grafana-fix1
monitoring/grafana: Fix OSD Capacity Utlization Grafana graph
2018-10-12 12:44:39 +02:00
Ricardo Marques
639c19e5fd
Merge pull request #24511 from a2batic/grafana-doc
mgr/dashboard: Improves documentation for Grafana Setting

Reviewed-by: Kai Wagner <kwagner@suse.com> 
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2018-10-12 10:19:48 +01:00
Ricardo Marques
cef650aa3d
Merge pull request #24513 from rhcs-dashboard/27047-landing-page-info-visibility
mgr/dashboard: Landing Page: info visibility

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2018-10-12 10:04:02 +01:00
Ricardo Marques
57b28c7fc5
Merge pull request #24222 from votdev/bug_36109
mgr/dashboard: The RGW backend doesn't handle IPv6 properly

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2018-10-12 10:01:39 +01:00
Jason Dillaman
c1f950236b qa/tasks: qemu task now uses a relative path in suite repo for test
This makes it easier to re-run tests against a suite branch without
requiring a full ceph-ci build and repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-10-11 20:23:39 -04:00
Matt Benjamin
9809e95308
Merge pull request #24402 from cbodley/wip-36302
librgw: initialize curl and http client for multisite
2018-10-11 15:51:49 -04:00
Sage Weil
f1dd265a36 Merge PR #24514 into master
* refs/pull/24514/head:
	mgr/restful: do not print warning message at seeing unknown request

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2018-10-11 14:35:45 -05:00
Jason Dillaman
ddb7e5eb80 qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh
Fixes: http://tracker.ceph.com/issues/36409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-10-11 15:25:20 -04:00
Sage Weil
29c9fbe534 Merge PR #24264 into master
* refs/pull/24264/head:
	tools/rados/rados.cc: fix rados rm --force-full blocking problem

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-10-11 14:24:12 -05:00
Sage Weil
28cca7bb5f Merge PR #24287 into master
* refs/pull/24287/head:
	osd/OSD: drop osd_lock only if pg osdmaps are truly laggy

Reviewed-by: Sage Weil <sage@redhat.com>
2018-10-11 14:22:44 -05:00
Sage Weil
65ea56d15b Merge PR #24331 into master
* refs/pull/24331/head:
	common: fix memory leaks in WeightedPriorityQueue.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2018-10-11 14:22:28 -05:00
Sage Weil
5ea5dc2b0e Merge PR #24365 into master
* refs/pull/24365/head:
	erasure-code: var-sized array init is done separatly.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-10-11 14:22:13 -05:00
Sage Weil
f5ffe4db69 Merge PR #24506 into master
* refs/pull/24506/head:
	doc/rados/troubleshooting: update notes on rebuilding mon db
	ceph-objectstore-tool: update-mon-db: remove pgmap references
	ceph-monstore-tool: remove remaining pgmap references, inflate-pgmap command
	ceph-monstore-tool: rebuild creating_pgs
	ceph-monstore-tool: rebuild: remove obsolete pgmap_meta fields

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-10-11 13:53:20 -05:00
Greg Farnum
f155851e5f mon: consider AUTH_NONE clients which "authenticate" to be finished
We check for AUTH_NONE success by examining the side effect of the caps being
filled in with allow_all, and we need that to be "finished" for purposes of
replying with a monmap as well.

Fixes: http://tracker.ceph.com/issues/36300

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2018-10-11 11:08:51 -07:00
Casey Bodley
4d8e2c0df6
Merge pull request #24409 from alimaredia/wip-rgw-dpp-auth
rgw: thread DoutPrefixProvider into RGW::Auth_S3::authorize

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-10-11 13:21:40 -04:00
Sage Weil
b371771383 doc/rados/troubleshooting-mon: update mondb recovery script
- some cleanup (e.g., use $ms throughput)
- behave if the local host is in the $hosts list (use $ms.remote)
- be clear about updating all mons
- mon.0 -> mon.foo

Signed-off-by: Sage Weil <sage@redhat.com>
2018-10-11 11:35:31 -05:00