Commit Graph

95828 Commits

Author SHA1 Message Date
Sage Weil
08d96fba1e osd/PG: do not touch this->cct after PG is destroyed
- thread A drops last second-to-last ref
- thread B drops last ref
- thread B deletes pg
- thread A does

  lgeneric_subdout(cct, refs, 1) << "PG::put " << this << " "
  ..

touching this->cct, a use-after-free that valgrind notices with

  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x74ED34</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>PG::put(char const*)</fn>
      <dir>/usr/src/debug/ceph-14.1.0-283-g569f086/src/osd</dir>
      <file>PG.cc</file>
      <line>179</line>
    </frame>
    <frame>
      <ip>0x7C1E76</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ContainerContext&lt;boost::intrusive_ptr&lt;PG&gt; &gt;::~ContainerContext()</fn>
      <dir>/usr/src/debug/ceph-14.1.0-283-g569f086/src/osd</dir>
      <file>PG.h</file>
      <line>566</line>
    </frame>
    <frame>
      <ip>0xC95675</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Finisher::finisher_thread_entry()</fn>
      <dir>/usr/src/debug/ceph-14.1.0-283-g569f086/src/common</dir>
      <file>Finisher.cc</file>
      <line>67</line>
    </frame>
    <frame>
      <ip>0xD656E24</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xE5B4BAC</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x470b29e8 is 152 bytes inside a block of size 11,632 free'd</auxwhat>
  <stack>
  ...

Fixes: http://tracker.ceph.com/issues/38484
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-04 09:13:55 -06:00
Casey Bodley
ee5f8ae082
Merge pull request #26682 from yuvalif/add_rgw_xml_unit_tests
Add rgw xml unit tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-03-04 10:04:17 -05:00
Casey Bodley
ab1b1a9fb9
Merge pull request #26680 from yuvalif/add_kill_9_to_mstop
mstop.sh allow kill -9 after failing to kill procs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-03-04 10:03:34 -05:00
Nathan Cutler
166ae7847c rpm: advance spec file copyright date
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-03-04 15:50:24 +01:00
Sage Weil
067f72f2a2 Merge PR #26704 into master
* refs/pull/26704/head:
	msg/async, v2: drop alloc_aligned_buffer().
	msg/async, v2: introduce frame late abort facility.
	Revert "msg/async, v2: move ceph_msg_header2 to last frame segment."
	msg, msg/async, v2: introduce late message abort facility.
	msg/async, v2: failure of msg decode doesn't block throtlles.
	msg/async, v2: move ceph_msg_header2 to last frame segment.
	fixup: use frame epilogue for crc32 integrity checking.
	msg/async, v2: epilogue size is variable in secure mode.
	msg/async, v2: drop support for the buggy rx_buffers mechanism.
	Revert "msg/async, v2: add flags field to frame's epilogue."
	msg/async, v2: add flags field to frame's epilogue.
	msg/async, v2: drop onwire_segment_t as epilogue had derogated it.
	msg, msg/async, v2: drop crc fields from ceph_msg_header2.
	msg/async, v2: use frame epilogue for crc32 integrity checking.
	msg/async, v2: clean the ProtocolV2::{front,middle,data} up.
	msg/async, v2: clean the ProtocolV2::epilogue up.
	msg/async, v2: move crypto processing to segment reader.
	msg/async, v2: handle epilogue separately from payload/data.
	msg/async, v2: dissect decryption from SignedEncryptedFrame.
	msg/async, v2: unify WAIT frames with other payload frames.
	msg/async, v2: implement epilogue handling in secure mode.
	msg/async, v2: message frames are pre-dispatched now.

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-04 08:44:56 -06:00
Casey Bodley
3729d6a985 test/rgw: fix race in test_rgw_reshard_wait
start the timer before scheduling the wait

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-03-04 09:44:26 -05:00
Sage Weil
634780fe07 Merge PR #26696 into master
* refs/pull/26696/head:
	osdc/Objecter: preserve read-into-existing-buffer behavior
	osdc/Objecter: don't use post_rx_buffer or revoke_rx_buffer at all
	msg/Connection: disable {post,revoke}_rx_buffer
	msg/async/ProtocolV1: disable rx_buffers behavior
	msg/simple/Pipe: disable rx_buffer code

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-04 08:32:18 -06:00
Sage Weil
0040c9915d Merge PR #26727 into master
* refs/pull/26727/head:
	qa/standalone/osd/pg-split-merge: fix import_after_merge_and_gap
	osd: add 'send_beacon' tell/asok command(s)

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-04 08:13:34 -06:00
Alfredo Deza
e90ebc3264 ceph-volume tests.simple replace testinfra command for py.test
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2019-03-04 09:05:07 -05:00
Alfredo Deza
c4597fc631 ceph-volume tests.lvm replace testinfra command for py.test
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2019-03-04 09:04:51 -05:00
Alfredo Deza
dc7501af69 ceph-volume tests.batch replace testinfra command for py.test
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2019-03-04 09:04:36 -05:00
Sebastian Wagner
d206992bd0 src/script: add run_mypy to run static type checking on Python code
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-03-04 14:35:00 +01:00
Changcheng Liu
839c07822a doc: correct vstart.sh file location
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
2019-03-04 16:55:31 +08:00
Mykola Golub
a277b5c1fe
Merge pull request #26683 from dillaman/wip-38500
librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@suse.com>
2019-03-04 10:43:35 +02:00
Radoslaw Zarzynski
fdcf7d3be8 msg/async, v2: drop alloc_aligned_buffer().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-04 01:51:57 +01:00
Radoslaw Zarzynski
2206a7f8e1 msg/async, v2: introduce frame late abort facility.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-04 01:50:14 +01:00
Sage Weil
533a9b6a07 Merge PR #26698 into master
* refs/pull/26698/head:
	common/str_map: better trim() impl using boost
	common/str_map: fix trim() on empty string

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-03-03 11:39:52 -06:00
Sage Weil
aae0e8c0c0 osdc/Objecter: preserve read-into-existing-buffer behavior
The libradosstriper::RadosStriperImpl::aio_read populates the target
outbl with a static buffer and relies on us reading into it.  This was
actually not reliable in the past (it could fail if the rx_buffers
optimization failed due to a retransmit or something else) but nevertheless
libradosstriper requires it to work *at all*.

Resolve this by modifying Objecter to copy the result into any provided
buffer at the lowest layer.  This should capture any other such user who
needed this behavior.

On the other hand, it will break any user who inadvertantly reads into a
non-empty bufferlist.  Given that any such user would already previously
have seen bad behavior due to the rx_buffers optimization, we expect
there to be 0 such instances.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-03 10:38:53 -06:00
Sage Weil
01316aa7bd qa/standalone/osd/pg-split-merge: fix import_after_merge_and_gap
This test introduces a map gap.  What *should* happen is that when there is
such a gap, we cannot import.  Previously, the test didn't reliably produce
a map gap at all, and didn't check that import failed--it verified that it
passed.

Fix the test so that it reliably produces a gap *and* reports
min_last_epoch_clean to the mon so we can trim.  Then verify we fail to
import, but can with --force.  But remove the pg again, because if we
force an import with a map gap the osd will refuse to start.

Fixes: http://tracker.ceph.com/issues/38525
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-03 10:23:27 -06:00
Sage Weil
6bcd631101 osd: add 'send_beacon' tell/asok command(s)
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-03 10:23:27 -06:00
J. Eric Ivancich
15c91a6eea
Merge pull request #25538 from mkogan1/wip-rgw-reshard-status
rgw: change the "rgw admin status" 'num_shards' output to signed int

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
2019-03-03 08:58:20 -05:00
Yuval Lifshitz
4780ef48ac rgw: add xml attributes test, override xml_handle_data
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-03-03 12:02:37 +02:00
Xie Xingguo
d793c3e67b
Merge pull request #26668 from xiexingguo/wip-38493
msg/async/Protocol*: send keep alive if existing wins

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-03 14:43:15 +08:00
Kefu Chai
b2da6c4e23
Merge pull request #26723 from liewegas/wip-38537
mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-03-03 10:09:14 +08:00
Radoslaw Zarzynski
fd211a234c Revert "msg/async, v2: move ceph_msg_header2 to last frame segment."
This reverts commit 69dbc92ba8.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-02 21:11:42 +01:00
Patrick Donnelly
e117299ab1
Merge PR #26726 into master
* refs/pull/26726/head:
	mds: Fix use of auto prototype, it is a non std extension

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-03-02 11:24:22 -08:00
Ricardo Dias
297d47640d
mgr/dashboard: nfs-ganesha: controller API documentation
Signed-off-by: Ricardo Dias <rdias@suse.com>
2019-03-02 15:18:12 +00:00
Sage Weil
5de132944a Merge PR #26720 into master
* refs/pull/26720/head:
	ceph-osd: infer objectstore type of 'type' file is missing

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-03-02 08:31:54 -06:00
Radoslaw Zarzynski
33c6cf2ab2 msg, msg/async, v2: introduce late message abort facility.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-02 14:19:39 +01:00
Radoslaw Zarzynski
7d4bd94008 msg/async, v2: failure of msg decode doesn't block throtlles.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-02 14:07:17 +01:00
Joao Eduardo Luis
6c097015bb ceph_features: deprecate MON_SINGLE_PAXOS
Signed-off-by: Joao Eduardo Luis <joao@suse.de>
2019-03-02 11:36:23 +00:00
Joao Eduardo Luis
a20ce5f427 osdmap: map versions to releases
Signed-off-by: Joao Eduardo Luis <joao@suse.de>
2019-03-02 11:36:23 +00:00
Joao Eduardo Luis
dfafdf3dae mon: retire some osd set/unset flags
Signed-off-by: Joao Eduardo Luis <joao@suse.de>
2019-03-02 11:36:23 +00:00
Xie Xingguo
46189eaa64
Merge pull request #26705 from dzafman/wip-23999
Improve docs osd_recovery_priority, osd_recovery_op_priority and related

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-02 09:43:33 +08:00
Brad Hubbard
c5036f912a
Merge pull request #26666 from badone/wip-kvstore_tool-ice
tools/ceph_kvstore_tool: Move summary output to print_summary

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-03-02 09:13:58 +10:00
David Zafman
f7bec341da common, doc: Improve docs osd_recovery_priority, osd_recovery_op_priority and related
Add option desciptions for osd_recovery_priority and osd_recovery_op_priority

Fixes: https://tracker.ceph.com/issues/23999

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-03-01 13:55:35 -08:00
Sage Weil
9c09ff2d0c ceph-osd: infer objectstore type of 'type' file is missing
The current value of osd_objectstore should have no bearing--we should
rely on the state in front of us.

- If there is a directory called current/, assume filestore.
- If there is a symlink called block, assume bluestore.
- If we see none of those, fail.

Fixes: http://tracker.ceph.com/issues/38517
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 15:51:38 -06:00
Noah Watkins
384d82a1bb mgr/dashboard: fix for using '::' on hosts without ipv6
CherryPy throws an uncaught exception using '::' when the host does not
have an ipv6 loopback address. This is currently the case in kubernetes.
This patch disables the routine in cherrypy that performs the validation
that requested server ports are actually bound.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2019-03-01 13:48:42 -08:00
Radoslaw Zarzynski
69dbc92ba8 msg/async, v2: move ceph_msg_header2 to last frame segment.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-01 22:47:53 +01:00
Sage Weil
b3eb525471 Merge PR #26692 into master
* refs/pull/26692/head:
	test/test_ipaddr: added test for pick_address when ipv4 and ipv6 are enabled
	common/pick_address: improves log message when no IP is found
	pick_address: error out in dual stack mode if both addr types cannot be provided

Reviewed-by: Sage Weil <sage@redhat.com>
2019-03-01 14:36:35 -06:00
Willem Jan Withagen
f94d5af52c mds: Fix use of auto prototype, it is a non std extension
Clang complains:
In file included from /home/jenkins/workspace/ceph-master/src/msg/Message.cc:128:
In file included from /home/jenkins/workspace/ceph-master/src/messages/MFSMap.h:20:
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:330:49: error: 'auto' not allowed in function prototype
  void modify_filesystem(fs_cluster_id_t fscid, auto&& fn)
                                                ^~~~
/home/jenkins/workspace/ceph-master/src/mds/FSMap.h:341:37: error: 'auto' not allowed in function prototype
  void modify_daemon(mds_gid_t who, auto&& fn)
                                    ^~~~

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2019-03-01 21:36:26 +01:00
Sage Weil
b1cc4d3444 doc/releases/nautilus: ask users to opt in to telemetry
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:06 -06:00
Sage Weil
d14c909408 doc/mgr/telemtry: update docs
- new commands
- more explicit summary of what we send
- multiple steps to enable, so that you can see what would be reported
  before opting in.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:04 -06:00
Sage Weil
0ca1a9cbb2 mgr/telemetry: drop config-set and config-show; add just show
Use normal config options to configure.

Add a 'telemetry show' command to show those options.

We lose the validation of the email address, since we are relying on the
generic config machinery, but I don't think we care.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:04 -06:00
Sage Weil
ec78dcf516 mgr/telemetry: make 'telemetry show' readable by a human
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:04 -06:00
Sage Weil
6ab90c9cb0 mgr/telemetry: add 'telemetry on' and 'telemetry off' commands
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:04 -06:00
Sage Weil
9320bdb8ac mgr/telemetry: off by default
This way a user can enable the module and look at the output before
deciding to send it to anyone.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:32:04 -06:00
Sage Weil
18ff587f1a osdc/Objecter: don't use post_rx_buffer or revoke_rx_buffer at all
Signed-off-by: Sage Weil <sage@redhat.com>
2019-03-01 14:12:37 -06:00
Radoslaw Zarzynski
4a12dd4a42 fixup: use frame epilogue for crc32 integrity checking.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-01 21:08:08 +01:00
Radoslaw Zarzynski
524202827b msg/async, v2: epilogue size is variable in secure mode.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-03-01 20:28:13 +01:00