Commit Graph

90340 Commits

Author SHA1 Message Date
Kefu Chai
9e7a0111da
Merge pull request #24108 from clebergnu/rm_qa_btrfs_binary
qa/btrfs/test_rmdir_async_snap: remove binary file

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-15 10:17:58 +08:00
Josh Durgin
53c3119d42
Merge pull request #23989 from dzafman/wip-35846
tools: ceph-objectstore-tool: Allow target level as first positional argument

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2018-09-14 16:07:09 -07:00
Radoslaw Zarzynski
184d0a75f9 osd: avoid unncecessary ref-counting in OSD::enqueue_op.
```
-    4,30%     0,32%  msgr-worker-0  ceph-osd  [.] OSD::enqueue_op
   - 3,98% OSD::enqueue_op
      + 2,40% OSD::ShardedOpWQ::_enqueue
      + 0,42% OpRequest::mark_flag_point
        0,37% operator new
        0,36% PerfCounters::tinc
      + 0,21% Mutex::Unlock
```

Annotate:
```
OSD::enqueue_op  /home/perf/src/ceph-rzarzynski3/build/bin/ceph-osd
       │93           if( px != 0 ) intrusive_ptr_add_ref( px );
       │       test   %rbx,%rbx
       │95   _ZN13PGOpQueueableC4E5spg_t():
       │175    explicit PGOpQueueable(spg_t pg) : pgid(pg) {}
  0,70 │       mov    %cl,0x18(%rax)
       │177  _ZN8PGOpItemC4E5spg_tN5boost13intrusive_ptrI9OpRequestEE():
       │204    PGOpItem(spg_t pg, OpRequestRef op) : PGOpQueueable(pg), op(op) {
       │       mov    %rdx,(%rax)
       │206  _ZN5boost13intrusive_ptrI9OpRequestEC4ERKS2_():
       │     ↓ je     166
       │94   _ZNSt13__atomic_baseIiEppEv():
 48,25 │       lock   addl   $0x1,0x30(%rbx)
       │166:   mov    0x158(%rbp),%ecx
       │       mov    0x15c(%rbp),%edx
       │299  _ZN17ShardedThreadPool9ShardedWQI11OpQueueItemE5queueEOS1_():
       │669        tp->set_wq(this);
       │670      }
       │671      ~ShardedWQ() override {}
       │
       │673      void queue(T&& item) {
       │674        _enqueue(std::move(item));
  2,80 │       mov    %r12,%rsi
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2018-09-14 21:39:10 +02:00
Casey Bodley
138be4fc53 crimson/net: messenger uses dispatch gate for authorizer callbacks
this also removes the checks for null dispatcher - start() must be
called with a valid dispatcher before getting here

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
cabc60aaec crimson/net: messenger uses dispatch gate for accept/connect callbacks
the dispatcher can throw to reject a new connection, so these callbacks
were moved ahead of the exception handling

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
2a978d5647 crimson/net: messenger uses dispatch gate for reset callbacks
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
35deda533e crimson/net: add unit test for concurrent dispatch
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
43abe8995a crimson/net: concurrent dispatch for SocketMessenger
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
f8d5eba4de crimson/net: don't return null from Connection::read_message()
SocketConnection::read_message() now loops until it has a message with
valid sequence number. this means SocketMessenger::dispatch() doesn't
have to handle the null message case

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:46:38 -04:00
Casey Bodley
a4019cdd45 crimson/net: use std::optional instead of boost
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:46:38 -04:00
Greg Farnum
47ed036753 msg/async: clean up local buffers on dispatch
The AsyncConnection keeps local (member variable) bufferlists of incoming
messages before they're placed into the Message's front/data/middle buffers.
Previously these were reset only when a new Message is being received, which
means in steady state we store a full Message for every Connection even if
it's inactive!

Instead we obviously want to drop our local references to Message state
once it's been dispatched, so that it can go away.

Fixes: http://tracker.ceph.com/issues/35987

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2018-09-14 10:58:49 -07:00
Cleber Rosa
f0f6e46084 qa/btrfs/test_rmdir_async_snap: remove binary file
On a quick look at the source code, I noticed this binary file, which
looks like was committed by mistake.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
2018-09-14 13:26:32 -04:00
Kefu Chai
07a3e8d7a6
Merge pull request #23650 from tchaikov/wip-dpdk-cmake
cmake: fix the build WITH_DPDK=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 23:44:01 +08:00
Jason Dillaman
862082792d librbd: test_flags helper should require snap id parameter
The HEAD and snapshots have potentially different flag states
since object maps get invalidated per revision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-14 11:09:23 -04:00
Sage Weil
0c75020cad common/hobject: add get_logical_pool()
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 08:44:37 -05:00
Sage Weil
cf4a4ebf27 common/hobject: define get_temp_pool() etc helpers
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 08:40:43 -05:00
Kefu Chai
b035df69f7
Merge pull request #24097 from cyx1231st/fix-seastar-msgr-read-racing
crimson/net: Fix racing for promise on_message

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-14 19:03:54 +08:00
Yingxin
6365e02926 crimson/net: Fix racing for promise on_message
read_tags_until_next_message() will forward the ready future and create
a new promise for on_message, which assumes there is already a
read_message() holding the previous promise, but it is not true.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>
2018-09-14 18:33:21 +08:00
Nathan Cutler
ee35f017fc
Merge pull request #23972 from badone/wip-documentation-add-12.2.8-release
doc/releases/schedule: Add Luminous 12.2.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-09-14 09:52:59 +02:00
Nathan Cutler
0e0c88d930
Merge pull request #24099 from badone/wip-mimic-13-2-0-date-correction
doc/releases: Mark Mimic first release as June

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-09-14 09:52:19 +02:00
xie xingguo
8fc491161b osd/PG: kill __queue_query__
which has no consumbers...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-14 13:33:47 +08:00
xie xingguo
8ef8cdba15 osd/OSD: deprecate __project_pg_history__
__project_pg_history__ does an inverse traverse of the series
of osdmaps passed in to get a pg's pg_history_t filled, which
can become super inefficient if the osdmap list to check is
very long.
E.g., in one of our clusters, we've observed it took approximate
10s for a PG to finish it's projecting:
```
2018-08-27 13:51:58.694823 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 from 821893 to 823276, start ec=380829/380829 l
is/c 820412/820412 les/c/f 820413/820413/0 821785/821785/821785
2018-08-27 13:52:08.634230 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 acting|up changed in 822265 from [57]/[57] 57/5
7 -> [58,57]/[58,57] 58/58
2018-08-27 13:52:08.634244 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 up changed in 822265 from [57] 57 -> [58,57] 58
2018-08-27 13:52:08.634248 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 primary changed in 822265
2018-08-27 13:52:08.634250 7f1e1335a700 15 osd.9 823276 project_pg_history end ec=380829/380829 lis/c 820412/820412 les/c/f 82041
3/820413/0 822265/822265/822265
```
Quote from Sage:
> let's just kill this off entirely, and make the handle_pg_query_nopg
reply unconditionally. Or, maybe, do a single sloppy check to see if
the primary has changed since the original epoch... if the osdmap
happens to be in cache... or not.
The querying end will discard the reply if it is out of date from it's
perspective, so it doesn't matter, and I suspect the overhead of doing
the check is larger than the overhead of sending a query reply that gets ignored.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: yanjun <yan.jun8@zte.com.cn>
2018-09-14 13:30:55 +08:00
Brad Hubbard
d109015436 doc/releases: Mark Mimic first release as June
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2018-09-14 15:05:52 +10:00
Brad Hubbard
5009d9ecc6 Rename "include/assert.h"
This conflicts with the system assert.h so rename and change includes to
reflect the new name.

Fixes: http://tracker.ceph.com/issues/35682

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2018-09-14 14:22:00 +10:00
Xie Xingguo
4e5ee667ce
Merge pull request #24094 from libingyang-zte/master
doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-14 10:46:48 +08:00
李丙洋 10208981
d1a38aa1ae doc: Fix Typos of Developer Guide
Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>
2018-09-14 09:23:22 +08:00
Brad Hubbard
b29c65623f mgr: Change signature of PyString_AsString to match return
PyUnicode_AsUTF8 now returns 'const char*'

Fixes: http://tracker.ceph.com/issues/35984

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2018-09-14 10:10:40 +10:00
Radoslaw Zarzynski
e8221ceb32 common: drop is_page_aligned() of buffer::raw entirely.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2018-09-13 22:54:24 +02:00
David Zafman
ef6940fbb6 test: osd-backfill-stats.sh: Fix subtests to get primary which can change
Fixes: http://tracker.ceph.com/issues/35982

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-13 13:19:23 -07:00
David Zafman
6d53e2c380 test: Fix for error message changed in ceph-objectstore-tool
Fixes: http://tracker.ceph.com/issues/35982

Caused by: 6bd682f53d

Signed-off-by: David Zafman <dzafman@redhat.com>
2018-09-13 13:19:11 -07:00
Radoslaw Zarzynski
3be42db16e common: drop/mark-as-final getters of buffer::raw for palign.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2018-09-13 22:01:36 +02:00
Sage Weil
0bd2546eac osd/osd_types: fix pg_t::contains() to check pool id too
This is used by bluestore fsck to ensure we're looking at the right
collection.  It needs to validate both that the pool id matches the
object (including the temp objects) and that the hash matches; we were
only checking the hash before.

Fixes: http://tracker.ceph.com/issues/32731
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-13 13:31:40 -05:00
Mykola Golub
ad9fc49776
Merge pull request #23896 from Songweibin/wip-group-snap-rollback
rbd: add group snap rollback method

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-09-13 19:35:10 +03:00
Andrew Schoen
641ff5e7c7
Merge pull request #24077 from alfredodeza/wip-rm35970
ceph-volume  fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>
2018-09-13 10:29:54 -05:00
Igor Fedotov
50919813a1 os/bluestore: fix bloom filter num entry miscalculation in the repairer
This could  cause an assertion due to an access to uninitialized bloom
filter. This happened  when detecting errors involving physical extents close to the end of a large partition.
E.g. it was observed in http://tracker.ceph.com/issues/25001 while trying to repair an OSD.

Fixes: https://tracker.ceph.com/issues/35971

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2018-09-13 17:24:26 +03:00
Alfredo Deza
11858c74a1 ceph-volume tests.functional comment why not use --destroy when zapping lvs in filestore
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2018-09-13 10:03:06 -04:00
Alfredo Deza
87af815691 ceph-volume tests.functional use --destroy when zapping lvs in bluestore
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2018-09-13 10:00:37 -04:00
Jason Dillaman
d3a1a831ac qa/workunits/rbd: new test for rbd-mirror peer config-key secrets
Fixes: http://tracker.ceph.com/issues/24688
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-13 09:58:34 -04:00
Jason Dillaman
900e30d1bb doc/rbd: update mirroring to include new peer secret config-key actions
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-13 09:58:34 -04:00
Jason Dillaman
5e768d5d18 mon: added bootstrap-rbd-mirror/rbd-mirror mon profiles
The rbd-mirror daemon will require permission to read config-keys
from the "rbd/mirror/" prefix. These new profiles (and the new
associated bootstrap-rbd-mirror user) provide the required
permissions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-13 09:58:34 -04:00
Jason Dillaman
58bddd6212 rbd: mirror pool peer commands now support mon_host/key overrides
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-13 09:58:33 -04:00
Kefu Chai
59e6fbbe70
Merge pull request #24024 from p-na/fix-flaky-qa-tests
mgr/dashboard: Fix flaky QA tests

Reviewed-by: Ricardo Dias <rdias@suse.com>
2018-09-13 21:52:35 +08:00
Alfredo Deza
036683fde7 ceph-volume lvm.zap use remove_pv only for physical devices
Signed-off-by: Alfredo Deza <adeza@redhat.com>
2018-09-13 09:34:54 -04:00
Casey Bodley
39f8375aee
Merge pull request #23660 from falcon78921/wip-multisite-docs
doc: updated multisite documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-09-13 08:53:59 -04:00
Kefu Chai
b61afd73e4
Merge pull request #23995 from tchaikov/wip-add-ceph-libboost-repo
install-deps,run-make-check: use ceph-libboost repo

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>
2018-09-13 19:40:26 +08:00
myoungwon oh
6ca2789a80 src/common: add a unit test (bufferlist.sha1())
Signed-off-by: Myoungwon Oh <omwmw@sk.com>
2018-09-13 17:07:45 +09:00
Xie Xingguo
d53dd387af
Merge pull request #24067 from libingyang-zte/master
doc: Fix typos in Developer Guide

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-13 12:12:47 +08:00
myoungwon oh
b8685d8552 osd, src/common: return sha1 value if zero-length buffer.
Signed-off-by: Myoungwon Oh <omwmw@sk.com>
2018-09-13 11:34:15 +09:00
李丙洋 10208981
da96cc919c doc: Fix Spelling Error in Developer Guide
Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>
2018-09-13 09:17:46 +08:00
Kefu Chai
3f93c42727
Merge pull request #23919 from adamemerson/wip-end-to-warnings
Clean up warnings

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-13 09:04:15 +08:00