Commit Graph

90375 Commits

Author SHA1 Message Date
Kefu Chai
237bc951c4
Merge pull request #24061 from liewegas/wip-35923
osd/PG: write pg epoch when resurrecting pg after delete vs merge race

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-17 15:52:49 +08:00
Kefu Chai
3d1d337428
Merge pull request #23306 from penglaiyxy/msg_branch
msg: ceph_abort() when there are enough accepter errors in msg server

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-17 15:50:57 +08:00
Kefu Chai
f46523e464
Merge pull request #23955 from wjwithagen/wjw-fix-ceph-helpers.sh
test: Start using GNU awk and fix archiving directory

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-17 15:44:06 +08:00
Patrick Nawracay
f95f231037 mgr/dashboard: Cleanup
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-17 08:25:41 +02:00
Patrick Nawracay
8ba41c5e2b mgr/dashboard: Refactoring of DeletionModalComponent
- Simpler variable names:

    Examples:

	- `actionDescription` and `itemDescription` instead of `metaType`
	- `bodyTemplate` instead of `description`
	- `validationPattern` instead of `pattern`

    Some of these variable names have been generalized to ease the
    unification/generalization of dialog components:

	- `submitAction` instead of `deletionMethod`

- Removed unique `setUp` method.

    Benefits:

    - Creation of the component is done as intended by the developers of
    the `ngx-boostrap` package and as expected by developers which use
    the package. The `setUp` method does not have to be called anymore
    on the `DeletionModalComponent` exclusively but instead the
    component is instantiated as all other modals. Property assignment
    on the instantiated object isn't handled by the `setUp` method
    anymore but by the `modalService`.

    - With the removal of the `setUp` method, some tests could be
    removed as well.

    - No need to pass the reference of the created modal to the modal
    manually.

    Preserved:

    - The provided check within the `setUp` method, which checked if the
    component had been correctly instantiated, has been moved to the
    `ngOnInit` method of the component.

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>
2018-09-17 08:25:41 +02:00
Kefu Chai
e69e50a859 rpm: use updated gperftools
make sure we only build with the higher version of gperftools on
distros where both 2.4 and 2.6.1 are packaged. see
https://git.centos.org/summary/rpms!gperftools.git . at the time of
writing, gperftools 2.6.1 is packaged for CentOS/RHEL 7, if gperftools
(>= 2.4) is required by Ceph, and user already has this version
installed, when new Ceph packages are installed, the updated gperftools
2.6.1 version won't be installed as a dependency. when launching
Ceph compiled with tcmalloc enabled, we will have

symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm

so, by bumping up the required version of gperftools, the updated
gperftools will be installed.

see https://software.opensuse.org/package/gperftools, openSUSE/SLE offer
2.5. so they are safe at this moment.

Fixes: http://tracker.ceph.com/issues/35969
Signed-off-by: Kefu Chai <kchai@redhat.com>
2018-09-17 14:20:51 +08:00
李丙洋 10208981
64613065ac doc: Fix Typos of Developer Guide
Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>
2018-09-17 09:59:29 +08:00
Patrick Donnelly
0ac789c435
Merge PR #24023 into master
* refs/pull/24023/head:
	mds: prevent rctime from going back

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-16 15:01:01 -07:00
Patrick Donnelly
b43f26c1d9
Merge PR #24022 into master
* refs/pull/24022/head:
	client: update ctime when modifying file content

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-16 14:58:38 -07:00
Patrick Donnelly
1b3b2744f9
Merge PR #24048 into master
* refs/pull/24048/head:
	mds: use common send message mechanisms
	mds: remove dead code

Reviewed-by: Zheng Yan <zyan@redhat.com>
2018-09-16 14:51:17 -07:00
Patrick Donnelly
cd1ce232ba
Merge PR #24053 into master
* refs/pull/24053/head:
	mds: translate CEPH_MDS_STATE_NULL state name as null

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-16 14:33:55 -07:00
Patrick Donnelly
ea43ce7aae
Merge PR #21992 into master
* refs/pull/21992/head:
	qa: extend timeout for SessionMap flush

Reviewed-by: John Spray <john.spray@redhat.com>
2018-09-16 14:30:28 -07:00
Jason Dillaman
0a31c55ea8 librbd: do not invalidate object map when attempting to delete non-existent snapshot
If duplicate snapshot remove requests are received by the lock owner from a peer
client, the first request will remove the object map. If the second request
arrives while the first is in-progress, it will again attempt to remove the
object map but fail to load it since it's already been deleted. This incorrectly
results in the next object map being flagged as invalid.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-16 12:46:25 -04:00
Jason Dillaman
58770188ab librbd: converted object map snapshot remove state machine to new style
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-16 12:46:25 -04:00
Kefu Chai
28d9982bde
Merge pull request #24029 from smithfarm/wip-35922
cmake: no libradosstriper headers if WITH_LIBRADOSSTRIPER=OFF

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
2018-09-17 00:04:03 +08:00
Mykola Golub
a68be2b452
Merge pull request #24036 from dillaman/wip-24688
rbd-mirror: optionally extract peer secrets from config-key

Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-09-16 11:53:04 +03:00
Mykola Golub
fa7bb832a3
Merge pull request #24063 from dillaman/wip-24140
rbd-mirror: prevent creation of clones when parents are syncing

Reviewed-by: Mykola Golub <mgolub@suse.com>
2018-09-16 11:52:32 +03:00
Patrick Donnelly
9c10509092
log: use smaller buffer for ConcreteEntry
This brings down the static size of the memory used by the logging infrastructure:

    If we used 1024, we'd have 1088*10000 = 10880000 = 10MB in use by the ring
    buffer and 2*1088*100 = 2*108800 = 2*106KB for the m_new and m_flush
    vectors.

In my testing, 1024 covers most log entries.

Note, I've kept 4096 for the StackStringStream via MutableEntry as these are
already allocated on the heap and cached in a thread local vector. Generally
there should only be about a dozen of these allocated so it's worth keeping a
larger buffer.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-14 19:46:12 -07:00
Patrick Donnelly
b86994e3f9
common: remove obsolete buffer classes
In favor of the simpler StackStreamBuffer.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-14 19:46:11 -07:00
Patrick Donnelly
a747aeac13
log: avoid heap allocations for most log entries
Each log Entry now exists on the stack and uses a large (4k) buffer for its log
stream. This Entry is std::move'd to the queues (std::vector and
boost::circular_buffer) in the Log, involving only memory copies in the general
case. There are two memory copies (std::move) for any given Entry, once in
Log::submit_entry and again in Log::_flush

In practice, this eliminates 100% of allocations outside of startup
allocations

I've run a simple experiment with the MDS that copies /usr/bin to CephFS. I got
measurements for the number of allocations from the heap profiler and the
profile of CPU usage in the MDS.

** Before this patch **

    == Heap profile: ==

    $ google-pprof --alloc_objects --text bin/ceph-mds out/mds.a.profile.0001.heap
    Total: 1105048 objects
      433329  39.2%  39.2%   433329  39.2% ceph::logging::Log::create_entry
      209311  18.9%  58.2%   209311  18.9% __gnu_cxx::__aligned_membuf::_M_addr (inline)
      192963  17.5%  75.6%   192963  17.5% __gnu_cxx::new_allocator::allocate (inline)
       61774   5.6%  81.2%    61774   5.6% std::__cxx11::basic_string::_M_mutate
       37689   3.4%  84.6%    37689   3.4% ceph::buffer::raw_combined::create (inline)
       22773   2.1%  86.7%    22773   2.1% mempool::pool_allocator::allocate (inline)
       17761   1.6%  88.3%    20523   1.9% std::pair::pair (inline)
       15795   1.4%  89.7%    15797   1.4% std::swap (inline)
       11011   1.0%  90.7%   130061  11.8% std::__cxx11::list::_M_insert (inline)
       10822   1.0%  91.7%    10822   1.0% std::__cxx11::basic_string::reserve
        9108   0.8%  92.5%    32721   3.0% __gnu_cxx::new_allocator::construct (inline)
        8608   0.8%  93.3%     8610   0.8% std::_Deque_base::_M_initialize_map (inline)
        7694   0.7%  94.0%     7694   0.7% std::__cxx11::basic_string::_M_capacity (inline)
        6160   0.6%  94.5%     6160   0.6% Journaler::wrap_finisher
        6084   0.6%  95.1%    70892   6.4% std::map::operator[] (inline)
        5347   0.5%  95.6%     5347   0.5% MutationImpl::add_projected_fnode
        4381   0.4%  96.0%     7706   0.7% mempool::pool_allocator::construct (inline)
        3588   0.3%  96.3%   182966  16.6% Locker::_do_cap_update
        3049   0.3%  96.6%     5280   0.5% std::__shared_count::__shared_count (inline)
        3043   0.3%  96.9%     3043   0.3% MDSLogContextBase::MDSLogContextBase (inline)
        3038   0.3%  97.1%    14763   1.3% std::__uninitialized_copy::__uninit_copy (inline)

   So approximately 430k heap allocations for Entry were created! The
   basic_string::_M_mutate is also another allocation via PrebufferedStreambuf

	== Profile data ==

	Selecting interesting functions

Samples: 798K of event 'cycles:pp', Event count (approx
  Children      Self  Com  Shared Object        Symbol
+    1.04%     1.04%  log  libceph-common.so.0  [.] ceph::logging::Log::_flush
+    0.05%     0.05%  log  libceph-common.so.0  [.] ceph::logging::Log::flush
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::_log_safe_write
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::entry
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::_flush_logbuf
...

  Children      Self  Command         Shared Object        Symbol
+    3.69%     0.00%  safe_timer      libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.53%     0.00%  ms_dispatch     libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.13%     0.00%  fn_anonymous    libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.00%     0.00%  ms_dispatch     libceph-common.so.0  [.] CachedPrebufferedStreambuf::create
+    0.00%     0.00%  fn_anonymous    libceph-common.so.0  [.] CachedPrebufferedStreambuf::create

  Children      Self  Command         Shared Object        Symbol
+    0.07%     0.07%  fn_anonymous    libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  ms_dispatch     libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  ms_dispatch     ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  md_submit       libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  fn_anonymous    ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  safe_timer      libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  mds_rank_progr  libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  mds_rank_progr  ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  msgr-worker-0   libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  msgr-worker-2   libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  md_submit       ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  msgr-worker-1   libceph-common.so.0  [.] ceph::logging::Log::create_entry

  Children      Self  Command         Shared Object        Symbol
+    8.29%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    7.55%     1.46%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::_M_insert<long>
+    3.87%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.92%     0.00%  md_submit       libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.38%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.22%     2.22%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::sentry::sentry
+    1.89%     0.13%  fn_anonymous    libstdc++.so.6.0.24  [.] std::__ostream_insert<char, std::char_traits<char> >
+    0.71%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    0.39%     0.06%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::_M_insert<long>
+    0.29%     0.21%  ms_dispatch     libstdc++.so.6.0.24  [.] std::__ostream_insert<char, std::char_traits<char> >
+    0.27%     0.27%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::sentry::~sentry
+    0.27%     0.27%  fn_anonymous    libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>
+    0.22%     0.22%  ms_dispatch     libstdc++.so.6.0.24  [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
+    0.20%     0.20%  ms_dispatch     libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>
+    0.15%     0.15%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::sentry::~sentry
+    0.14%     0.14%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::sentry::sentry
+    0.13%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::_M_insert<unsigned long>
+    0.13%     0.13%  fn_anonymous    libstdc++.so.6.0.24  [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
+    0.00%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::_M_insert<unsigned long>
+    0.00%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<unsigned long>

    And the unittest_log time:

    $ bin/unittest_log
    [==========] Running 15 tests from 1 test case
    [----------] Global test environment set-up
    [----------] 15 tests from Log
    [ RUN      ] Log.Simple
    [       OK ] Log.Simple (0 ms)
    [ RUN      ] Log.ReuseBad
    [       OK ] Log.ReuseBad (1 ms)
    [ RUN      ] Log.ManyNoGather
    [       OK ] Log.ManyNoGather (0 ms)
    [ RUN      ] Log.ManyGatherLog
    [       OK ] Log.ManyGatherLog (12 ms)
    [ RUN      ] Log.ManyGatherLogStringAssign
    [       OK ] Log.ManyGatherLogStringAssign (27 ms)
    [ RUN      ] Log.ManyGatherLogStringAssignWithReserve
    [       OK ] Log.ManyGatherLogStringAssignWithReserve (27 ms)
    [ RUN      ] Log.ManyGatherLogPrebuf
    [       OK ] Log.ManyGatherLogPrebuf (15 ms)
    [ RUN      ] Log.ManyGatherLogPrebufOverflow
    [       OK ] Log.ManyGatherLogPrebufOverflow (15 ms)
    [ RUN      ] Log.ManyGather
    [       OK ] Log.ManyGather (8 ms)
    [ RUN      ] Log.InternalSegv

    [WARNING] /home/pdonnell/cephfs-shell/src/googletest/googletest/src/gtest-death-test.cc:836:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads
    [       OK ] Log.InternalSegv (8 ms)
    [ RUN      ] Log.LargeLog
    [       OK ] Log.LargeLog (43 ms)
    [ RUN      ] Log.TimeSwitch
    [       OK ] Log.TimeSwitch (1 ms)
    [ RUN      ] Log.TimeFormat
    [       OK ] Log.TimeFormat (0 ms)
    [ RUN      ] Log.Speed_gather
    [       OK ] Log.Speed_gather (1779 ms)
    [ RUN      ] Log.Speed_nogather
    [       OK ] Log.Speed_nogather (64 ms)
    [----------] 15 tests from Log (2000 ms total)

    [----------] Global test environment tear-down
    [==========] 15 tests from 1 test case ran. (2000 ms total)
    [  PASSED  ] 15 tests

** After Patch **

The StackStreamBuf uses 4k for its default buffer. This appears to be more than
reasonable for preventing allocations for logging

    == Heap profile: ==

	$ google-pprof --alloc_objects --text bin/ceph-mds out/mds.a.profile.0001.heap
	Total: 1052127 objects
      384957  36.6%  36.6%   384957  36.6% __gnu_cxx::new_allocator::allocate (inline)
      274720  26.1%  62.7%   274720  26.1% __gnu_cxx::__aligned_membuf::_M_addr (inline)
       68496   6.5%  69.2%    68496   6.5% std::__cxx11::basic_string::_M_mutate
       44140   4.2%  73.4%    51828   4.9% std::pair::pair (inline)
       43091   4.1%  77.5%    43091   4.1% ceph::buffer::raw_combined::create (inline)
       27706   2.6%  80.1%   236407  22.5% std::__cxx11::list::_M_insert (inline)
       25699   2.4%  82.6%    25699   2.4% std::__cxx11::basic_string::reserve
       23183   2.2%  84.8%    23183   2.2% mempool::pool_allocator::allocate (inline)
       19466   1.9%  86.6%    81716   7.8% __gnu_cxx::new_allocator::construct (inline)
       17606   1.7%  88.3%    17606   1.7% std::__cxx11::basic_string::_M_capacity (inline)
       16879   1.6%  89.9%    16881   1.6% std::swap (inline)
        8572   0.8%  90.7%     8574   0.8% std::_Deque_base::_M_initialize_map (inline)
        8477   0.8%  91.5%    11808   1.1% mempool::pool_allocator::construct (inline)
        6166   0.6%  92.1%     6166   0.6% Journaler::wrap_finisher
        6080   0.6%  92.7%   134975  12.8% std::map::operator[] (inline)
        6079   0.6%  93.3%     6079   0.6% MutationImpl::add_projected_fnode

	== Profile data ==

    Samples: 62K of event 'cycles:u', Event count (approx.)
      Overhead  Command         Shared Object         Symbol
    +    5.91%  log             libc-2.23.so          [.] vfprintf
    +    5.75%  ms_dispatch     libstdc++.so.6.0.24   [.] std::__ostream_insert<char, std::char_traits<char> >
    +    4.59%  ms_dispatch     ceph-mds              [.] StackStringBuf<4096ul>::xsputn
    +    4.26%  ms_dispatch     libc-2.23.so          [.] __memmove_ssse3_back
    +    4.07%  log             libceph-common.so.0   [.] ceph::logging::Log::_flush
    +    2.48%  ms_dispatch     libstdc++.so.6.0.24   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<l
    +    2.13%  fn_anonymous    libstdc++.so.6.0.24   [.] std::__ostream_insert<char, std::char_traits<char> >
    +    2.09%  ms_dispatch     ceph-mds              [.] CDir::check_rstats
    +    2.06%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::sentry::sentry
    +    1.98%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::sentry::~sentry
    +    1.87%  log             libc-2.23.so          [.] __strcpy_sse2_unaligned
    +    1.60%  fn_anonymous    ceph-mds              [.] StackStringBuf<4096ul>::xsputn
    +    1.46%  log             libc-2.23.so          [.] _IO_default_xsputn
    +    1.45%  log             libc-2.23.so          [.] _itoa_word
    +    1.43%  fn_anonymous    libc-2.23.so          [.] __memmove_ssse3_back
    +    1.40%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::_M_insert<long>
    +    0.98%  fn_anonymous    libstdc++.so.6.0.24   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<l
    +    0.89%  ms_dispatch     libstdc++.so.6.0.24   [.] 0x
    +    0.88%  ms_dispatch     libstdc++.so.6.0.24   [.] std::_Rb_tree_increment

    And the unittest_log time:

    $ bin/unittest_log
    [==========] Running 13 tests from 1 test case.
    [----------] Global test environment set-up.
    [----------] 13 tests from Log
    [ RUN      ] Log.Simple
    [       OK ] Log.Simple (1 ms)
    [ RUN      ] Log.ReuseBad
    [       OK ] Log.ReuseBad (0 ms)
    [ RUN      ] Log.ManyNoGather
    [       OK ] Log.ManyNoGather (0 ms)
    [ RUN      ] Log.ManyGatherLog
    [       OK ] Log.ManyGatherLog (83 ms)
    [ RUN      ] Log.ManyGatherLogStringAssign
    [       OK ] Log.ManyGatherLogStringAssign (79 ms)
    [ RUN      ] Log.ManyGatherLogStackSpillover
    [       OK ] Log.ManyGatherLogStackSpillover (81 ms)
    [ RUN      ] Log.ManyGather
    [       OK ] Log.ManyGather (80 ms)
    [ RUN      ] Log.InternalSegv

    [WARNING] /home/pdonnell/ceph/src/googletest/googletest/src/gtest-death-test.cc:836:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads.
    [       OK ] Log.InternalSegv (7 ms)
    [ RUN      ] Log.LargeLog
    [       OK ] Log.LargeLog (55 ms)
    [ RUN      ] Log.TimeSwitch
    [       OK ] Log.TimeSwitch (4 ms)
    [ RUN      ] Log.TimeFormat
    [       OK ] Log.TimeFormat (1 ms)
    [ RUN      ] Log.Speed_gather
    [       OK ] Log.Speed_gather (1441 ms)
    [ RUN      ] Log.Speed_nogather
    [       OK ] Log.Speed_nogather (63 ms)
    [----------] 13 tests from Log (1895 ms total)

    [----------] Global test environment tear-down
    [==========] 13 tests from 1 test case ran. (1895 ms total)
    [  PASSED  ] 13 tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2018-09-14 19:46:08 -07:00
Kefu Chai
9e7a0111da
Merge pull request #24108 from clebergnu/rm_qa_btrfs_binary
qa/btrfs/test_rmdir_async_snap: remove binary file

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-15 10:17:58 +08:00
Josh Durgin
53c3119d42
Merge pull request #23989 from dzafman/wip-35846
tools: ceph-objectstore-tool: Allow target level as first positional argument

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2018-09-14 16:07:09 -07:00
Robin H. Johnson
3b864482d6
rgw: fix chunked-encoding for chunks >1MiB
For HTTP responses sent with chunked-encoding, and greater than 1MiB in
size, the chunk-size field was being printed wrong.

Specifically, the chunk-size field was being sent with a mangled or
missing trailer of '\r\n'.

This bug manifested as HTTP clients being unable to read the response:
Chrome generates ERR_INCOMPLETE_CHUNKED_ENCODING
Python/boto generates httplib.LineTooLong: got more than 65536 bytes when reading chunk size

The wrong variable was being used to determine the size of the buffer
used for the chunk-size field.

Fix it by using the correct variable, and rename the variables to
clearly reflect their purpose.

Prior to PR#23940, this would only have been seen in some Swift
operations. PR#23940 changed some S3 operations to also use chunked
encoding to get responses sent faster, and made the bug easier to
detect. It was initially reported for a ListBucket call with a high
max-keys argument.

Backport: luminous, mimic
Reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
Reference: https://github.com/ceph/ceph/pull/23940
Fixes: http://tracker.ceph.com/issues/35990
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>
2018-09-14 14:38:24 -07:00
Radoslaw Zarzynski
184d0a75f9 osd: avoid unncecessary ref-counting in OSD::enqueue_op.
```
-    4,30%     0,32%  msgr-worker-0  ceph-osd  [.] OSD::enqueue_op
   - 3,98% OSD::enqueue_op
      + 2,40% OSD::ShardedOpWQ::_enqueue
      + 0,42% OpRequest::mark_flag_point
        0,37% operator new
        0,36% PerfCounters::tinc
      + 0,21% Mutex::Unlock
```

Annotate:
```
OSD::enqueue_op  /home/perf/src/ceph-rzarzynski3/build/bin/ceph-osd
       │93           if( px != 0 ) intrusive_ptr_add_ref( px );
       │       test   %rbx,%rbx
       │95   _ZN13PGOpQueueableC4E5spg_t():
       │175    explicit PGOpQueueable(spg_t pg) : pgid(pg) {}
  0,70 │       mov    %cl,0x18(%rax)
       │177  _ZN8PGOpItemC4E5spg_tN5boost13intrusive_ptrI9OpRequestEE():
       │204    PGOpItem(spg_t pg, OpRequestRef op) : PGOpQueueable(pg), op(op) {
       │       mov    %rdx,(%rax)
       │206  _ZN5boost13intrusive_ptrI9OpRequestEC4ERKS2_():
       │     ↓ je     166
       │94   _ZNSt13__atomic_baseIiEppEv():
 48,25 │       lock   addl   $0x1,0x30(%rbx)
       │166:   mov    0x158(%rbp),%ecx
       │       mov    0x15c(%rbp),%edx
       │299  _ZN17ShardedThreadPool9ShardedWQI11OpQueueItemE5queueEOS1_():
       │669        tp->set_wq(this);
       │670      }
       │671      ~ShardedWQ() override {}
       │
       │673      void queue(T&& item) {
       │674        _enqueue(std::move(item));
  2,80 │       mov    %r12,%rsi
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2018-09-14 21:39:10 +02:00
Sage Weil
3e791d1f9e doc: update docs for device management
Move this out of mgr section and into rados operations.

Leave a placeholder for failure prediction, which is still a work in
progress.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 14:06:41 -05:00
Sage Weil
5d68f77360 mgr: make devicehealth always-on
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 14:05:57 -05:00
Casey Bodley
764d6a8599 rgw: abort_bucket_multiparts() ignores individual NoSuchUpload errors
if the bucket index lists multipart meta objects that don't actually
exist in rados, this error prevents the bucket from being deleted

Fixes: http://tracker.ceph.com/issues/35986

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 15:05:07 -04:00
Casey Bodley
138be4fc53 crimson/net: messenger uses dispatch gate for authorizer callbacks
this also removes the checks for null dispatcher - start() must be
called with a valid dispatcher before getting here

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
cabc60aaec crimson/net: messenger uses dispatch gate for accept/connect callbacks
the dispatcher can throw to reject a new connection, so these callbacks
were moved ahead of the exception handling

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
2a978d5647 crimson/net: messenger uses dispatch gate for reset callbacks
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
35deda533e crimson/net: add unit test for concurrent dispatch
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
43abe8995a crimson/net: concurrent dispatch for SocketMessenger
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:47:32 -04:00
Casey Bodley
f8d5eba4de crimson/net: don't return null from Connection::read_message()
SocketConnection::read_message() now loops until it has a message with
valid sequence number. this means SocketMessenger::dispatch() doesn't
have to handle the null message case

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:46:38 -04:00
Casey Bodley
a4019cdd45 crimson/net: use std::optional instead of boost
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 14:46:38 -04:00
Greg Farnum
47ed036753 msg/async: clean up local buffers on dispatch
The AsyncConnection keeps local (member variable) bufferlists of incoming
messages before they're placed into the Message's front/data/middle buffers.
Previously these were reset only when a new Message is being received, which
means in steady state we store a full Message for every Connection even if
it's inactive!

Instead we obviously want to drop our local references to Message state
once it's been dispatched, so that it can go away.

Fixes: http://tracker.ceph.com/issues/35987

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2018-09-14 10:58:49 -07:00
Sage Weil
a9cd3d172e mgr/devicehealth: do not create metrics pool on get-device-metrics
If we are just reading and the pool doesn't exist, no need to create it.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 12:53:13 -05:00
Sage Weil
8d7bf77586 mgr/devicehealth: converge OPTIONS and DEFAULTS
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 12:44:40 -05:00
Sage Weil
230e293a33 mgr/devicehealth: squelch health warnings for unused devices
We may also want to do the same for OSDs that are out and fully offloaded,
but that can wait until those OSDs go into a special "drive device to
failure" mode.

Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 12:43:58 -05:00
Sage Weil
53cb6ac0d4 mgr/devicehealth: show-health-metrics -> get-health-metrics
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 12:43:58 -05:00
Cleber Rosa
f0f6e46084 qa/btrfs/test_rmdir_async_snap: remove binary file
On a quick look at the source code, I noticed this binary file, which
looks like was committed by mistake.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
2018-09-14 13:26:32 -04:00
Kefu Chai
07a3e8d7a6
Merge pull request #23650 from tchaikov/wip-dpdk-cmake
cmake: fix the build WITH_DPDK=ON

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2018-09-14 23:44:01 +08:00
Jason Dillaman
862082792d librbd: test_flags helper should require snap id parameter
The HEAD and snapshots have potentially different flag states
since object maps get invalidated per revision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2018-09-14 11:09:23 -04:00
Sage Weil
0c75020cad common/hobject: add get_logical_pool()
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 08:44:37 -05:00
Sage Weil
cf4a4ebf27 common/hobject: define get_temp_pool() etc helpers
Signed-off-by: Sage Weil <sage@redhat.com>
2018-09-14 08:40:43 -05:00
Kefu Chai
b035df69f7
Merge pull request #24097 from cyx1231st/fix-seastar-msgr-read-racing
crimson/net: Fix racing for promise on_message

Reviewed-by: Kefu Chai <kchai@redhat.com>
2018-09-14 19:03:54 +08:00
Yingxin
6365e02926 crimson/net: Fix racing for promise on_message
read_tags_until_next_message() will forward the ready future and create
a new promise for on_message, which assumes there is already a
read_message() holding the previous promise, but it is not true.

Signed-off-by: Yingxin <yingxin.cheng@intel.com>
2018-09-14 18:33:21 +08:00
Nathan Cutler
ee35f017fc
Merge pull request #23972 from badone/wip-documentation-add-12.2.8-release
doc/releases/schedule: Add Luminous 12.2.8

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-09-14 09:52:59 +02:00
Nathan Cutler
0e0c88d930
Merge pull request #24099 from badone/wip-mimic-13-2-0-date-correction
doc/releases: Mark Mimic first release as June

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2018-09-14 09:52:19 +02:00
xie xingguo
8fc491161b osd/PG: kill __queue_query__
which has no consumbers...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2018-09-14 13:33:47 +08:00
xie xingguo
8ef8cdba15 osd/OSD: deprecate __project_pg_history__
__project_pg_history__ does an inverse traverse of the series
of osdmaps passed in to get a pg's pg_history_t filled, which
can become super inefficient if the osdmap list to check is
very long.
E.g., in one of our clusters, we've observed it took approximate
10s for a PG to finish it's projecting:
```
2018-08-27 13:51:58.694823 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 from 821893 to 823276, start ec=380829/380829 l
is/c 820412/820412 les/c/f 820413/820413/0 821785/821785/821785
2018-08-27 13:52:08.634230 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 acting|up changed in 822265 from [57]/[57] 57/5
7 -> [58,57]/[58,57] 58/58
2018-08-27 13:52:08.634244 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 up changed in 822265 from [57] 57 -> [58,57] 58
2018-08-27 13:52:08.634248 7f1e1335a700 15 osd.9 823276 project_pg_history 34.6e9 primary changed in 822265
2018-08-27 13:52:08.634250 7f1e1335a700 15 osd.9 823276 project_pg_history end ec=380829/380829 lis/c 820412/820412 les/c/f 82041
3/820413/0 822265/822265/822265
```
Quote from Sage:
> let's just kill this off entirely, and make the handle_pg_query_nopg
reply unconditionally. Or, maybe, do a single sloppy check to see if
the primary has changed since the original epoch... if the osdmap
happens to be in cache... or not.
The querying end will discard the reply if it is out of date from it's
perspective, so it doesn't matter, and I suspect the overhead of doing
the check is larger than the overhead of sending a query reply that gets ignored.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: yanjun <yan.jun8@zte.com.cn>
2018-09-14 13:30:55 +08:00