RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2024-12-29 06:52:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
90,340 Commits 1 Branch 394 Tags 1.4 GiB
c1bff2684f
Commit Graph

90340 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kefu Chai
c1bff2684f
Merge pull request #24076 from ifed01/wip-ifed-fix-repairer 
os/bluestore: fix bloom filter num entry miscalculation in repairer

Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-19 18:03:21 +08:00
Kefu Chai
374931d210
Merge pull request #24085 from liewegas/wip-32731 
osd/osd_types: fix pg_t::contains() to check pool id too

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-19 18:02:13 +08:00
Kefu Chai
1c5f2cca66
Merge pull request #23721 from batrick/log-entry-stack 
log: avoid heap allocations for most log entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>
 2018-09-19 18:01:06 +08:00
Kefu Chai
58a08c6d4e
Merge pull request #24111 from gregsfortytwo/wip-35987-connection-buffers 
msg/async: clean up local buffers on dispatch

Reviewed-by: Haomai Wang <haomai@xsky.com>
 2018-09-19 17:56:58 +08:00
Kefu Chai
184e200110
Merge pull request #24124 from tchaikov/wip-35969 
rpm: use updated gperftools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
 2018-09-19 17:55:51 +08:00
Laura Paduano
46d55fcbaf
Merge pull request #23754 from p-na/down-osd-raises-error 
mgr/dashboard: Fix OSD down error display

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
 2018-09-19 08:56:33 +02:00
Sage Weil
7591ccd718 Merge PR #24098 into master 
* refs/pull/24098/head:
	Rename "include/assert.h"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-18 17:24:12 -05:00
Sage Weil
b2a9c6082d Merge PR #24112 into master 
* refs/pull/24112/head:
	osd: avoid unncecessary ref-counting in OSD::enqueue_op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-18 11:57:27 -05:00
Kefu Chai
3c2004c0ac
Merge pull request #24090 from cbodley/wip-crimson-concurrent-dispatch 
crimson/net: concurrent dispatch for SocketMessenger

Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-18 21:21:41 +08:00
Sage Weil
69edc10568 Merge PR #24092 into master 
* refs/pull/24092/head:
	mgr: Change signature of PyString_AsString to match return

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-18 07:10:40 -05:00
Sage Weil
359f6efe9b Merge PR #24057 into master 
* refs/pull/24057/head:
	src/common: add a unit test (bufferlist.sha1())
	osd, src/common: return sha1 value if zero-length buffer.
	src/common/buffer.cc: remove unnecessary copy in sha1()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-18 07:10:21 -05:00
Sage Weil
62bad1b987 Merge PR #24072 into master 
* refs/pull/24072/head:
	osd/PG: kill __queue_query__
	osd/OSD: deprecate __project_pg_history__

Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-18 07:09:52 -05:00
Sage Weil
5cd067b16b Merge PR #24104 into master 
* refs/pull/24104/head:
	mgr/diskprediction: add prototype diskprediction module

Reviewed-by: John Spray <john.spray@redhat.com>
 2018-09-18 07:08:05 -05:00
Lenz Grimmer
d8a4b4bf47
Merge pull request #24103 from b-ranto/wip-dashboard-http 
mgr/dashboard: Do not require cert for http

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
 2018-09-18 13:40:54 +02:00
Lenz Grimmer
2d02d5c9eb
Merge pull request #24015 from ricardoasmarques/wip-update-selection-on-refresh 
mgr/dashboard: More configs for table `updateSelectionOnRefresh`

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
 2018-09-18 11:02:43 +02:00
Boris Ranto
035b3a8b33 mgr/dashboard: Rename SSLCherryPyConfig 
The class does not always set up SSL, dropping the SSL bit.

Fixes: http://tracker.ceph.com/issues/36069

Signed-off-by: Boris Ranto <branto@redhat.com>
 2018-09-18 10:49:01 +02:00
Boris Ranto
20ed0de477 mgr/dashboard: Do not require cert for http 
The ceph dashboard currently requires a SSL certificate even if it is
not running in the SSL mode since it is always querying for the
certificate file/key pair.

This patch fixes the behaviour by querying for the certificate file/key
only if it is running in the SSL mode.

Fixes: http://tracker.ceph.com/issues/36069

Signed-off-by: Boris Ranto <branto@redhat.com>
 2018-09-18 10:48:51 +02:00
Josh Durgin
202b27363d
Merge pull request #24031 from rzarzynski/wip-common-bl-drop-0c 
common: drop the unused zero-copy facilities in ceph::bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-17 13:48:41 -07:00
Patrick Donnelly
7da1e283cd
Merge PR #24120 into master 
* refs/pull/24120/head:
	doc: Fix Typos of Developer Guide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-17 11:10:15 -07:00
Mykola Golub
5fc71af857
Merge pull request #24105 from dillaman/wip-24516 
librbd: object map improperly flagged as invalidated

Reviewed-by: Mykola Golub <mgolub@suse.com>
 2018-09-17 20:31:36 +03:00
Matt Benjamin
9cfdf3d7e8
Merge pull request #23981 from cbodley/wip-35830 
rgw: raise debug level on redundant data sync error messages
 2018-09-17 11:26:50 -04:00
Kefu Chai
338612ad88
Merge pull request #24088 from dzafman/wip-35982 
qa/standalone: Standalone test corrections

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-17 22:35:43 +08:00
Radoslaw Zarzynski
b71cbb97b7 common: get_data() of buffer_raw isn't virtual anymore. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:34:33 +02:00
Radoslaw Zarzynski
a9ff787071 common: make the get_data() of buffer_raw interface final. 
This is just to ensure the just dropped buffer::raw_pipe
was the solely user of this facility. After successful
validation, we can drop `virtual` on the method entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:34:33 +02:00
Radoslaw Zarzynski
2fa78f59ad common: drop can_zero_copy() from ceph::buffer interfaces. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:34:33 +02:00
Radoslaw Zarzynski
e81d55803f common: drop zero_copy_to_fd() from ceph::buffer interfaces. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:33:26 +02:00
Radoslaw Zarzynski
df90885ed1 common: drop write_fd_zero_copy() from bufferlist. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:31:45 +02:00
Radoslaw Zarzynski
6fc91d1a68 common: drop buffer::raw_pipe altogether with its factory method. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:31:45 +02:00
Radoslaw Zarzynski
7518eed56b common: drop buffer::list::read_fd_zero_copy. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:31:45 +02:00
Radoslaw Zarzynski
53e491b6a6 common, test: drop unit tests for 0c in ceph::bufferlist. 
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
 2018-09-17 15:31:45 +02:00
Rick Chen
4abb79f159 mgr/diskprediction: add prototype diskprediction module 
This module is written by Rick Chen <rick.chen@prophetstor.com> and
provides both a built-in local predictor and a cloud mode that queries
a cloud service (provided by ProphetStor) to predict device failures.

Signed-off-by: Rick Chen <rick.chen@prophetstor.com>
Signed-off-by: Sage Weil <sage@redhat.com>
 2018-09-17 08:20:57 -05:00
Kefu Chai
f557a78540
Merge pull request #24087 from rzarzynski/wip-common-buffer_raw-drop-page-align-getters 
common: drop/mark-as-final getters of buffer::raw for palign.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
 2018-09-17 15:55:10 +08:00
Kefu Chai
237bc951c4
Merge pull request #24061 from liewegas/wip-35923 
osd/PG: write pg epoch when resurrecting pg after delete vs merge race

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
 2018-09-17 15:52:49 +08:00
Kefu Chai
3d1d337428
Merge pull request #23306 from penglaiyxy/msg_branch 
msg: ceph_abort() when there are enough accepter errors in msg server

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-17 15:50:57 +08:00
Kefu Chai
f46523e464
Merge pull request #23955 from wjwithagen/wjw-fix-ceph-helpers.sh 
test: Start using GNU awk and fix archiving directory

Reviewed-by: Kefu Chai <kchai@redhat.com>
 2018-09-17 15:44:06 +08:00
Kefu Chai
e69e50a859 rpm: use updated gperftools 
make sure we only build with the higher version of gperftools on
distros where both 2.4 and 2.6.1 are packaged. see
https://git.centos.org/summary/rpms!gperftools.git . at the time of
writing, gperftools 2.6.1 is packaged for CentOS/RHEL 7, if gperftools
(>= 2.4) is required by Ceph, and user already has this version
installed, when new Ceph packages are installed, the updated gperftools
2.6.1 version won't be installed as a dependency. when launching
Ceph compiled with tcmalloc enabled, we will have

symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm

so, by bumping up the required version of gperftools, the updated
gperftools will be installed.

see https://software.opensuse.org/package/gperftools, openSUSE/SLE offer
2.5. so they are safe at this moment.

Fixes: http://tracker.ceph.com/issues/35969
Signed-off-by: Kefu Chai <kchai@redhat.com>
 2018-09-17 14:20:51 +08:00
李丙洋 10208981
64613065ac doc: Fix Typos of Developer Guide 
Signed-off-by: Li Bingyang <li.bingyang1@zte.com.cn>
 2018-09-17 09:59:29 +08:00
Patrick Donnelly
0ac789c435
Merge PR #24023 into master 
* refs/pull/24023/head:
	mds: prevent rctime from going back

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-16 15:01:01 -07:00
Patrick Donnelly
b43f26c1d9
Merge PR #24022 into master 
* refs/pull/24022/head:
	client: update ctime when modifying file content

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-16 14:58:38 -07:00
Patrick Donnelly
1b3b2744f9
Merge PR #24048 into master 
* refs/pull/24048/head:
	mds: use common send message mechanisms
	mds: remove dead code

Reviewed-by: Zheng Yan <zyan@redhat.com>
 2018-09-16 14:51:17 -07:00
Patrick Donnelly
cd1ce232ba
Merge PR #24053 into master 
* refs/pull/24053/head:
	mds: translate CEPH_MDS_STATE_NULL state name as null

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-16 14:33:55 -07:00
Patrick Donnelly
ea43ce7aae
Merge PR #21992 into master 
* refs/pull/21992/head:
	qa: extend timeout for SessionMap flush

Reviewed-by: John Spray <john.spray@redhat.com>
 2018-09-16 14:30:28 -07:00
Jason Dillaman
0a31c55ea8 librbd: do not invalidate object map when attempting to delete non-existent snapshot 
If duplicate snapshot remove requests are received by the lock owner from a peer
client, the first request will remove the object map. If the second request
arrives while the first is in-progress, it will again attempt to remove the
object map but fail to load it since it's already been deleted. This incorrectly
results in the next object map being flagged as invalid.

Fixes: http://tracker.ceph.com/issues/24516
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
 2018-09-16 12:46:25 -04:00
Jason Dillaman
58770188ab librbd: converted object map snapshot remove state machine to new style 
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
 2018-09-16 12:46:25 -04:00
Kefu Chai
28d9982bde
Merge pull request #24029 from smithfarm/wip-35922 
cmake: no libradosstriper headers if WITH_LIBRADOSSTRIPER=OFF

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
 2018-09-17 00:04:03 +08:00
Mykola Golub
a68be2b452
Merge pull request #24036 from dillaman/wip-24688 
rbd-mirror: optionally extract peer secrets from config-key

Reviewed-by: Mykola Golub <mgolub@suse.com>
 2018-09-16 11:53:04 +03:00
Mykola Golub
fa7bb832a3
Merge pull request #24063 from dillaman/wip-24140 
rbd-mirror: prevent creation of clones when parents are syncing

Reviewed-by: Mykola Golub <mgolub@suse.com>
 2018-09-16 11:52:32 +03:00
Patrick Donnelly
9c10509092
log: use smaller buffer for ConcreteEntry 
This brings down the static size of the memory used by the logging infrastructure:

    If we used 1024, we'd have 1088*10000 = 10880000 = 10MB in use by the ring
    buffer and 2*1088*100 = 2*108800 = 2*106KB for the m_new and m_flush
    vectors.

In my testing, 1024 covers most log entries.

Note, I've kept 4096 for the StackStringStream via MutableEntry as these are
already allocated on the heap and cached in a thread local vector. Generally
there should only be about a dozen of these allocated so it's worth keeping a
larger buffer.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-14 19:46:12 -07:00
Patrick Donnelly
b86994e3f9
common: remove obsolete buffer classes 
In favor of the simpler StackStreamBuffer.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-14 19:46:11 -07:00
Patrick Donnelly
a747aeac13
log: avoid heap allocations for most log entries 
Each log Entry now exists on the stack and uses a large (4k) buffer for its log
stream. This Entry is std::move'd to the queues (std::vector and
boost::circular_buffer) in the Log, involving only memory copies in the general
case. There are two memory copies (std::move) for any given Entry, once in
Log::submit_entry and again in Log::_flush

In practice, this eliminates 100% of allocations outside of startup
allocations

I've run a simple experiment with the MDS that copies /usr/bin to CephFS. I got
measurements for the number of allocations from the heap profiler and the
profile of CPU usage in the MDS.

** Before this patch **

    == Heap profile: ==

    $ google-pprof --alloc_objects --text bin/ceph-mds out/mds.a.profile.0001.heap
    Total: 1105048 objects
      433329  39.2%  39.2%   433329  39.2% ceph::logging::Log::create_entry
      209311  18.9%  58.2%   209311  18.9% __gnu_cxx::__aligned_membuf::_M_addr (inline)
      192963  17.5%  75.6%   192963  17.5% __gnu_cxx::new_allocator::allocate (inline)
       61774   5.6%  81.2%    61774   5.6% std::__cxx11::basic_string::_M_mutate
       37689   3.4%  84.6%    37689   3.4% ceph::buffer::raw_combined::create (inline)
       22773   2.1%  86.7%    22773   2.1% mempool::pool_allocator::allocate (inline)
       17761   1.6%  88.3%    20523   1.9% std::pair::pair (inline)
       15795   1.4%  89.7%    15797   1.4% std::swap (inline)
       11011   1.0%  90.7%   130061  11.8% std::__cxx11::list::_M_insert (inline)
       10822   1.0%  91.7%    10822   1.0% std::__cxx11::basic_string::reserve
        9108   0.8%  92.5%    32721   3.0% __gnu_cxx::new_allocator::construct (inline)
        8608   0.8%  93.3%     8610   0.8% std::_Deque_base::_M_initialize_map (inline)
        7694   0.7%  94.0%     7694   0.7% std::__cxx11::basic_string::_M_capacity (inline)
        6160   0.6%  94.5%     6160   0.6% Journaler::wrap_finisher
        6084   0.6%  95.1%    70892   6.4% std::map::operator[] (inline)
        5347   0.5%  95.6%     5347   0.5% MutationImpl::add_projected_fnode
        4381   0.4%  96.0%     7706   0.7% mempool::pool_allocator::construct (inline)
        3588   0.3%  96.3%   182966  16.6% Locker::_do_cap_update
        3049   0.3%  96.6%     5280   0.5% std::__shared_count::__shared_count (inline)
        3043   0.3%  96.9%     3043   0.3% MDSLogContextBase::MDSLogContextBase (inline)
        3038   0.3%  97.1%    14763   1.3% std::__uninitialized_copy::__uninit_copy (inline)

   So approximately 430k heap allocations for Entry were created! The
   basic_string::_M_mutate is also another allocation via PrebufferedStreambuf

	== Profile data ==

	Selecting interesting functions

Samples: 798K of event 'cycles:pp', Event count (approx
  Children      Self  Com  Shared Object        Symbol
+    1.04%     1.04%  log  libceph-common.so.0  [.] ceph::logging::Log::_flush
+    0.05%     0.05%  log  libceph-common.so.0  [.] ceph::logging::Log::flush
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::_log_safe_write
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::entry
+    0.00%     0.00%  log  libceph-common.so.0  [.] ceph::logging::Log::_flush_logbuf
...

  Children      Self  Command         Shared Object        Symbol
+    3.69%     0.00%  safe_timer      libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.53%     0.00%  ms_dispatch     libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.13%     0.00%  fn_anonymous    libceph-common.so.0  [.] CachedPrebufferedStreambuf::~CachedPrebufferedStreambuf
+    0.00%     0.00%  ms_dispatch     libceph-common.so.0  [.] CachedPrebufferedStreambuf::create
+    0.00%     0.00%  fn_anonymous    libceph-common.so.0  [.] CachedPrebufferedStreambuf::create

  Children      Self  Command         Shared Object        Symbol
+    0.07%     0.07%  fn_anonymous    libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  ms_dispatch     libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  ms_dispatch     ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  md_submit       libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  fn_anonymous    ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  safe_timer      libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  mds_rank_progr  libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  mds_rank_progr  ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  msgr-worker-0   libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  msgr-worker-2   libceph-common.so.0  [.] ceph::logging::Log::create_entry
+    0.00%     0.00%  md_submit       ceph-mds             [.] _ZN4ceph7logging3Log12create_entryEiiPm@plt
+    0.00%     0.00%  msgr-worker-1   libceph-common.so.0  [.] ceph::logging::Log::create_entry

  Children      Self  Command         Shared Object        Symbol
+    8.29%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    7.55%     1.46%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::_M_insert<long>
+    3.87%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.92%     0.00%  md_submit       libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.38%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] virtual thunk to std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    2.22%     2.22%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::sentry::sentry
+    1.89%     0.13%  fn_anonymous    libstdc++.so.6.0.24  [.] std::__ostream_insert<char, std::char_traits<char> >
+    0.71%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::basic_ostream<char, std::char_traits<char> >::~basic_ostream
+    0.39%     0.06%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::_M_insert<long>
+    0.29%     0.21%  ms_dispatch     libstdc++.so.6.0.24  [.] std::__ostream_insert<char, std::char_traits<char> >
+    0.27%     0.27%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::sentry::~sentry
+    0.27%     0.27%  fn_anonymous    libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>
+    0.22%     0.22%  ms_dispatch     libstdc++.so.6.0.24  [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
+    0.20%     0.20%  ms_dispatch     libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<long>
+    0.15%     0.15%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::sentry::~sentry
+    0.14%     0.14%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::sentry::sentry
+    0.13%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::ostream::_M_insert<unsigned long>
+    0.13%     0.13%  fn_anonymous    libstdc++.so.6.0.24  [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
+    0.00%     0.00%  fn_anonymous    libstdc++.so.6.0.24  [.] std::ostream::_M_insert<unsigned long>
+    0.00%     0.00%  ms_dispatch     libstdc++.so.6.0.24  [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<unsigned long>

    And the unittest_log time:

    $ bin/unittest_log
    [==========] Running 15 tests from 1 test case
    [----------] Global test environment set-up
    [----------] 15 tests from Log
    [ RUN      ] Log.Simple
    [       OK ] Log.Simple (0 ms)
    [ RUN      ] Log.ReuseBad
    [       OK ] Log.ReuseBad (1 ms)
    [ RUN      ] Log.ManyNoGather
    [       OK ] Log.ManyNoGather (0 ms)
    [ RUN      ] Log.ManyGatherLog
    [       OK ] Log.ManyGatherLog (12 ms)
    [ RUN      ] Log.ManyGatherLogStringAssign
    [       OK ] Log.ManyGatherLogStringAssign (27 ms)
    [ RUN      ] Log.ManyGatherLogStringAssignWithReserve
    [       OK ] Log.ManyGatherLogStringAssignWithReserve (27 ms)
    [ RUN      ] Log.ManyGatherLogPrebuf
    [       OK ] Log.ManyGatherLogPrebuf (15 ms)
    [ RUN      ] Log.ManyGatherLogPrebufOverflow
    [       OK ] Log.ManyGatherLogPrebufOverflow (15 ms)
    [ RUN      ] Log.ManyGather
    [       OK ] Log.ManyGather (8 ms)
    [ RUN      ] Log.InternalSegv

    [WARNING] /home/pdonnell/cephfs-shell/src/googletest/googletest/src/gtest-death-test.cc:836:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads
    [       OK ] Log.InternalSegv (8 ms)
    [ RUN      ] Log.LargeLog
    [       OK ] Log.LargeLog (43 ms)
    [ RUN      ] Log.TimeSwitch
    [       OK ] Log.TimeSwitch (1 ms)
    [ RUN      ] Log.TimeFormat
    [       OK ] Log.TimeFormat (0 ms)
    [ RUN      ] Log.Speed_gather
    [       OK ] Log.Speed_gather (1779 ms)
    [ RUN      ] Log.Speed_nogather
    [       OK ] Log.Speed_nogather (64 ms)
    [----------] 15 tests from Log (2000 ms total)

    [----------] Global test environment tear-down
    [==========] 15 tests from 1 test case ran. (2000 ms total)
    [  PASSED  ] 15 tests

** After Patch **

The StackStreamBuf uses 4k for its default buffer. This appears to be more than
reasonable for preventing allocations for logging

    == Heap profile: ==

	$ google-pprof --alloc_objects --text bin/ceph-mds out/mds.a.profile.0001.heap
	Total: 1052127 objects
      384957  36.6%  36.6%   384957  36.6% __gnu_cxx::new_allocator::allocate (inline)
      274720  26.1%  62.7%   274720  26.1% __gnu_cxx::__aligned_membuf::_M_addr (inline)
       68496   6.5%  69.2%    68496   6.5% std::__cxx11::basic_string::_M_mutate
       44140   4.2%  73.4%    51828   4.9% std::pair::pair (inline)
       43091   4.1%  77.5%    43091   4.1% ceph::buffer::raw_combined::create (inline)
       27706   2.6%  80.1%   236407  22.5% std::__cxx11::list::_M_insert (inline)
       25699   2.4%  82.6%    25699   2.4% std::__cxx11::basic_string::reserve
       23183   2.2%  84.8%    23183   2.2% mempool::pool_allocator::allocate (inline)
       19466   1.9%  86.6%    81716   7.8% __gnu_cxx::new_allocator::construct (inline)
       17606   1.7%  88.3%    17606   1.7% std::__cxx11::basic_string::_M_capacity (inline)
       16879   1.6%  89.9%    16881   1.6% std::swap (inline)
        8572   0.8%  90.7%     8574   0.8% std::_Deque_base::_M_initialize_map (inline)
        8477   0.8%  91.5%    11808   1.1% mempool::pool_allocator::construct (inline)
        6166   0.6%  92.1%     6166   0.6% Journaler::wrap_finisher
        6080   0.6%  92.7%   134975  12.8% std::map::operator[] (inline)
        6079   0.6%  93.3%     6079   0.6% MutationImpl::add_projected_fnode

	== Profile data ==

    Samples: 62K of event 'cycles:u', Event count (approx.)
      Overhead  Command         Shared Object         Symbol
    +    5.91%  log             libc-2.23.so          [.] vfprintf
    +    5.75%  ms_dispatch     libstdc++.so.6.0.24   [.] std::__ostream_insert<char, std::char_traits<char> >
    +    4.59%  ms_dispatch     ceph-mds              [.] StackStringBuf<4096ul>::xsputn
    +    4.26%  ms_dispatch     libc-2.23.so          [.] __memmove_ssse3_back
    +    4.07%  log             libceph-common.so.0   [.] ceph::logging::Log::_flush
    +    2.48%  ms_dispatch     libstdc++.so.6.0.24   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<l
    +    2.13%  fn_anonymous    libstdc++.so.6.0.24   [.] std::__ostream_insert<char, std::char_traits<char> >
    +    2.09%  ms_dispatch     ceph-mds              [.] CDir::check_rstats
    +    2.06%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::sentry::sentry
    +    1.98%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::sentry::~sentry
    +    1.87%  log             libc-2.23.so          [.] __strcpy_sse2_unaligned
    +    1.60%  fn_anonymous    ceph-mds              [.] StackStringBuf<4096ul>::xsputn
    +    1.46%  log             libc-2.23.so          [.] _IO_default_xsputn
    +    1.45%  log             libc-2.23.so          [.] _itoa_word
    +    1.43%  fn_anonymous    libc-2.23.so          [.] __memmove_ssse3_back
    +    1.40%  ms_dispatch     libstdc++.so.6.0.24   [.] std::ostream::_M_insert<long>
    +    0.98%  fn_anonymous    libstdc++.so.6.0.24   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<l
    +    0.89%  ms_dispatch     libstdc++.so.6.0.24   [.] 0x
    +    0.88%  ms_dispatch     libstdc++.so.6.0.24   [.] std::_Rb_tree_increment

    And the unittest_log time:

    $ bin/unittest_log
    [==========] Running 13 tests from 1 test case.
    [----------] Global test environment set-up.
    [----------] 13 tests from Log
    [ RUN      ] Log.Simple
    [       OK ] Log.Simple (1 ms)
    [ RUN      ] Log.ReuseBad
    [       OK ] Log.ReuseBad (0 ms)
    [ RUN      ] Log.ManyNoGather
    [       OK ] Log.ManyNoGather (0 ms)
    [ RUN      ] Log.ManyGatherLog
    [       OK ] Log.ManyGatherLog (83 ms)
    [ RUN      ] Log.ManyGatherLogStringAssign
    [       OK ] Log.ManyGatherLogStringAssign (79 ms)
    [ RUN      ] Log.ManyGatherLogStackSpillover
    [       OK ] Log.ManyGatherLogStackSpillover (81 ms)
    [ RUN      ] Log.ManyGather
    [       OK ] Log.ManyGather (80 ms)
    [ RUN      ] Log.InternalSegv

    [WARNING] /home/pdonnell/ceph/src/googletest/googletest/src/gtest-death-test.cc:836:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 3 threads.
    [       OK ] Log.InternalSegv (7 ms)
    [ RUN      ] Log.LargeLog
    [       OK ] Log.LargeLog (55 ms)
    [ RUN      ] Log.TimeSwitch
    [       OK ] Log.TimeSwitch (4 ms)
    [ RUN      ] Log.TimeFormat
    [       OK ] Log.TimeFormat (1 ms)
    [ RUN      ] Log.Speed_gather
    [       OK ] Log.Speed_gather (1441 ms)
    [ RUN      ] Log.Speed_nogather
    [       OK ] Log.Speed_nogather (63 ms)
    [----------] 13 tests from Log (1895 ms total)

    [----------] Global test environment tear-down
    [==========] 13 tests from 1 test case ran. (1895 ms total)
    [  PASSED  ] 13 tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
 2018-09-14 19:46:08 -07:00