Commit Graph

73293 Commits

Author SHA1 Message Date
Sage Weil
765b9d25e4 Merge pull request #15469 from wjwithagen/wip-wjw-revert-wvla
build: revert -Wvla from #15342

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2017-06-05 13:33:41 -05:00
Sage Weil
06f097c4ac Merge pull request #15470 from xiexingguo/wip-rm-false-assert
os/bluestore: fix false asserts in Cache::trim_all()

Reviewed-by: Sage Weil <sage@redhat.com>
2017-06-05 13:33:16 -05:00
Sage Weil
ac7c2c1174 Merge pull request #15478 from xiexingguo/wip-perf-avg-time
common/perf_counters: add average time for PERFCOUNTER_TIME

Reviewed-by: Sage Weil <sage@redhat.com>
2017-06-05 13:32:53 -05:00
xie xingguo
767502e26a common/perf_counters: add average time for PERFCOUNTER_TIME
Otherwise we'll have to calculate this manually, which is annoying.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2017-06-05 18:25:57 +08:00
xie xingguo
34b1ff73aa os/bluestore: fix false asserts in Cache::trim_all()
These asserts are true if we are going to shutdown the BlueStore instance.
But the caller can also be something like "ceph daemon out/osd.1.asok flush_store_cache",
which can fire these asserts as a result.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2017-06-05 18:08:00 +08:00
Nathan Cutler
b4560572b2 Merge pull request #15231 from smithfarm/wip-20052
build/ops: rpm: make librbd1 %post scriptlet depend on coreutils

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2017-06-05 11:38:40 +02:00
Kefu Chai
4722abe95c Merge pull request #15165 from badone/wip-cls-optimize-header-file-dependency
cls: optimize header file dependency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-06-05 10:15:56 +08:00
Willem Jan Withagen
bdf524e05b build: revert -Wvla from #15342
- VLAs are in GCC and Clang, and are there to stay forever,
   if only to be compatible with all the software that is already
   out there.
   - Theoretical debates about VLA being hard to implement are
     long superceded by th actual implentations
 - Before setting this flag is would be required to first start
   work on fixing all the fallout/warnings that will arise from
   setting -Wvla
 - Allocating large variable/stuctures on the stack could be asking
   for trouble, but changes that ceph tools are going to be running
   on small embedded devices are rather slim.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2017-06-04 21:25:08 +02:00
Sage Weil
554cf8394a Merge pull request #15073 from liewegas/wip-mgr-stats
mon,mgr: extricate PGmap from monitor

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2017-06-04 13:36:01 -05:00
Kefu Chai
8185bc059d qa/suites/upgrade/hammer-jewel-x: don't initially start mgr daemons
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-04 14:35:15 -04:00
Kefu Chai
69bb53b06f qa/workunits/rados/test_health_warning: misc fixes
* do not let osd shutdown itself by enlarge osd_max_markdown_count and
  shorten osd_max_markdown_period
* do not shutdown all osds in the last test. if all osds are shutdown at
  the same time. none of them will get updated osdmap after noup is
  unset. we should leave at least one of them, so the gossip protocol
  can kick in, and populate the news to all osds.

Fixes: http://tracker.ceph.com/issues/20174
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-04 14:35:15 -04:00
Kefu Chai
c24a08d6f0 mon/OSDMonitor: add a space after __func__ in log message
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-04 01:37:05 +08:00
Jason Dillaman
4cb62c2f5c Merge pull request #15443 from trociny/wip-rbd_image_options_t
librbd: remove unused rbd_image_options_t ostream operator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-06-03 09:43:11 -04:00
Nathan Cutler
28a5098796 build/ops: rpm: make librbd1 %post scriptlet depend on coreutils
Fixes: http://tracker.ceph.com/issues/20052
Signed-off-by: Giacomo Comes <comes@naic.edu>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2017-06-03 15:01:43 +02:00
Kefu Chai
97d6e150e5 Merge pull request #15347 from tchaikov/wip-cmake
cmake: rgw: do not link against boost in a wholesale

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2017-06-03 20:44:24 +08:00
Jos Collin
2b56d24f52 Merge pull request #15353 from wjwithagen/wip-wjw-compare-object_catcher
test/osdc: fix comparison error and silence warning from -Wunused-value

Reviewed-by: Jos Collin <jcollin@redhat.com>
2017-06-03 17:24:40 +05:30
Kefu Chai
ba1ca1ce17 cmake: rgw: do not link against boost in a wholesale
With the new Beast frontend, RGW now has a small Boost dependency [1] which was
being addressed by statically (and unconditionally) linking *all* the Boost
libraries. This patch ensures that only the necessary Boost components are
linked.

We use the target_link_libraries(<target> <item>...) [2] syntax to ensure that the
library dependencies are transitive: i.e. "when this target is linked into
another target then the libraries linked to this target will appear on the link
line for the other target too."

[1] The boost/asio/spawn.hpp header used by rgw_asio_frontend.cc depends on
    boost::coroutine/boost::context

[2] https://cmake.org/cmake/help/v3.3/command/target_link_libraries.html#libraries-for-both-a-target-and-its-dependents

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-03 17:25:20 +08:00
Mykola Golub
c8717ca88c librbd: remove unused rbd_image_options_t ostream operator
This fixes librbd crashes currently observed on master, when
debug is on, because `rbd_image_options_t` is typedef-ed `void *`
and it's operator is used when attempting to print out an
address (`void *`) of any object.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
2017-06-03 10:43:01 +02:00
Sage Weil
cef5c1e213 mon: clean up PGMapDigest encoding
Remove compat cruft due to intitial testing on bigbang.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 15:12:18 -04:00
Casey Bodley
bf72bd8dcb Merge pull request #15119 from ceph/wip-rgw-config-docs
doc: mention certain conf vars should be in global

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-06-02 14:09:33 -04:00
Kefu Chai
8f6a526f9a selinux: clip the ceph context to ceph-mgr also
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:50 -04:00
Kefu Chai
e8b23d6852 qa/tasks: add a blacklist for flush_pg_stats()
so we don't wait for marked out osds.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:50 -04:00
Kefu Chai
befe07f197 mon/OSDMonitor: filter the added creating_pgs added from pgmap
the creating_pgs added from pgmap might contains pgs whose containing
pools have been deleted. this is fine with the PGMonitor, as it has the
updated pg mapping which is consistent with itself. but it does not work
with OSDMonitor's creating_pgs, whose pg mapping is calculated by
itself. so we need to filter the pgmap's creating_pgs when adding them to
OSDMonitor's creating_pgs with the latest osdmap.get_pools().

Fixes: http://tracker.ceph.com/issues/20067
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:50 -04:00
Sage Weil
5281a0ca96 mon/MonmapMonitor: initialize new cluster monmap with default persistent features
This is similar to what we do for OSDMonitor::create_initial().

Avoid setting these initial features just for teh mon test that verifies
persistent features get set on a full quorum.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:49 -04:00
Greg Farnum
27c5f6b6e9 mon: mgr: remove osd_stats map from PGMapDigest
We use this information only for dumps. Stop dumping per-OSD stats as they're
not needed. In order to maintain pool "fullness" information, calculate
the OSDMap-based rule availibility ratios on the monitor and include those
values in the PGMapDigest. Also do it whenever we call dump_pool_stats_full()
on the manager.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:49 -04:00
Greg Farnum
ce38a6c17a mon: pgstat: remove unused get_osd_sum from post-luminous state
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:49 -04:00
Greg Farnum
8b164903be mon: mgr: pgstats: do available-space calculations on mgr instead of mon
This means we don't need the osd_stat map on the monitor at all. We're
about to remove it entirely!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:49 -04:00
Greg Farnum
ad630462e7 mon: pgstat: remove unused get_pg_sum interface
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:48 -04:00
Greg Farnum
9fd76f57fd mon: pgstat: remove get_num_pg_by_osd() from post-luminous state
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:48 -04:00
Greg Farnum
459ec61901 mon: pgstat: remove get_osd_stat() from post-luminous state
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:48 -04:00
Greg Farnum
fc070255e8 osdmonitor: use a new get_full_osd_counts to generate health without osdstat
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
2017-06-02 13:06:48 -04:00
Kefu Chai
d1eca5f669 mon/OSDMonitor: print pgid before looking it up in mapping
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:48 -04:00
Sage Weil
f42182c5e0 mon: handle MGetPoolStats using PGStatService
otherwise ceph_test_rados_api_stat: LibRadosStat.PoolStat will always
timeout once the cluster is switched to luminous

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:47 -04:00
Sage Weil
2de0e07c40 mon: handle MStatfs using PGStatService
otherwise ceph_test_rados_api_stat: LibRadosStat.ClusterStat will always
timeout once the cluster is switched to luminous

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:47 -04:00
Sage Weil
6a68877f59 mon/PGMap: strip out PGMapDigest compat cruft
This was needed for bigbang testing, but not for the final version.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:47 -04:00
Kefu Chai
aad1afcb96 mgr: reset pending_inc after applying it
we cannot apply pending_inc twice and expect the result is the same. in
other words, pg_map.apply_incremental(pending_inc) is not an idempotent
operation.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:47 -04:00
Sage Weil
af2994dcae osd: do_shutdown takes precedence over fetching more maps
This is making my osd-markdown.sh test fail reliably.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:47 -04:00
Sage Weil
ebc496d7f4 osd/OSDMap: more efficient PGMapTemp
Use a flat_map with pointers into a buffer with the actual data.  For a
decoded mapping, we have just two allocations (one for flat_map and one
for the encoded buffer).

This can get slow if you make lots of incremental changes after the fact
since flat_map is not efficient for modifications at large sizes.  :/

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:46 -04:00
Kefu Chai
60a30ff9d9 mon: print log for the creating_pgs changes
print more log messages when updating creating_pgs.

see-also: http://tracker.ceph.com/issues/20067
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-06-02 13:06:46 -04:00
Sage Weil
9332bb9be4 mgr: mgr_tick_period = 2
5 seconds is driving me nuts.  We cap the health message size so the
digest is now small and lightweight.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:46 -04:00
Sage Weil
2f881dbf2c mon/PGMap: count 'unknown' pgs
Also, count "not active" (inactive) pgs instead of active so that we
list "bad" things consistently, and so that 'inactive' is a separate
bucket of pgs than the 'unknown' ones.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:06:46 -04:00
Sage Weil
15f823b43e mgr/MgrStandby: reset subscriptions when we become non-active
This is a goofy workaround that we're also doing in Mgr::init().  Someday
we should come up with a more elegant solution.  In the meantime, this
works just fine!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:54 -04:00
Sage Weil
6afca3beb2 mgr/ClusterState: make pg stat filtering less fragile
We want to drop updates for pgs for pools that don't exist.  Keep an
updated set of those pools instead of relying on the previous PGMap
having them instantiated.  (The previous map may drift due to bugs.)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:54 -04:00
Sage Weil
dcc4c52ee8 mgr/DaemonServer: log pgmap usage to cluster log
Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:54 -04:00
Sage Weil
d6d1db62ed mgr: apply PGMap incremental at same interval as reports
We were doing an incremental per osd stat report; this screws up the
delta stats updates when there are more than a handful of OSDs.  Instead,
do it with the same period as the mgr->mon reports.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:53 -04:00
Sage Weil
2e1f8fdb85 mon/PGMap: encode delta info in digest
It was already in PGMapDigest, but not encoded.

One field we didn't need; move that back to PGMap.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:53 -04:00
Sage Weil
3b049712ff mon/OSDMonitor: use newest creation epoch for pgs that we can
If we have a huge pool it may take a while for the PGs to get out of the
queue and be created.  If we use the epoch the pool was created it may
mean a lot of old OSDMaps the OSD has to process.  If we use the current
epoch (the first epoch in which any OSD learned that this PG should
exist) we limit PastIntervals as much as possible.

It is still possible that we start trying to create a PG but the cluster
is unhealthy for a long time, resulting in a long PastIntervals that
needs to be generated by a primary OSD when it eventually comes up. So
this only partially

Partially-fixes: http://tracker.ceph.com/issues/20050
Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:53 -04:00
Sage Weil
50c617a447 mon/OSDMonitor: clean up no-beacon message
Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:53 -04:00
Sage Weil
578cd59dd8 mgr: drop useless __func__ prints
This is part of the default prefix.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:53 -04:00
Sage Weil
85cc2595c7 osd: work around bluestore fragmetned buffers in get_map_bl
Signed-off-by: Sage Weil <sage@redhat.com>
2017-06-02 13:02:52 -04:00