Commit Graph

100902 Commits

Author SHA1 Message Date
Kefu Chai
e36332595e mgr/dashboard: add python-common to $PYTHONPATH
fix the regression introduced by 8c50be5df6, so ceph-mgr's python
modules are able to import python-common.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 15:00:13 +08:00
Jan Fajerski
47fecd80d8
Merge pull request #29382 from MI-OSiRIS/prometheus_scrape_interval
mgr/prometheus: Cast collect_timeout (scrape_interval) to float
2019-08-07 08:05:43 +02:00
Kiefer Chang
a1a3817659
mgr/dashboard: fix HACKING.rst is not rendered on github
The file is not rendered because of inconsistent section markers.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2019-08-07 11:40:09 +08:00
Jianpeng Ma
e782ac6fcf os/bluestore: Don't forget sub kv_submitted_waiters.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:17:11 +08:00
Jianpeng Ma
a2f63f4681 os/bluestore: No need check !q.emtpy().
when call drain_preceding, this TransContext already added into q.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:10:42 +08:00
Jianpeng Ma
62049dd179 os/bluestore: narrow deferred_lock in _deferred_aio_finish.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Jianpeng Ma
b6de4ab540 os/bluestore: reduce one lock for deferred_aggressive > 0.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Sage Weil
4ac4768e23 Merge PR #29116 into master
* refs/pull/29116/head:
	osd: move heartbeat connection cleanup to helper
	osd: break con<->session cycle when removing heartbeat peers
	osd: mark down heartbeat connections on shutdown
	crimson/: move get_mnow() to ShardServices, pass to heartbeat
	crimson/osd: stubs for get_mnow, get_hbstamps
	crimson/osd/heartbeat: adapt to new MOSDPing fields
	crimson/osdmap_service: add get_mnow(), get_up_epoch()
	osd/PeeringState: take HeartbeatStamps refs for current interval
	osd: track clock delta between peer OSDs
	osd: add get_mnow() interface to OSDService, PG, PeeringState
	osd: record startup_time
	osd: some minor refactoring/cleanup in handle_osd_ping

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-08-06 16:22:14 -05:00
Sage Weil
f35b5e5fa4 Merge PR #29498 into master
* refs/pull/29498/head:
	qa/tasks/ceph_manager: remove race from all_active_or_peered()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-06 16:21:58 -05:00
Willem Jan Withagen
32732b4b7e script/ceph-backport.sh: Use secure access for tracker.ceph.com
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2019-08-06 22:54:41 +02:00
Patrick Donnelly
bbd63e0a7c
Merge PR #29509 into master
* refs/pull/29509/head:
	doc: indicate imperative mood for commit titles

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-06 13:23:23 -07:00
Patrick Donnelly
e72f590477
doc: indicate imperative mood for commit titles
This is convention in the kernel and Ceph but is not clearly documented.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-06 11:18:21 -07:00
Rafael Quintero
7e408632b2 mgr/dashboard: Write E2E test to verify that Manager modules have editing functionality
Fixes: https://tracker.ceph.com/issues/40823

Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>
2019-08-06 14:05:35 -04:00
Sage Weil
25f28e610f common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Sage Weil
77b634ad08 common/config: let diff show non-build defaults
Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Jason Dillaman
7adb17f831 pybind/mgr/rbd_support: use image ids to detect duplicate tasks
This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.

Fixes: https://tracker.ceph.com/issues/41032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 12:17:46 -04:00
Kefu Chai
b6b3724f04
Merge pull request #29501 from tchaikov/wip-journal/test_JournalRecorder.cc
journal: always shutdown JournalRecoreder before destructing it

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 23:57:45 +08:00
Sage Weil
712409cac4 common/config: do no include multiple 'default' values
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 10:47:04 -05:00
Sridhar Seshasayee
1034782d4c mon/OSDMonitor: Add standalone test for mon_memory_target
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
e4d236492a mon/OSDMonitor: Implement config observer to handle changes to cache sizes
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
3b96417e18 mon/OSDMonitor: Use generic priority cache tuner for mon caches
Use priority cache manager to tune inc, full and rocksdb caches.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sage Weil
b2119ffb50 os/bluestore: teach fsck to tolerate per-pool omap
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:15 -05:00
Sage Weil
3cab0b3b09 os/bluestore: ondisk format change to 3 for per-pool omap
Move to ondisk format v3.  This means that per-pool omap keys may exist,
but does not imply that *all* objects use the new form until the
per_pool_omap=1 super key is also set.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
a076260e00 mon/PGMap: add data/omap breakouts for 'df detail' view
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
ab2eb6b832 osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
5a6ede0ec1 mon/PGMap: fix stored_raw calculation
The get_user_bytes() helper is a bit weird because it uses the
raw_used_rate (replication/EC factor) so that it can work *backwards*
from raw usage to normalized user usage.  However, the legacy case that
works from PG stats does not use this factor... and the stored_raw value
(in the JSON output only) was incorrectly passing in a factor of 1.0,
which meant that for legacy mode it was a bogus value.

Fix by calculating stored_raw as stored_normalized * raw_used_rate.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
b207973ae9 mon/PGMap: add in actual omap usage into per-pool stats
This is a minimal change: we aren't separately reporting data vs omap
usage (like we do in 'osd df' output for individual osds).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
d6ff61ed1f osd: report per-pool omap support via store_statfs_t
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
19f497c9b0 os/bluestore: set per_pool_omap key on mkfs
This key indicates that *all* objects put omap in the per-pool prefix and
key format.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
aa56c41ae8 osd/osd_types: count per-pool omap capable OSDs
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
e2a0717047 os/bluestore: report omap_allocated per-pool
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
91f533be71 os/bluestore: add pool prefix to omap keys
Set per-onode flag to indicate whether the object has per-pool keys or
not.  This will allow us to incrementally transition objects later.

Put the new keys under a different prefix.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
22a969acb1 kv/KeyValueDB: take key_prefix for estimate_prefix_size()
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
072039f822 os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
Make this code more general and robust.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:23:58 -05:00
Jan Fajerski
f0746bbbec
Merge pull request #27859 from jan--f/update-bluestore-cache-doc
doc: update bluestore cache settings and clarify data fraction
2019-08-06 13:32:58 +02:00
Adam Kupczyk
355dc85372 test/objectstore: Allocator_test. Add test for dumping free regions and fragmentation_score.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-08-06 12:19:29 +02:00
Adam Kupczyk
16a9dac167 BlueStore/allocator: Add command to inspect how much BlueStore's block can go to BlueFS.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-08-06 12:19:29 +02:00
Kefu Chai
f646704081 journal: always shutdown JournalRecoreder before destructing it
otherwise when we destruct `journal::JournalRecorder::m_object_locks`,
it/they would be still being waited by some condition variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-06 14:12:29 +08:00
Sage Weil
9719920920 qa/tasks/ceph_manager: no newlines in 'ceph -s' output
This gets dumped to the log, making it hard to read.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 22:02:31 -05:00
Sage Weil
3e7c185bd4 mon: make mon summary more concise in 'ceph -s'
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 22:02:31 -05:00
Kefu Chai
7265b55d09
Merge pull request #29475 from tchaikov/wip-cmake3-for-xenial
install-deps.sh: use chacra for cmake repo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-06 08:15:19 +08:00
Sage Weil
a4e860bc39 mgr/hello: fully document our options
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 14:45:27 -05:00
Sage Weil
41e4056174 qa/tasks/ceph_manager: remove race from all_active_or_peered()
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 14:01:02 -05:00
Sage Weil
8d155d0f67 osd: move heartbeat connection cleanup to helper
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00
Sage Weil
dcaba832e4 osd: break con<->session cycle when removing heartbeat peers
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00
Sage Weil
c6351315fb osd: mark down heartbeat connections on shutdown
This ensures that heartbeat_reset() gets call and we clean up the
ref loop between the Connections and Sessions.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00
Samuel Just
8f906e3cd7 crimson/: move get_mnow() to ShardServices, pass to heartbeat
get_mnow isn't clearly at home in OSDMapService, and the other methods
are needed from PeeringState, so let's consolidate on ShardServices
for now.  We probably ought OSDMapService state out of OSD into its
own module at some point.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-08-05 13:53:25 -05:00
Sage Weil
49478dd3d2 crimson/osd: stubs for get_mnow, get_hbstamps
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00
Sage Weil
9923c3fe76 crimson/osd/heartbeat: adapt to new MOSDPing fields
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00
Sage Weil
91c0df8121 crimson/osdmap_service: add get_mnow(), get_up_epoch()
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-05 13:53:25 -05:00