Commit Graph

100921 Commits

Author SHA1 Message Date
Casey Bodley
b8a60eb046 rgw: add rgw_remove_user_buckets_index
helper function to remove the user:buckets object.
rgw_remove_uid_index() now omits the object version tracker argument to
avoid reading the user info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 16:25:12 -04:00
Casey Bodley
4375204a4a rgw: simplify bucket chown
removes unused 'attrs' argument, and takes uid + display name instead
of RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 16:25:12 -04:00
Casey Bodley
17fc695047
Merge pull request #28813 from smanjara/wip-user-rename-working
rgw : Bucket mv, bucket chown and user rename utilities

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-08-07 16:05:07 -04:00
Neha Ojha
c9d2833b25
Merge pull request #29425 from aclamk/wip-bluestore-monitor-allocations
[bluestore][tools] Inspect allocations in bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-07 11:37:34 -07:00
Casey Bodley
553d97ca95 rgw: fix unlock of shared lock in RGWDataChangesLog
std::shared_mutex expects a call to unlock_shared() after lock_shared().
use the std::shared_lock guard to make it more obviously correct

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 14:02:49 -04:00
Adam Kupczyk
713f9b4d09 doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-08-07 19:18:21 +02:00
Tatjana Dehler
ce613b955a mgr/dashboard: fix RBD snapshot protected label
Fixes: https://tracker.ceph.com/issues/41153
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-08-07 17:45:10 +02:00
Nathan Cutler
e6514a0567
Merge pull request #29438 from wjwithagen/wjw-fix-ceph-backport.sh-https-tracker
script/ceph-backport.sh: Use secure access for tracker.ceph.com

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-08-07 17:04:13 +02:00
Sage Weil
7e10b70615 mgr/hello: typos
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:21 -05:00
Sage Weil
6a744f65a0 mgr/hello: a better command example
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:21 -05:00
Sage Weil
9318fb0d66 mgr/hello: boilerplate for handling config options
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:19 -05:00
Rafael Quintero
c3a2bdeebe mgr/dashboard: E2E Dashboard Page Test Suite
Fixes: https://tracker.ceph.com/issues/40549
Fixes: https://tracker.ceph.com/issues/40550
Fixes: https://tracker.ceph.com/issues/40561

Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>
2019-08-07 07:29:21 -04:00
Yingxin Cheng
03fbb3603a crimson/net: improve get_global_seq()
Implement single global_seq and non-racing get_global_seq() interface.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-07 19:14:27 +08:00
Kefu Chai
ace8cb15a6 admin/build-doc: use python3
to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
e049f86b2d
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 17:55:06 +08:00
Kefu Chai
d9a385da8c
Merge pull request #29525 from tchaikov/wip-mgr/dashboard/run-backend-api-tests.sh
mgr/dashboard: add python-common to $PYTHONPATH

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
2019-08-07 17:29:22 +08:00
Lenz Grimmer
e83a7a623c
Merge pull request #29451 from rhcs-dashboard/wip-41047-master
github: Add CODEOWNERs for designated code-owner reviews

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2019-08-07 11:14:08 +02:00
Lenz Grimmer
62c495b8e0
mgr/dashboard: fix HACKING.rst is not rendered on github (#29521)
mgr/dashboard: fix HACKING.rst is not rendered on github

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2019-08-07 10:53:06 +02:00
Kefu Chai
8bfd3cb1e1
Merge pull request #29461 from tchaikov/wip-mgr/dashboard/test_mgr_module
qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-07 16:34:08 +08:00
Kefu Chai
3303116f56 qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry
* use primitive types instead of `JLeaf(the_type)` as they are
  equivalent in this context
* remove fields which are added only if certain channels are
  activated.
* allow unknown fields, as we are including various stuff
  in the report, for instance, osdmap, usage, crash info, etc.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 15:25:18 +08:00
Kefu Chai
e36332595e mgr/dashboard: add python-common to $PYTHONPATH
fix the regression introduced by 8c50be5df6, so ceph-mgr's python
modules are able to import python-common.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 15:00:13 +08:00
Jan Fajerski
47fecd80d8
Merge pull request #29382 from MI-OSiRIS/prometheus_scrape_interval
mgr/prometheus: Cast collect_timeout (scrape_interval) to float
2019-08-07 08:05:43 +02:00
Kiefer Chang
a1a3817659
mgr/dashboard: fix HACKING.rst is not rendered on github
The file is not rendered because of inconsistent section markers.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2019-08-07 11:40:09 +08:00
Jianpeng Ma
e782ac6fcf os/bluestore: Don't forget sub kv_submitted_waiters.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:17:11 +08:00
Jianpeng Ma
a2f63f4681 os/bluestore: No need check !q.emtpy().
when call drain_preceding, this TransContext already added into q.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:10:42 +08:00
Jianpeng Ma
62049dd179 os/bluestore: narrow deferred_lock in _deferred_aio_finish.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Jianpeng Ma
b6de4ab540 os/bluestore: reduce one lock for deferred_aggressive > 0.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Sage Weil
4ac4768e23 Merge PR #29116 into master
* refs/pull/29116/head:
	osd: move heartbeat connection cleanup to helper
	osd: break con<->session cycle when removing heartbeat peers
	osd: mark down heartbeat connections on shutdown
	crimson/: move get_mnow() to ShardServices, pass to heartbeat
	crimson/osd: stubs for get_mnow, get_hbstamps
	crimson/osd/heartbeat: adapt to new MOSDPing fields
	crimson/osdmap_service: add get_mnow(), get_up_epoch()
	osd/PeeringState: take HeartbeatStamps refs for current interval
	osd: track clock delta between peer OSDs
	osd: add get_mnow() interface to OSDService, PG, PeeringState
	osd: record startup_time
	osd: some minor refactoring/cleanup in handle_osd_ping

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-08-06 16:22:14 -05:00
Sage Weil
f35b5e5fa4 Merge PR #29498 into master
* refs/pull/29498/head:
	qa/tasks/ceph_manager: remove race from all_active_or_peered()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-06 16:21:58 -05:00
Willem Jan Withagen
32732b4b7e script/ceph-backport.sh: Use secure access for tracker.ceph.com
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2019-08-06 22:54:41 +02:00
Patrick Donnelly
bbd63e0a7c
Merge PR #29509 into master
* refs/pull/29509/head:
	doc: indicate imperative mood for commit titles

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-06 13:23:23 -07:00
Patrick Donnelly
e72f590477
doc: indicate imperative mood for commit titles
This is convention in the kernel and Ceph but is not clearly documented.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-06 11:18:21 -07:00
Rafael Quintero
7e408632b2 mgr/dashboard: Write E2E test to verify that Manager modules have editing functionality
Fixes: https://tracker.ceph.com/issues/40823

Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>
2019-08-06 14:05:35 -04:00
Sage Weil
25f28e610f common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Sage Weil
77b634ad08 common/config: let diff show non-build defaults
Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Jason Dillaman
7adb17f831 pybind/mgr/rbd_support: use image ids to detect duplicate tasks
This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.

Fixes: https://tracker.ceph.com/issues/41032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 12:17:46 -04:00
Kefu Chai
b6b3724f04
Merge pull request #29501 from tchaikov/wip-journal/test_JournalRecorder.cc
journal: always shutdown JournalRecoreder before destructing it

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 23:57:45 +08:00
Sage Weil
712409cac4 common/config: do no include multiple 'default' values
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 10:47:04 -05:00
Sridhar Seshasayee
1034782d4c mon/OSDMonitor: Add standalone test for mon_memory_target
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
e4d236492a mon/OSDMonitor: Implement config observer to handle changes to cache sizes
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
3b96417e18 mon/OSDMonitor: Use generic priority cache tuner for mon caches
Use priority cache manager to tune inc, full and rocksdb caches.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sage Weil
b2119ffb50 os/bluestore: teach fsck to tolerate per-pool omap
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:15 -05:00
Sage Weil
3cab0b3b09 os/bluestore: ondisk format change to 3 for per-pool omap
Move to ondisk format v3.  This means that per-pool omap keys may exist,
but does not imply that *all* objects use the new form until the
per_pool_omap=1 super key is also set.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
a076260e00 mon/PGMap: add data/omap breakouts for 'df detail' view
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
ab2eb6b832 osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
5a6ede0ec1 mon/PGMap: fix stored_raw calculation
The get_user_bytes() helper is a bit weird because it uses the
raw_used_rate (replication/EC factor) so that it can work *backwards*
from raw usage to normalized user usage.  However, the legacy case that
works from PG stats does not use this factor... and the stored_raw value
(in the JSON output only) was incorrectly passing in a factor of 1.0,
which meant that for legacy mode it was a bogus value.

Fix by calculating stored_raw as stored_normalized * raw_used_rate.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
b207973ae9 mon/PGMap: add in actual omap usage into per-pool stats
This is a minimal change: we aren't separately reporting data vs omap
usage (like we do in 'osd df' output for individual osds).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
d6ff61ed1f osd: report per-pool omap support via store_statfs_t
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
19f497c9b0 os/bluestore: set per_pool_omap key on mkfs
This key indicates that *all* objects put omap in the per-pool prefix and
key format.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
aa56c41ae8 osd/osd_types: count per-pool omap capable OSDs
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
e2a0717047 os/bluestore: report omap_allocated per-pool
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00