Commit Graph

100754 Commits

Author SHA1 Message Date
Casey Bodley
ff707aa303 rgw: user rename operates on 'stub' user
user rename creates a 'stub' user and links all buckets to that before
modifying the existing user. once everything succeeds, the stub user is
updated to match the existing user and all index objects are updated to
point at it

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 16:25:12 -04:00
Casey Bodley
b8a60eb046 rgw: add rgw_remove_user_buckets_index
helper function to remove the user:buckets object.
rgw_remove_uid_index() now omits the object version tracker argument to
avoid reading the user info

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 16:25:12 -04:00
Casey Bodley
4375204a4a rgw: simplify bucket chown
removes unused 'attrs' argument, and takes uid + display name instead
of RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 16:25:12 -04:00
Casey Bodley
17fc695047
Merge pull request #28813 from smanjara/wip-user-rename-working
rgw : Bucket mv, bucket chown and user rename utilities

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-08-07 16:05:07 -04:00
Neha Ojha
c9d2833b25
Merge pull request #29425 from aclamk/wip-bluestore-monitor-allocations
[bluestore][tools] Inspect allocations in bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-07 11:37:34 -07:00
Casey Bodley
553d97ca95 rgw: fix unlock of shared lock in RGWDataChangesLog
std::shared_mutex expects a call to unlock_shared() after lock_shared().
use the std::shared_lock guard to make it more obviously correct

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-08-07 14:02:49 -04:00
Adam Kupczyk
713f9b4d09 doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-08-07 19:18:21 +02:00
Tatjana Dehler
ce613b955a mgr/dashboard: fix RBD snapshot protected label
Fixes: https://tracker.ceph.com/issues/41153
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
2019-08-07 17:45:10 +02:00
Nathan Cutler
e6514a0567
Merge pull request #29438 from wjwithagen/wjw-fix-ceph-backport.sh-https-tracker
script/ceph-backport.sh: Use secure access for tracker.ceph.com

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-08-07 17:04:13 +02:00
Sage Weil
7e10b70615 mgr/hello: typos
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:21 -05:00
Sage Weil
6a744f65a0 mgr/hello: a better command example
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:21 -05:00
Sage Weil
9318fb0d66 mgr/hello: boilerplate for handling config options
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-07 09:27:19 -05:00
Rafael Quintero
c3a2bdeebe mgr/dashboard: E2E Dashboard Page Test Suite
Fixes: https://tracker.ceph.com/issues/40549
Fixes: https://tracker.ceph.com/issues/40550
Fixes: https://tracker.ceph.com/issues/40561

Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>
2019-08-07 07:29:21 -04:00
Yingxin Cheng
03fbb3603a crimson/net: improve get_global_seq()
Implement single global_seq and non-racing get_global_seq() interface.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-07 19:14:27 +08:00
Kefu Chai
ace8cb15a6 admin/build-doc: use python3
to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
e049f86b2d
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 17:55:06 +08:00
Kefu Chai
d9a385da8c
Merge pull request #29525 from tchaikov/wip-mgr/dashboard/run-backend-api-tests.sh
mgr/dashboard: add python-common to $PYTHONPATH

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
2019-08-07 17:29:22 +08:00
Lenz Grimmer
e83a7a623c
Merge pull request #29451 from rhcs-dashboard/wip-41047-master
github: Add CODEOWNERs for designated code-owner reviews

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2019-08-07 11:14:08 +02:00
Lenz Grimmer
62c495b8e0
mgr/dashboard: fix HACKING.rst is not rendered on github (#29521)
mgr/dashboard: fix HACKING.rst is not rendered on github

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2019-08-07 10:53:06 +02:00
Kefu Chai
8bfd3cb1e1
Merge pull request #29461 from tchaikov/wip-mgr/dashboard/test_mgr_module
qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-07 16:34:08 +08:00
Kefu Chai
3303116f56 qa/tasks/mgr/dashboard/test_mgr_module: sync w/ telemetry
* use primitive types instead of `JLeaf(the_type)` as they are
  equivalent in this context
* remove fields which are added only if certain channels are
  activated.
* allow unknown fields, as we are including various stuff
  in the report, for instance, osdmap, usage, crash info, etc.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 15:25:18 +08:00
Kefu Chai
e36332595e mgr/dashboard: add python-common to $PYTHONPATH
fix the regression introduced by 8c50be5df6, so ceph-mgr's python
modules are able to import python-common.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-07 15:00:13 +08:00
Jan Fajerski
47fecd80d8
Merge pull request #29382 from MI-OSiRIS/prometheus_scrape_interval
mgr/prometheus: Cast collect_timeout (scrape_interval) to float
2019-08-07 08:05:43 +02:00
Kiefer Chang
a1a3817659
mgr/dashboard: fix HACKING.rst is not rendered on github
The file is not rendered because of inconsistent section markers.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
2019-08-07 11:40:09 +08:00
Jianpeng Ma
e782ac6fcf os/bluestore: Don't forget sub kv_submitted_waiters.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:17:11 +08:00
Jianpeng Ma
a2f63f4681 os/bluestore: No need check !q.emtpy().
when call drain_preceding, this TransContext already added into q.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 10:10:42 +08:00
Jianpeng Ma
62049dd179 os/bluestore: narrow deferred_lock in _deferred_aio_finish.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Jianpeng Ma
b6de4ab540 os/bluestore: reduce one lock for deferred_aggressive > 0.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-08-07 09:58:16 +08:00
Sage Weil
4ac4768e23 Merge PR #29116 into master
* refs/pull/29116/head:
	osd: move heartbeat connection cleanup to helper
	osd: break con<->session cycle when removing heartbeat peers
	osd: mark down heartbeat connections on shutdown
	crimson/: move get_mnow() to ShardServices, pass to heartbeat
	crimson/osd: stubs for get_mnow, get_hbstamps
	crimson/osd/heartbeat: adapt to new MOSDPing fields
	crimson/osdmap_service: add get_mnow(), get_up_epoch()
	osd/PeeringState: take HeartbeatStamps refs for current interval
	osd: track clock delta between peer OSDs
	osd: add get_mnow() interface to OSDService, PG, PeeringState
	osd: record startup_time
	osd: some minor refactoring/cleanup in handle_osd_ping

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-08-06 16:22:14 -05:00
Sage Weil
f35b5e5fa4 Merge PR #29498 into master
* refs/pull/29498/head:
	qa/tasks/ceph_manager: remove race from all_active_or_peered()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-06 16:21:58 -05:00
Willem Jan Withagen
32732b4b7e script/ceph-backport.sh: Use secure access for tracker.ceph.com
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
2019-08-06 22:54:41 +02:00
Patrick Donnelly
bbd63e0a7c
Merge PR #29509 into master
* refs/pull/29509/head:
	doc: indicate imperative mood for commit titles

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-06 13:23:23 -07:00
Patrick Donnelly
e72f590477
doc: indicate imperative mood for commit titles
This is convention in the kernel and Ceph but is not clearly documented.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-08-06 11:18:21 -07:00
Rafael Quintero
7e408632b2 mgr/dashboard: Write E2E test to verify that Manager modules have editing functionality
Fixes: https://tracker.ceph.com/issues/40823

Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>
2019-08-06 14:05:35 -04:00
Sage Weil
25f28e610f common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
If a kubernetes pod spec specifies a limit of X, then the pod gets both
the limits.memory and requests.memory resource fields set, and rook passes
those as POD_MEMORY_LIMIT and POD_MEMORY_REQUEST environment variables.

This is a problem if only the limit is set, because we will end up
setting our osd_memory_target (and, in the future, other *_memory_targets)
to the hard limit, and the daemon will inevitably reach that threshold
and get killed.

Fix this by also looking at the POD_MEMORY_LIMIT value, and applying the
ratio (default: .8) to it, and setting our actual target to the min of
that and the POD_MEMORY_REQUEST.

Also, set the "default" target to ratio*limit, so that it will apply in
general when no request is specified.

When both request and limit are 10M, we then see

        "osd_memory_target": {
            "default": "800000000000",
            "env": "800000000000",
            "final": "800000000000"
        },

In a more "normal" situation where limit is 10M and request is 5M, we get

        "osd_memory_target": {
            "default": "800000000000",
            "env": "500000000000",
            "final": "500000000000"
        },

If only limit is specified (to 10M), we get

        "osd_memory_target": {
            "default": "800000000000",
            "final": "800000000000"
        },

Fixes: https://tracker.ceph.com/issues/41037
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Sage Weil
77b634ad08 common/config: let diff show non-build defaults
Notably, this includes things from set_val_default(), which may be set
at runtime.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 12:50:36 -05:00
Jason Dillaman
7adb17f831 pybind/mgr/rbd_support: use image ids to detect duplicate tasks
This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.

Fixes: https://tracker.ceph.com/issues/41032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 12:17:46 -04:00
Kefu Chai
b6b3724f04
Merge pull request #29501 from tchaikov/wip-journal/test_JournalRecorder.cc
journal: always shutdown JournalRecoreder before destructing it

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-08-06 23:57:45 +08:00
Sage Weil
712409cac4 common/config: do no include multiple 'default' values
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 10:47:04 -05:00
Sridhar Seshasayee
1034782d4c mon/OSDMonitor: Add standalone test for mon_memory_target
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
e4d236492a mon/OSDMonitor: Implement config observer to handle changes to cache sizes
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sridhar Seshasayee
3b96417e18 mon/OSDMonitor: Use generic priority cache tuner for mon caches
Use priority cache manager to tune inc, full and rocksdb caches.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2019-08-06 20:22:16 +05:30
Sage Weil
b2119ffb50 os/bluestore: teach fsck to tolerate per-pool omap
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:15 -05:00
Sage Weil
3cab0b3b09 os/bluestore: ondisk format change to 3 for per-pool omap
Move to ondisk format v3.  This means that per-pool omap keys may exist,
but does not imply that *all* objects use the new form until the
per_pool_omap=1 super key is also set.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
a076260e00 mon/PGMap: add data/omap breakouts for 'df detail' view
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
ab2eb6b832 osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
5a6ede0ec1 mon/PGMap: fix stored_raw calculation
The get_user_bytes() helper is a bit weird because it uses the
raw_used_rate (replication/EC factor) so that it can work *backwards*
from raw usage to normalized user usage.  However, the legacy case that
works from PG stats does not use this factor... and the stored_raw value
(in the JSON output only) was incorrectly passing in a factor of 1.0,
which meant that for legacy mode it was a bogus value.

Fix by calculating stored_raw as stored_normalized * raw_used_rate.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
b207973ae9 mon/PGMap: add in actual omap usage into per-pool stats
This is a minimal change: we aren't separately reporting data vs omap
usage (like we do in 'osd df' output for individual osds).

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
d6ff61ed1f osd: report per-pool omap support via store_statfs_t
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
19f497c9b0 os/bluestore: set per_pool_omap key on mkfs
This key indicates that *all* objects put omap in the per-pool prefix and
key format.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00
Sage Weil
aa56c41ae8 osd/osd_types: count per-pool omap capable OSDs
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-06 09:24:14 -05:00