Commit Graph

79028 Commits

Author SHA1 Message Date
Adam C. Emerson
27eb13fe56 rgw: Check bucket CORS operations in policy
Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2017-09-27 15:42:27 -04:00
Sage Weil
d8c3756d26 osd/PG: handle racy preemption
If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-09-27 15:40:18 -04:00
Sage Weil
a8534cce1a osd/PG: allow local recovery reservations to be preempted
If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-09-27 15:40:18 -04:00
Sage Weil
dbc002eaa9 common/AsyncReserver: support preemption
If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-09-27 15:40:18 -04:00
David Zafman
14c5c834b9 ceph-objectstore-tool: performance: Add --pgid for faster search for object by name
Fixes: http://tracker.ceph.com/issues/21327

Signed-off-by: David Zafman <dzafman@redhat.com>
2017-09-27 12:35:41 -07:00
Adam C. Emerson
16de0fc1c5 rgw: Check bucket versioning operations in policy
Add code to check s3:GetBucketVersioning and s3:PutBucketVersioning
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2017-09-27 14:35:59 -04:00
Casey Bodley
a5b3db7acb rgw: include SSE-KMS headers in encrypted upload response
Fixes: http://tracker.ceph.com/issues/21576

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-09-27 14:26:29 -04:00
John Spray
88163749b5 mgr: respect perf counter prio_adjust in MgrClient
This awkwardly involves re-ordering some definitions
in perf_counters.h in order to refer to the prio
names defined in PerfCountersBuilder.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:22 -04:00
John Spray
0f531f7871 test: update perfcounters test for priority in output
Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:22 -04:00
John Spray
99352ceced qa: add mgr module selftest task
The module self test commands give us a chance to
catch any other ceph changes that change something
that a module was relying on reading.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:22 -04:00
John Spray
ad5a31efbe mgr/prometheus: remove explicit counter list
These have had their priorities bumped up to
USEFUL, so they'll appear in the default
get_all_counters output.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:22 -04:00
John Spray
ac8320f23d mon: elevate priority of many perf counters
We can be quite liberal here, because mons are
small in number.  However, we don't want to expose
KV database counters at this database from OSDs, so
use the prio_adjust mechanism for that.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:22 -04:00
John Spray
a1cc4ba299 osd: upgrade a bunch of perf counters to PRIO_USEFUL
These are broadly the OSD-wide IO stats, which happen
to also be the ones that were named in the
prometheus plugin until I changed it to be
priority-based.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
66f61eeda6 common: PerfCountersBuilder helper for priorities
Let the caller set a priority as the defaul, to enable them
to create a bunch at a given priority.  This is just a
convenience.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
76e1ba52b1 mgr/prometheus: add a self-test command
Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
6776d4645a mgr/influx: remove file-based config
...and also trim down the configuration to what's really
needed.  In general users don't need to pick and choose
metrics.  We could add it back if there was a strong
motivation.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
125294ab9d mgr/influx: enable self-test without dependencies
The idea of self-test commands is that they're self
contained and just exercise the module's calls
to the Ceph-side.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
59b48e7660 mgr/influx: revise perf counter handling
- Use new get_all_perf_counters path
- Consequently get counters for all daemons, not just OSD
- Tag stats with ceph_daemon rather than osd_id, as some
  stats appear from more than one daemon type
- Remove summing of perf counters, external TSDB and/or queries
  can do this.
- Remove mgr_id tag: this would change depending on which
  mgr was active, which is certainly not desirable.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
8d1277fa5c mgr: omit module list in beacon logging
This is useful in itself, but awkward when dealing
with logs generally, because it means that when you
grep on the name of a module, you get mostly beacon
messages rather than the log messages from the
module.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:21 -04:00
John Spray
39ab28ed47 mgr: define perf counter constants in mgr_module
So that modules can consume perf counter data
intelligently without having to hunt around
in C land for these constants and redefine them.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
30a74ce343 ceph.in: use PRIO_INTERESTING as daemonperf threshold
Using PRIO_USEFUL as the threshold for what goes into
time series databases.  I'm claiming that we have
more "useful" counters than fit on the screen,
so daemonperf's "a screen's worth" threshold
should be at the "interesting" level.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
29a71c35c3 mon: set some priorities on perf counters
Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
eb524c272c mgr/prometheus: tag stats by daemon name
Using osd=0 or similar tags was problematic because
daemons of different types have some same-named
counters (e.g. MDS and OSD both have objecter
perf counters).

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
11137aa269 mgr/prometheus: use new get_all_perf_counters interface
Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
ba08fc1008 common: used fixed size int for perf counter prio
...to avoid any ambiguity in allowed range and
make clear how to encode it down the wire.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:20 -04:00
John Spray
f304f84cfb mgr: transmit perf counter prio to the mgr
Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:20:16 -04:00
John Spray
e631f1a727 common: always include priority in perf counter dump
JSON output with inconsistent sets of members is
annoying to use on the receiving side.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:18:01 -04:00
John Spray
9a42d4255d mgr: add get_all_perf_counters to MgrModule interface
This is for use by modules that dump counters
in bulk, e.g. to a TSDB.

Signed-off-by: John Spray <john.spray@redhat.com>
2017-09-27 14:18:01 -04:00
Jan Fajerski
13b1236b96 pybind/mgr/prometheus: export cluster-wide pg stats, not per osd
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:01 -04:00
Jan Fajerski
e7704fa9cc pybind/mgr/prometheus: add more osd metadata
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:01 -04:00
Jan Fajerski
d4ba07d044 pybind/mgr/prometheus: don't get perf counters that are not in schema
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:01 -04:00
Jan Fajerski
fa25d31263 pybind/mgr/prometheus: add mon and osd perf counters to export
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
d99a506ed3 pybind/mgr/prometheus: add index page, export metrics under metrics/
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
f6e2e36ba7 pybind/mgr/prometheus: export selected perf_counters
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
2bea381469 pybind/mgr/prometheus: export osd and pool metadata
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
c288624eed pybind/mgr/prometheus: actually emit reported pg counts
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
ead0973d7d pybind/mgr/prometheus: no need to wait for notify event
If stats or perf counters are not available they won't be emitted.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:18:00 -04:00
Jan Fajerski
5e4b4b5ea2 pybind/mgr/prometheus: no need to convert perf_schema to ordered_dict
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:17:59 -04:00
Jan Fajerski
76d1918724 pybind/mgr/prometheus: add device_class label to osd metrics
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:17:59 -04:00
Jan Fajerski
49b3ff83cd pybind/mgr/prometheus: add cluster wide metrics; no perf counters for now
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:17:59 -04:00
Jan Fajerski
177afcc788 pybind/mgr/prometheus: prefix metrics with 'ceph'; replace :: with _
Both follow prometheus best practices. While : is a legal metric
character, "Exposed metrics should not contain colons, these are for
users to use when aggregating."

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2017-09-27 14:17:59 -04:00
mhdo2
e345fe3c57 doc/mgr: add influx plugin docs
Signed-off-by: My Do <mhdo@umich.edu>
2017-09-27 14:17:59 -04:00
mhdo2
68ae26c014 mgr/influx: added influx plugin
Signed-off-by: My Do <mhdo@umich.edu>
2017-09-27 14:17:59 -04:00
Adam C. Emerson
f9d1ae1d15 rgw: Check payment operations in policy
Add code to check s3:GetBucketRequestPayment and
s3:PutBucketRequestPayment operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
2017-09-27 13:58:32 -04:00
Sage Weil
2237624846 Merge pull request #17978 from alram/rbdmap-fix
rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2017-09-27 12:57:33 -05:00
J. Eric Ivancich
7ba15ebab8 Merge commit 'b6374d40bcaa4b73d440bb6d8ed42838dec57566' into wip-bring-in-latest-dmclock
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
2017-09-27 13:37:51 -04:00
J. Eric Ivancich
b6374d40bc Squashed 'src/dmclock/' changes from a9e777f08f..93f760c57c
93f760c57c Merge pull request #40 from ivancich/wip-change-client-rec-init
824d92dd3d Merge pull request #38 from ivancich/wip-improve-next-request-return
941d1bef54 Change initialization of IndIntruHeapData to C++'s value-initialization to better future-proof the code. Since at the momeent they are scalars they'll be zero-initialized (i.e., to zero). However if they ever become something more complex, their default constructors will be called.
19153d979f Merge pull request #39 from ivancich/wip-delta-rho-plugin
a94c4e086c Allow the calculations of rho and delta to be handled by a "tracker" specified via template parameter (i.e., by static polymorphism). The tracker follows a simple interface constisting of three functions and one static function.
856a26c466 Clarify code surrounding the return value of do_next_request.
b632cfda4f Merge pull request #37 from ivancich/wip-fix-uninit-data
e6df585153 The coverity scan published in ceph-devel on 2017-09-21 revealed some uninitialized data in a constructor. This fixes that.
165a02542d Merge pull request #34 from TaewoongKim/anticipate
72e4df95cf Make anticipation_timeout configurable with config file
2f06d632d5 Add anticipation duration that keeps from resetting tag values to the current time

git-subtree-dir: src/dmclock
git-subtree-split: 93f760c57c75b9eb88382bcba29fcac3ce365e7f
2017-09-27 13:37:50 -04:00
Patrick Donnelly
bfc4902495
mds: fix conf types
This correct an assertion failure.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2017-09-27 09:29:39 -07:00
Patrick Donnelly
8a5d71bf45
mds: fix whitespace
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2017-09-27 09:29:23 -07:00
Casey Bodley
f3e0e0171d Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning
test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-09-27 10:26:49 -04:00