Commit Graph

102361 Commits

Author SHA1 Message Date
Casey Bodley
20159f03e0
Merge pull request #30189 from linuxbox2/wip-rgw-ss3lite
rgw: crypt: permit RGW-AUTO/default with SSE-S3 headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-09-19 14:18:14 -04:00
Casey Bodley
2c82b290e9
Merge pull request #30194 from vumrao/wip-vumrao-41589
radosgw-admin: add --uid check in bucket list command

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-09-19 14:17:35 -04:00
Casey Bodley
49d0395317
Merge pull request #30332 from linuxbox2/wip-ldap-denied
rgw: ldap auth: S3 auth failure should return InvalidAccessKeyId

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-09-19 14:16:29 -04:00
Casey Bodley
c18db0c40b
Merge pull request #30384 from ZVampirEM77/bucket-create-time
rgw: add creation time information into bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-09-19 14:15:33 -04:00
Abhishek L
89945b1bf3
Merge pull request #29680 from liuchang0812/s3-lock-bugfix
rgw: DefaultRetention requires either Days or Years

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2019-09-19 19:41:00 +02:00
Sage Weil
5685bed71e osd/osd_types: fix {omap,hitset_bytes}_stats_invalid handling on split/merge
Fixes: https://tracker.ceph.com/issues/41936
Signed-off-by: Sage Weil <sage@redhat.com>
2019-09-19 11:51:18 -05:00
Venky Shankar
7cdad344b9
Merge pull request #30451 from ajarr/fix-41903
mgr/volumes: return string type to ceph-manager
2019-09-19 18:58:32 +05:30
Lenz Grimmer
bbb580e938
mgr/dashboard: Add smartctl data as an details tab of an OSD (#30324)
mgr/dashboard: Add smartctl data as an details tab of an OSD

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-09-19 12:48:51 +00:00
Samuel Just
c30fa85f1d crimson/osd: suppress warnings about unused future<> return values
These are dropped on purpose, suppress warnings.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
e541750959 crimson/osd: ShardServices::send_pg_temp() returns future
it should not run in background unless necessary.

please note, we don't have a caller of this method yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
1a8841cfe3 crimson/osd: evict object state in foreground
when doing error handling, we need to make sure the stale object state
is evicted before we return.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
c345b593da crimson/osd: add_peer() in parallel
we should not add peers in background, and we can do this in parallel!

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
6d5f5ed69c crimson/osd: Heartbeat::add_reporter_peers() returns future
the sole caller `Heartbeat::update_peers()` also returns future, so
there is no reason that `add_reporter_peers()` runs in background.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
4aed49b626 crimson/osd: fix the order in OSD::update_heartbeat_peers()
before this chance, we do following things in background and in
parallel:

* add peers from pg map
* and peers from whole osdmap
* remove dead peers

but we should prioritize the peer candidates, apparently, the ones from
pgmap should have higher priority. as they are more likely to be used in
cluser and they are more important to current OSD, as they serve the
same set of PGs.

so we need to do these things in a proper order:

1. add peers from pg map
2. and peers from whole osdmap
3. remove dead peers

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
b102c44291 crimson/osd/heartbeat: avoid racing when connecting to peer
before this change, we could initialize connections to newly added peers
in parallel repeatly, if an OSD is listed in up/acting of multiple PGs
served by current OSD.

after this change, we try to add the peer to `Heartbeat.peers` first,
and then try to connect to it with the hb front/back addresses, to avoid
potential repeatly connecting to peer.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Ricardo Dias
efe91beccf
Merge pull request #30270 from ricardoasmarques/wip-41727
mgr/dashboard: Check if `num_sessions` is available

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
2019-09-19 11:08:48 +01:00
Venky Shankar
88aa0576a6 mgr/volumes: return string type to ceph-manager
The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.

Introduced-by: 3d63cd947f
Fixes: https://tracker.ceph.com/issues/41903
Signed-off-by: Venky Shankar <vshankar@redhat.com>
2019-09-19 14:56:26 +05:30
Kefu Chai
b82fd0320d crimson/osd: do not cluster_msgr.set_myaddrs() in background
we should do the initialization in foreground

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 16:31:26 +08:00
Samuel Just
bb2bd5f42a crimson/net: avoid warning about signed comparison with enum classes
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-09-19 15:16:23 +08:00
Yang Honggang
a1331a6ece pybind/mgr/balancer/module.py: add max/min info in stats_by_root
$ ceph balancer eval-verbose hahapool
...
count_by_pool {'hahapool': {'objects': {0: 1L, 1: 1L, 2: 0L, 3: 0L, 4: 1L}, 'bytes': {0: 0L, 1: 0L, 2: 0L, 3: 0L, 4: 0L}, 'pgs': {0: 35, 1: 42, 2: 23, 3: 30, 4: 26}}}
...
stats_by_root {'default': {'objects': {'min': 0.0, 'max': 1.0, 'score': 0.2970089549436925, 'stddev': 0.5477225575051662, 'avg': 0.6, 'sum_weight': 0.6000000000000001}, 'bytes': {'min': 0, 'max': 0, 'score': 0, 'stddev': 0, 'avg': 0, 'sum_weight': 0}, 'pgs': {'min': 23.0, 'max': 42.0, 'score': 0.07354228523075643, 'stddev': 7.52994023880668, 'avg': 31.2, 'sum_weight': 0.4}}}

max/min can give us an intuitive feel of the pg/bytes/object's distribution.
In addition, sometimes we really want to know which osd hold the most
number of pgs/bytes/objects for specified pool.

Signed-off-by: Yang Honggang <yanghonggang@umcloud.com>
2019-09-19 14:00:24 +08:00
Kefu Chai
3d7e5b0e3f
Merge pull request #30410 from tchaikov/wip-debian-g++
debian/control: Build-Depends on g++

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
2019-09-19 10:02:02 +08:00
Kefu Chai
e7287257ef
Merge pull request #30405 from p-se/wip-pna-ubuntu-disco-deps
install-deps.sh: add support for Ubuntu Disco Dingo

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-19 10:01:07 +08:00
Kefu Chai
51dd3740db
Merge pull request #30453 from tchaikov/wip-doc-crimson
doc/dev/crimson: transpose options of compare.py

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-19 09:22:19 +08:00
Jason Dillaman
b1b934d24f doc/rbd: document new rbd mirror bootstrap commands
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:06:49 -04:00
Jason Dillaman
18e1d29fa3 doc/rbd: clarify cluster naming by using site-a/b
The local vs remote cluster naming has been confusing for multiple
users.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:06:49 -04:00
Jason Dillaman
c18e15a637 rbd: new 'mirror pool peer bootstrap create/import' commands
These two commands simplify the process of creating users and
exchanging keys between two clusters.

Fixes: https://tracker.ceph.com/issues/41653
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:06:49 -04:00
Jason Dillaman
41b75a0496 librbd: new RBD mirroring peer bootstrap API methods
These new methods will help facilitate the creation of keys and
the transfer of cluster connection metadata between Ceph clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:06:45 -04:00
David Zafman
7ea365920b
Merge pull request #30440 from dzafman/wip-41900
Revert "common: default pg_autoscale_mode=on for new pools"

Reviewed-by: Sage Weil <sage@redhat.com>
2019-09-18 09:18:46 -07:00
Kefu Chai
3914de3fc6
Merge pull request #30458 from tchaikov/wip-crimson-cbt-test
test/crimson: lower the bar for cbt test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-09-18 23:47:59 +08:00
Pooja
fb179b2dc1 mgr/dashboard Adds multiple selection to osd table
Fixes: https://tracker.ceph.com/issues/38091

Signed-off-by: Pooja <pooja.gautam@ts.fujitsu.com>
2019-09-18 17:43:13 +02:00
Kefu Chai
a6bf7671a9
Merge pull request #30433 from tchaikov/wip-crimson-pgls
crimson/osd: add pgls support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-09-18 23:38:03 +08:00
Kefu Chai
c25a9d7233 crimson/osd: add pgls_filter support
it's modeled after pgnls_filter. and use seastar::map_reduce() for
filtering the objects into the response.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:59:18 +08:00
Kefu Chai
d7be213b8c crimson/osd: return hobject_t from pgls_filter()
so we can use it as the mapper function in seastar::map_reduce().

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:57:31 +08:00
Kefu Chai
b9e2c808c6 crimson/osd: add pgls support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:57:07 +08:00
Kefu Chai
eec27f5832 crimson/osd: handle pg ops with do_pg_ops()
currently, OpsExecuter tries to load the object info for all ops to be
executed, but the PG ops do not require the existence of object
specified by op. so we need to specialize for them.

in this change,

* `OpsExecuter::execute_pg_op()` is added to handle pg ops without
  loading object info first.
* `pgnls` and `pgnls_filter` are handled by
  `OpsExecuter::execute_pg_op()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:54:39 +08:00
Kefu Chai
05b88f644e crimson/osd/OpsExecuter: pass nspace in msg to do_pg_op()
as pg ops should not require the object info, as there is chance that
the object specified in pg ops does not exist at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:51:28 +08:00
Kefu Chai
a967e40198 crimson/osd/OpsExecuter: s/do_osd_op/execute_osd_op/
will add execute_pg_op() for performing pg specific ops, which do not
need the object state.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:51:28 +08:00
Kefu Chai
8bc359fef6 test/crimson: lower the bar for cbt test
sometimes we have

13:27:59 - WARNING  - cbt      -prefill/incerta08.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.05)):: 13.0/9.53939  => rejected

i think it's acceptable

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:26:48 +08:00
Kefu Chai
044b7620e4 github: add "jenkins test crimson perf" to triggers doc
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:24:04 +08:00
Jason Dillaman
c0c6b25d4f cls/rbd: sanitize entity instance messenger version type
The entity_addr_t::type field does not uniquely identify the connection
and might switch back and forth between LEGACY, V2, and ALL depending on
the client, the OSD, and the OSD map feature bits. Therefore, always
fix the entity address type to ANY prior to any comparisons.

Fixes: https://tracker.ceph.com/issues/41833
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 10:03:13 -04:00
Jason Dillaman
8871c5c688 osd/PrimaryLogPG: do not append outdata to TMAPUP ops
Write operations should not have data but TMAPUP incorrectly appends
the previously read TMAP object to the outdata buferlist.

Fixes: https://tracker.ceph.com/issues/41908
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 09:52:17 -04:00
Ilya Dryomov
d3c19d531d
Merge pull request #30452 from idryomov/wip-krbd-discard-direct
qa: avoid page cache for krbd discard round off tests

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:53:43 +02:00
Patrick Seidensal
44068fc941 osd: add common smartctl output to JSON output
Enables consumers like the dashboard to provide the regular smartctl
output to users.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
2019-09-18 14:34:06 +02:00
Kefu Chai
14e73478b0
Merge pull request #30386 from tchaikov/wip-run_tox.sh-cleanup
script/run_tox.sh: remove unused code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-09-18 20:11:23 +08:00
Kefu Chai
374425dbbb
Merge pull request #29734 from hselin/rados_bench_rand_read
tools/rados: use num ops instead of num objs for tracking outstanding IO

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-18 19:42:23 +08:00
Ulrich Weigand
0065f76d53 headers: Remove __le16/__le32/__le64
These types should no longer be used; use ceph_le16/32/64 instead.

Also removes the cmake check whether those types are provided by
kernel headers (likewise for the __be types, which already were
not used anywhere).

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
2019-09-18 13:35:21 +02:00
Ulrich Weigand
8f67fec1ba headers: Remove le16/32/64_to_cpu
There are no remaining callers of these routines, which are no longer
required when using ceph_le16/32/64 (and were no-ops already).

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
2019-09-18 13:35:21 +02:00
Ulrich Weigand
080a9fae5e tools: Clean up endian handling
Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation, or where objects are only used locally
and do not require endian swaps.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
2019-09-18 13:35:21 +02:00
Ulrich Weigand
ac9b370ad6 crimson: Clean up endian handling
Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation.

Also, remove unnecessary calls to le16/32/64_to_cpu.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
2019-09-18 13:34:57 +02:00
Ulrich Weigand
36e8cd7ac6 osd: Clean up endian handling
Replace remaining uses of __le16/__le32/__le64 in contexts that are
not relevant to code generation, or where objects are only used locally
and do not require endian swaps.

Also, remove unnecessary calls to le16/32/64_to_cpu.

This change is a no-op on all architectures.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
2019-09-18 13:33:59 +02:00