Commit Graph

70154 Commits

Author SHA1 Message Date
Kefu Chai
ea093baec9 Merge pull request #13999 from tchaikov/wip-extract-creating-pgs
mon,osd: decouple creating pgs from pgmap

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-31 00:35:45 +08:00
Kefu Chai
d3e6a2869b tools/ceph-objectstore-tool: always set first_committed
otherwise the workaround of 7fb3804fb is triggered. we need to remove
that workaround later on.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
9331cedd96 mon: switch to osdmon when handling osd_pg_creates subs
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
616e970ada mon/OSDMonitor: avoid search and lookup anti-pattern
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
ebde28b683 mon/OSDMonitor: pass by reference not pointer of const param
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
f61853ae89 mon/OSDMonitor: switch to the new creating_pgs
* prime_pg_temp(): switch to the new creating_pgs
* get_trim_to(): switch to the new creating_pgs

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
a316eb3078 mon: handle MOSDPGCreated messages
MOSDPGCreated messages are used to prune the creating_pgs_by_osd_epoch
and creating_pgs, by updating created_pools. as once a pool is created
we will not send MOSDPGCreate to its acting_primary OSD anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
a2e39d91c9 osd: send pg-created message if any pg is newly created
add an option named "osd_created_report_interval" to specify the
interval to check and send the "pg_created" mesages to mon

because pg could update its state when it is still in the pg_stat_queue,
for example, to change its state to PG_STATE_CLEAN, we cannot tell if we
have sent a "pg-created" message for it or not without introducing a new
member variable in PG.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:18 +08:00
Kefu Chai
e5ea630e9f messages/MPGCreated: add MPGCreated
add a new message type MPGCreated. osd is supposed to send this message
to monitor to inform that any pg(s) is created and activated.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
43ded67243 mon/OSDMonitor: add send_pg_create() to OSDMonitor
OSDMonitor will handle the pg-create subscriptions after luminous.
1. scan new pools to get the pgs to be created
2. send pg creates using the collected pgs
3. trim the creating_pgs using the "created!" messages from OSD.

please note that we need to wait for the OSDMonitor::mapping to be fully
populated, so we cannot scan the incrementa map for creating pgs until
it is applied and accepted by other monitors.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
df6dafe1bb mon: acquire lock when accessing mon->session_map
we will access the mon->session_map for sending the osd-pg-creates
messages when finishing osdmapping in OSDMonitor, this could happen in
another thread without the protection of Monitor::lock, or in the same
thread already guarded by Monitor::lock. so instead of changing
Monitor::lock to a recursive lock, a new lock is introduced to protect
session_map.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
11f82480a0 mon/OSDMonitor: run mapping on peons also
otherwise subcriptions on peons won't get the creating_pgs notification
mapping updated. we want to send the notification from peons also. and
the notifications should be updated with the updated pg mapping.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
19f8db9692 mon/OSDMonitor: mapping is not optional anymore
as pg_creatings needs mapping to get the acting_primary osd of the
creating pg, so we can send the pg-create message to it if it subscribes
to this information, mapping should always be available now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
472c64268d mon: dispatch osd_beacon message to OSDMonitor
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
78484dc71b osd: send osd-beacon to mon periodically
add an option named "osd_beacon_report_interval" to specify the interval
to send osd-beacon.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
8f3432d5b6 messages/MOSDBeacon: add beacon msg
osd will send beacon message to monitor periodically to inform it that
"i am still alive!", previously, monitor use the pg-stats to check the
status of OSD, but since osd will only send pg stat to mgr after
luminous, we use a dedicated msg for this purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
20900e41fb mon/OSDMonitor: s/check_sub/check_osdmap_sub/
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
738f947aa9 mon/OSDMonitor: update comment in update_from_paxos()
this change updates the comment for 7fb3804fb, 97462a3 and e807770,
to reflect the reason why we need to fix latest_full in current code.
as the fix is not a workaround for cuttlefish anymore, it resolves the
issue where

0. mon.c has a latest_full of 5
1. mon.c is shutdown and out of sync with the quorum
2. mon.c starts sync
3. mon.c now has osdmap[31..50], and the latest_full is still 5.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
89810eeb31 mon/PGMonitor: remove unused variable
last_pg_scan is not used in PGMonitor::update_from_paxos(), so remove
it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
d86f15ca65 mon/PGmonitor: remove unused last_sent_pg_create
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Kefu Chai
454da5e7b2 mon: pass const variables by const ref not pointer
* PGMapUpdater::check_down_pgs(): pass a const reference to pgmap
  instead of a pointer
* PGMapUpdater::register_new_pgs(): pass a const reference to pgmap
  instead of a pointer

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 20:21:17 +08:00
Haomai Wang
1f6750063a Merge pull request #14250 from Adirl/fix_last_poll_dev
msg/async/rdma: Device::last_poll_dev must be positive

Reviewed-by: Haomai Wang <haomai@xsky.com>
2017-03-30 18:37:39 +08:00
Amir Vadai
fb7787dd58 msg/async/rdma: Device::last_poll_dev must be positive
Make Device::last_poll_dev `unsigned` - it could overlap and should not
be negative.
Also, add missing initialization.

Change-Id: I558f02292ae4f5d1ce48d9bcda5047d382604237
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-03-30 13:29:08 +03:00
Haomai Wang
20bbdb93a2 Merge pull request #14249 from Adirl/rdma-cm-4_1
msg/async/rdma: Debug prints for ibv_*

Reviewed-by: Haomai Wang <haomai@xsky.com>
2017-03-30 17:40:49 +08:00
Amir Vadai
42078045bc msg/async/rdma: Debug prints for ibv_*
Setting RDMA_DEBUG to 1, will enable debug prints for every ibv_*
function called.
This makes it easier to debug RDMA issues.

Issue: 995322
Change-Id: I49d327d5d4386b44264f5619d50f2dbc7d76dcae
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-03-30 11:42:19 +03:00
Haomai Wang
5d23e6431c Merge pull request #14245 from tchaikov/wip-revert-14201
Revert "msg/async/rdma: Debug prints for ibv_*"
2017-03-30 13:21:36 +08:00
Kefu Chai
7e0f0748ef Revert "msg/async/rdma: Debug prints for ibv_*"
This reverts commit 7d1475f627.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-30 12:58:48 +08:00
Haomai Wang
4100803cfe Merge pull request #14201 from Adirl/rdma-cm-4
msg/async/rdma: Introduce RDMAConnMgr + Debug prints

Reviewed-by: Haomai Wang <haomai@xsky.com>
2017-03-30 08:00:40 +08:00
Dan Mick
b227a70e53 Merge pull request #14214 from dmick/wip-workunit-shallow-clone
tasks/workunit.py: when cloning, use --depth=1

Reviewed-by: Kefu Chai <kchai@redhat.com>
2017-03-29 11:36:05 -07:00
Jason Dillaman
49366c20de Merge pull request #14202 from tchaikov/wip-cast-size_t-arm32
rbd: use min<uint64_t>() explicitly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-03-29 13:33:49 -04:00
Casey Bodley
4b73fcb070 Merge pull request #14122 from hrchu/rgwadmin-doc-quota
doc: correct the quota section

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2017-03-29 10:38:16 -04:00
Amir Vadai
7d1475f627 msg/async/rdma: Debug prints for ibv_*
Setting RDMA_DEBUG to 1, will enable debug prints for every ibv_*
function called.
This makes it easier to debug RDMA issues.

Issue: 995322
Change-Id: I49d327d5d4386b44264f5619d50f2dbc7d76dcae
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-03-29 17:18:39 +03:00
Amir Vadai
91bb13dced msg/async/rdma: Introduce RDMAConnMgr
Encapsulate all connection establishment stuff in a new class -
RDMAConnMgr and make it a friend class of RDMAConnectedSocketImpl.
This class will be inherited for every type of connection establishment
- Currently only TCP is supported, very soon CM will be added too.

RDMAServerConnImpl which only handle connection establishment became an
abstract class and RDMAServerConnTCP is inherting it for connections of
type TCP.

Some of the code was left in its original file and place, and therefore
it looks misplaced. This was done to make it easier to review and rebase.
Once it is accepted a cleanup patch will be sent to move the code into
the right place.

Issue: 995322
Change-Id: I8b0e163525ec80c2452f4b6481bf696968cc1e51
Signed-off-by: Amir Vadai <amir@vadai.me>
2017-03-29 17:18:16 +03:00
Kefu Chai
76d467c10b Merge pull request #14078 from tchaikov/wip-19258
mgr: shutdown py_modules in Mgr::shutdown()

Reviewed-by: John Spray <john.spray@redhat.com>
2017-03-29 22:02:51 +08:00
Sage Weil
a66d65847c Merge pull request #13337 from ifed01/wip-bluestore-minimiza-blobs2
os/bluestore: optimize blob usage when doing appends/overwrites

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-29 08:33:40 -05:00
Kefu Chai
06256a53f3 Merge pull request #14223 from tchaikov/wip-rbd-nbd-ftbfs
rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>
2017-03-29 21:21:24 +08:00
Kefu Chai
523b7cafc4 mgr: shutdown py_modules in Mgr::shutdown()
Fixes: http://tracker.ceph.com/issues/19258
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-29 19:57:42 +08:00
Kefu Chai
c12ecd984d rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS
the build failure was introduced by ff4dcf0

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-29 19:19:23 +08:00
Mykola Golub
6e77724464 Merge pull request #14079 from liupan1111/wip-support-signal-handler
rbd-nbd: support signal handle for SIGHUP, SIGINT and SIGTERM.

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
2017-03-29 12:31:12 +03:00
Dan Mick
c1309fbef3 tasks/workunit.py: when cloning, use --depth=1
Help avoid killing git.ceph.com.  A depth 1 clone takes about
7 seconds, whereas a full one takes about 3:40 (much of it
waiting for the server to create a huge compressed pack)

Signed-off-by: Dan Mick <dan.mick@redhat.com>
2017-03-28 20:09:44 -07:00
Sage Weil
97ea67fcee Merge pull request #14184 from SUSE/wip-mgr-rest-timezone
pybind/mgr/rest: don't set timezone to Chicago

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-28 21:30:18 -05:00
Sage Weil
da51861d71 Merge pull request #14199 from andihit/doc-update-default-osd-op-thread-timeout
doc: change osd_op_thread_timeout default value to 15

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-28 21:26:08 -05:00
hrchu
51a88267f0 doc: rgw: correct the quota section
Add the missing option and fix typo.

Fixes: http://tracker.ceph.com/issues/19397

Signed-off-by: Chu, Hua-Rong <hrchu@cht.com.tw>
2017-03-29 02:17:04 +00:00
Sage Weil
4a8732eba7 Merge pull request #14203 from cbodley/wip-rgw-slo-auth-identity
rgw: fix build of conflict after auth rework

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-28 17:23:36 -05:00
Casey Bodley
2e82401c91 rgw: fix build of conflict after auth rework
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-03-28 15:01:59 -04:00
Matt Benjamin
b0fc817458 Merge pull request #13764 from guihecheng/rgwfs-errcode
rgw_file: fix non-posix errcode EINVAL to ENAMETOOLONG
2017-03-28 14:31:14 -04:00
Kefu Chai
2ad4aa0c2a rbd: use min<uint64_t>() explicitly
on arm32, size_t is actually int, which cannot be compared with uint64_t
using std::min().

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-03-29 02:26:53 +08:00
John Wilkins
f2dfa64da5 Merge pull request #14108 from sven15/doc-fix-pools
doc: typo in hit_set_search_last_n

Reviewed-by: John Wilkins <jowilkin@redhat.com>
2017-03-28 10:53:48 -07:00
Sage Weil
50b9becaf8 Merge pull request #14128 from yehudasa/wip-19367
cls/user: cls_user_bucket backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-28 11:31:39 -05:00
Yuri Weinstein
9137790062 Merge pull request #13987 from shinobu-x/wp-sk-clean-up-timeindexlistctx-20170316
cls/timeindex: clean up cls_timeindex_client.h|cc

Reviewed-by: Sage Weil <sage@redhat.com>
2017-03-28 09:25:26 -07:00