Commit Graph

100751 Commits

Author SHA1 Message Date
Kefu Chai
3a27c3c800
Merge pull request #29615 from tchaikov/wip-qa/tasks/mgr/dashboard/test_health
qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
2019-08-13 13:59:28 +08:00
Kefu Chai
b1c05009f9 qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health
fix regressions introduced by a076260e and d6ff61ed

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-13 11:30:09 +08:00
Kefu Chai
f13c7c83d9
Merge pull request #29342 from Jeegn-Chen/wip-scrub-extended-sleep
osd: support osd_scrub_extended_sleep

Reviewed-by: David Zafman <dzafman@redhat.com>
2019-08-13 09:09:52 +08:00
Kefu Chai
9666fabc67
Merge pull request #29522 from majianpeng/bluestore-optimization
os/bluestore: deferred IO notify and locking optimization

Reviewed-by: Sage Weil <sage@redhat.com>
2019-08-13 09:08:00 +08:00
Kefu Chai
308125d3f0
Merge pull request #29418 from junxfl/test1
OSD: modify n.cookie to op.notify.cookie

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-13 09:06:52 +08:00
Kefu Chai
1b8ee44dcb
Merge pull request #29571 from tchaikov/wip-41177
ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-08-13 09:04:58 +08:00
Casey Bodley
1937f74c33
Merge pull request #29605 from yaozongyou/remove-unused-marco
common: remove unused _STR and STRINGIFY macro

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-08-12 10:14:52 -04:00
Casey Bodley
456fd7c1c1
Merge pull request #29553 from cbodley/wip-rgw-http-unlock-suspend
rgw: http client drops lock before suspending coroutine

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
2019-08-12 10:04:40 -04:00
Yao Zongyou
a140dba8c8 common: remove unused _STR and STRINGIFY macro
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
2019-08-12 20:49:30 +08:00
Jan Fajerski
84498bfcb4
Merge pull request #29547 from jan--f/c-v-always-log-to-stdout
ceph-volume: never log to stdout, use stderr instead
2019-08-12 14:13:19 +02:00
Kefu Chai
aed743aee4
Merge pull request #29076 from tchaikov/wip-crimson-rep-op
crimson/osd: implement replicated write

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-08-12 18:51:46 +08:00
Kefu Chai
9759005452
Merge pull request #29378 from cyx1231st/rfc-seastar-msgr-lossless-master
crimson/net: lossless policy for v2 protocol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:50:40 +08:00
Kefu Chai
4f7d7769c0 crimson/osd: log if the dest of send_to_osd() is not valid
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Kefu Chai
f95c69d0df crimson/osd: do not return "void"
to avoid potential confusions.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Kefu Chai
11ac2af314 crimson/osd: write pg info after done with peering
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Kefu Chai
a7a52efffd crimson/osd: handle MOSDRepOp
* add a `RepRequest` operation which is blocked by `ConnectionPipeline`
  and `PGPipeline`. these two pipelines are modeled after their
  counterparts of `ClientRequest`.
* add these two blockers to `PG` and `OSDConnectionPriv` accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Kefu Chai
4add5fd47b crimson/osd: replicate transaction to peers
* handle `MOSDRepOpReply` message in osd, and pass it all the way down
  to `PGBackend`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Kefu Chai
e26ba3aeed crimson/osd: implement PeeringListener::on_activate()
see also f7b55ec144

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 18:01:46 +08:00
Yingxin Cheng
acca474339 crimson/net: minor logging cleanup
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
5491bc48ae crimson/net: throw bad_peer_address when reconnect address doesn't match
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
c41c44b2e9 crimson/net: REPLACING state to resolve racing and retain session
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
d1cd196981 crimson/net: handle fault for READY, CONNECTING and ACCEPTING
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
d75c9e884a crimson/net: WAIT state and backoff for client
Client goes to WAIT state when it is delayed to reconnect, or wants to
be replaced by a newly established socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
8ea10a6a75 crimson/net: SERVER_WAIT state for accepting server
Server wait for peer client close the socket at SERVER_WAIT.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:45 +08:00
Yingxin Cheng
f8053d08ee crimson/net: STANDBY state for lossless server or peer
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:38 +08:00
Yingxin Cheng
dd59586ef0 crimson/net: allow REPLACING state wait for protocol exit
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:22:23 +08:00
Yingxin Cheng
49a08e8bc3 crimson/net: send AckFrame for lossless policy
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:18:15 +08:00
Yingxin Cheng
6cacf1f7b2 crimson/net: maintain the sent queue for lossless policy
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:17:43 +08:00
Yingxin Cheng
492263962c crimson/net: reset write state with reset_write()
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:52 +08:00
Yingxin Cheng
babc9c24fd crimson/net: allow connecting state reentrant
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:52 +08:00
Yingxin Cheng
675a50326c crimson/net: reset handshake status when connecting/accepting
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:52 +08:00
Yingxin Cheng
b7c7dc0b26 crimson/net: pending_q to store the pending(sending) messages
We cannot left the pending messages in the out_q, because with lossless
policy, they can be partially sent and even acknowledged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:52 +08:00
Yingxin Cheng
4fa1c4c07d crimson/net: wait_write_exit() to wait for writer stopped
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:51 +08:00
Yingxin Cheng
b3f1e56d6c crimson/net: is_queued() to check if there's any pending writes
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:51 +08:00
Yingxin Cheng
04f8a35d79 crimson/net: fix variables for stateful connection
server_cookie, client_cookie, connect_seq and global_seq are identifiers
of a stateful connection.

We already have some related implementations, but they are stub code
when implement lossy policy and cannot work properly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 17:02:42 +08:00
Jeegn Chen
3bfb5c2621 osd: support osd_scrub_extended_sleep
1. always take osd_scrub_sleep for manually initiated
   scrubs
2. when scrub_time_permit() return true for scheduled
   ones, the existing osd_scrub_sleep is used
3. when scrub_time_permit() return false for scheduled
   ones, there may be 2 scenarios
   3.1 if osd_scrub_extended_sleep <= osd_scrub_sleep,
       let's take osd_scrub_sleep
   3.2 otherwise, let's take osd_scrub_extended_sleep

Fixes: http://tracker.ceph.com/issues/40955
Signed-off-by: Jeegn Chen <jeegnchen@tencent.com>
2019-08-12 16:54:36 +08:00
Yingxin Cheng
469a9cda73 crimson/net: clean up, exsiting_conn and existing_proto
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 16:34:45 +08:00
Yingxin Cheng
014a662b20 crimson/net: next_step_t for explicit decision of next state
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 16:34:45 +08:00
Yingxin Cheng
b41af731b4 crimson/net: introduce 3 ways to abort the active protocol state
* abort_in_fault(): a fault is happening and needs to be handled.
* abort_protocol(): abort the current protocol state due to preemptive
                    state change.
* abort_in_close(): close this connection and abort the current protocol
                    state due to some fatal error.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-08-12 16:34:38 +08:00
Kefu Chai
2e2414b3df ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing
there is chance that we could use an OSD which does not have incmap of a
certain epoch for rebuilding the monstore. and since OSD does not read
and store the incmap if the MOSDMap message already has the fullmap of
that fullmap, and if an OSD does not have previous fullmap, monitor
will just send it the fullmao. so it's not unusual that an OSD has
a fullmap of some epoch without corresponding incmap.

Fixes: https://tracker.ceph.com/issues/41177
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-12 13:06:01 +08:00
Matt Benjamin
801d2f0449
Merge pull request #28157 from Kriechi/docs-rgw-ldap
docs: improve rgw ldap auth options
2019-08-11 20:45:29 -04:00
Josh Durgin
3f18ed55aa
Merge pull request #28227 from sseshasa/monCachePriority
mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-09 14:23:39 -07:00
Casey Bodley
bc45261470
Merge pull request #29540 from cbodley/wip-rgw-user-rename
rgw: followup for 'user rename'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-08-09 16:57:25 -04:00
Casey Bodley
13f1595335
Merge pull request #29558 from theanalyst/rgw-cache-lock
rgw: fix unlock of shared lock in RGWCache

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-08-09 13:03:35 -04:00
Abhishek Lekshmanan
2b6dbe31c8 rgw: fix unlock of shared lock in RGWCache
similar to https://github.com/ceph/ceph/pull/29538/ we unlock a shared_lock with
unlock causing a crash. Also scope the single line if statements to make the
code more concise

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-08-09 17:54:22 +02:00
Sage Weil
f011c13547 Merge PR #29292 into master
* refs/pull/29292/head:
	os/bluestore: warn on no per-pool omap
	os/bluestore: fsck: warning (not error) by default on no per-pool omap
	os/bluestore: fsck: int64_t for error count
	os/bluestore: default size of 1 TB for testing
	os/bluestore: behave if we *do* set PGMETA and PERPOOL flags
	os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP
	os/bluestore: fsck: only generate 1 error per omap_head
	os/bluestore: make fsck repair convert to per-pool omap
	os/bluestore: teach fsck to tolerate per-pool omap
	os/bluestore: ondisk format change to 3 for per-pool omap
	mon/PGMap: add data/omap breakouts for 'df detail' view
	osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
	mon/PGMap: fix stored_raw calculation
	mon/PGMap: add in actual omap usage into per-pool stats
	osd: report per-pool omap support via store_statfs_t
	os/bluestore: set per_pool_omap key on mkfs
	osd/osd_types: count per-pool omap capable OSDs
	os/bluestore: report omap_allocated per-pool
	os/bluestore: add pool prefix to omap keys
	kv/KeyValueDB: take key_prefix for estimate_prefix_size()
	os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
	os/bluestore: make omap key helpers Onode methods
	os/bluestore: add Onode::get_omap_prefix() helper
	os/bluestore: change _do_omap_clear() args

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-08-09 10:40:45 -05:00
Alfredo Deza
8363d89a4d
Merge pull request #29528 from tchaikov/wip-build-doc-with-python3
admin/build-doc: use python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>
2019-08-09 11:17:19 -04:00
Matt Benjamin
a7b29647fd
Merge pull request #29560 from linuxbox2/wip-rgwf-advance
rgw_file: dont deadlock in advance_mtime()
2019-08-09 10:01:03 -04:00
Sage Weil
b8501164ef os/bluestore: warn on no per-pool omap
Signed-off-by: Sage Weil <sage@redhat.com>
2019-08-09 08:21:18 -05:00
Jan Fajerski
b8d6dcbe9f ceph-volume: never log to stdout, use stderr instead
We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-08-09 14:26:16 +02:00