Commit Graph

102214 Commits

Author SHA1 Message Date
Kefu Chai
c345b593da crimson/osd: add_peer() in parallel
we should not add peers in background, and we can do this in parallel!

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
6d5f5ed69c crimson/osd: Heartbeat::add_reporter_peers() returns future
the sole caller `Heartbeat::update_peers()` also returns future, so
there is no reason that `add_reporter_peers()` runs in background.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
4aed49b626 crimson/osd: fix the order in OSD::update_heartbeat_peers()
before this chance, we do following things in background and in
parallel:

* add peers from pg map
* and peers from whole osdmap
* remove dead peers

but we should prioritize the peer candidates, apparently, the ones from
pgmap should have higher priority. as they are more likely to be used in
cluser and they are more important to current OSD, as they serve the
same set of PGs.

so we need to do these things in a proper order:

1. add peers from pg map
2. and peers from whole osdmap
3. remove dead peers

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
b102c44291 crimson/osd/heartbeat: avoid racing when connecting to peer
before this change, we could initialize connections to newly added peers
in parallel repeatly, if an OSD is listed in up/acting of multiple PGs
served by current OSD.

after this change, we try to add the peer to `Heartbeat.peers` first,
and then try to connect to it with the hb front/back addresses, to avoid
potential repeatly connecting to peer.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 18:23:20 +08:00
Kefu Chai
b82fd0320d crimson/osd: do not cluster_msgr.set_myaddrs() in background
we should do the initialization in foreground

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-19 16:31:26 +08:00
Samuel Just
bb2bd5f42a crimson/net: avoid warning about signed comparison with enum classes
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-09-19 15:16:23 +08:00
Kefu Chai
3d7e5b0e3f
Merge pull request #30410 from tchaikov/wip-debian-g++
debian/control: Build-Depends on g++

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
2019-09-19 10:02:02 +08:00
Kefu Chai
e7287257ef
Merge pull request #30405 from p-se/wip-pna-ubuntu-disco-deps
install-deps.sh: add support for Ubuntu Disco Dingo

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-19 10:01:07 +08:00
Kefu Chai
51dd3740db
Merge pull request #30453 from tchaikov/wip-doc-crimson
doc/dev/crimson: transpose options of compare.py

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-19 09:22:19 +08:00
David Zafman
7ea365920b
Merge pull request #30440 from dzafman/wip-41900
Revert "common: default pg_autoscale_mode=on for new pools"

Reviewed-by: Sage Weil <sage@redhat.com>
2019-09-18 09:18:46 -07:00
Kefu Chai
3914de3fc6
Merge pull request #30458 from tchaikov/wip-crimson-cbt-test
test/crimson: lower the bar for cbt test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-09-18 23:47:59 +08:00
Kefu Chai
a6bf7671a9
Merge pull request #30433 from tchaikov/wip-crimson-pgls
crimson/osd: add pgls support

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-09-18 23:38:03 +08:00
Kefu Chai
c25a9d7233 crimson/osd: add pgls_filter support
it's modeled after pgnls_filter. and use seastar::map_reduce() for
filtering the objects into the response.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:59:18 +08:00
Kefu Chai
d7be213b8c crimson/osd: return hobject_t from pgls_filter()
so we can use it as the mapper function in seastar::map_reduce().

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:57:31 +08:00
Kefu Chai
b9e2c808c6 crimson/osd: add pgls support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:57:07 +08:00
Kefu Chai
eec27f5832 crimson/osd: handle pg ops with do_pg_ops()
currently, OpsExecuter tries to load the object info for all ops to be
executed, but the PG ops do not require the existence of object
specified by op. so we need to specialize for them.

in this change,

* `OpsExecuter::execute_pg_op()` is added to handle pg ops without
  loading object info first.
* `pgnls` and `pgnls_filter` are handled by
  `OpsExecuter::execute_pg_op()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:54:39 +08:00
Kefu Chai
05b88f644e crimson/osd/OpsExecuter: pass nspace in msg to do_pg_op()
as pg ops should not require the object info, as there is chance that
the object specified in pg ops does not exist at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:51:28 +08:00
Kefu Chai
a967e40198 crimson/osd/OpsExecuter: s/do_osd_op/execute_osd_op/
will add execute_pg_op() for performing pg specific ops, which do not
need the object state.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:51:28 +08:00
Kefu Chai
8bc359fef6 test/crimson: lower the bar for cbt test
sometimes we have

13:27:59 - WARNING  - cbt      -prefill/incerta08.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.05)):: 13.0/9.53939  => rejected

i think it's acceptable

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:26:48 +08:00
Kefu Chai
044b7620e4 github: add "jenkins test crimson perf" to triggers doc
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 22:24:04 +08:00
Ilya Dryomov
d3c19d531d
Merge pull request #30452 from idryomov/wip-krbd-discard-direct
qa: avoid page cache for krbd discard round off tests

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-09-18 14:53:43 +02:00
Kefu Chai
14e73478b0
Merge pull request #30386 from tchaikov/wip-run_tox.sh-cleanup
script/run_tox.sh: remove unused code

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-09-18 20:11:23 +08:00
Kefu Chai
374425dbbb
Merge pull request #29734 from hselin/rados_bench_rand_read
tools/rados: use num ops instead of num objs for tracking outstanding IO

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-18 19:42:23 +08:00
Laura Paduano
b36d721c8a
Merge pull request #29767 from votdev/issue_41314
mgr/dashboard: Editing RGW bucket fails because of name is already in use

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
2019-09-18 11:40:45 +02:00
Kefu Chai
60a4dfd8ad doc/dev/crimson: transpose options of compare.py
they were `-a baseline -b test` before this change, and they should be
`-a test -b baseline`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-18 17:15:27 +08:00
Kefu Chai
b694cf201f
Merge pull request #30313 from cyx1231st/wip-seastar-msgr-test-racing-crimson
crimson/net: v2 racing tests, stall tests and bug fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-18 17:08:12 +08:00
Yingxin Cheng
08681e1cea crimson/net: log out_q size when execute_ready()
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
76b902b3ff crimson/net: set exit_open atomically with write_dispatching
exit_open should be set atomically with state checks and
write_dispatching changes, or there would be chances to be blocked
forever by exit_open.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
33e4ad49f3 test/crimson: add timeout feature for blockers to fail
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
e9fc2207ee test/crimson: improve loggings for test operations
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
7324f9bd6e crimson/net: chain non-discardable futures
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
5d0eff7818 crimson/net: discard and add note to discardable futures
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
ac48476fd1 crimson/net: discard future returned by close()
It's OK to discard the returned future of Connection::close() because
Messenger::shutdown() will wait for all connections closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
16f2d75582 crimson/net: explicitly discard future from with_gate()
We are using with_gate() to wrap a slow operation when we want to start
it but don't want/need to wait it to be finished in the same function.
In this case, the future returned from with_gate() is discarded, and we
need to make sure there is another continuation responsible to waiting
for the gate closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
212120f244 test/crimson: test connection stall cases for v2
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
2d2ee4e5db test/crimson: implement STALL action for breakpoint
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
e3d4a67410 test/crimson: test connect-reconnect racing cases for v2
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
c0ed141d4d crimson/net: always dispatch accept when replaced/accepted
Fix issues of missing dispatches of accept event during connection
racing.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:31 +08:00
Yingxin Cheng
8463fbbdf2 test/crimson: test connect racing cases for v2
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:32:28 +08:00
Yingxin Cheng
00cc47817a test/crimson: fix tests with the new ESTABLISHING state
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:30:36 +08:00
Yingxin Cheng
4660615235 crimson/net: add ESTABLISHING state
With the new ESTABLISHING state, connection lookup and acceptance can be
atomic, solving the issues related to racing connect.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:30:36 +08:00
Yingxin Cheng
3677af941a test/crimson: test reconnect racing cases for v2
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:30:14 +08:00
Yingxin Cheng
2145132040 test/crimson: implement BLOCK action for breakpoint
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
2b83e18592 test/crimson: implement different ways to wait in tests
wait_replaced(count): wait for connection replaced;
wait_established(): wait for all pending messages sent and received;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
2e052cf973 test/crimson: log intercepted breakpoint with counter
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
c19890e159 crimson/net: improve loggings related to state change
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
f059e2e603 crimson/net: always validate state when switch to another
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
a79d09a377 test/crimson: use bp_type_t for breakpoint type
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
75f35a5f50 test/crimson: don't send msg from lossy server
It is possible to lose message at server side due to lossy policy,
causing unit test blocked and waiting for lost replies.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:24:05 +08:00
Yingxin Cheng
9ce9fb77e8 crimson/net: add SocketConnection::peer_wins()
Also expose Connection::peer_wins() to unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
2019-09-18 12:22:09 +08:00