Commit Graph

55880 Commits

Author SHA1 Message Date
Patrick Donnelly
42e02369f0
doc: fix standby replay config
I tried using these settings in tests without success. The correct config names
are prefixed with "mds".

Fixes: http://tracker.ceph.com/issues/16664

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2016-07-12 15:44:00 -04:00
Kefu Chai
6ed1c41fed ceph-detect-init: fix the py3 test
the mock_open() does not support "for i in open()", and readline() and
readlines() support was added in py3.3 and py3.4 so for better backwards
compatibility, we should change the code being tested to use the plain
read() call.
and also use open(path) instead of open(path, 'rb') for simplicity,
otherwise we need to use bytestring for comparison and pass the same
parameters to mock.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-13 02:25:07 +08:00
Kefu Chai
4366145a09 test: ceph-detect-init: fix the indent
this fixes the pep8 failure:
```
pep8 runtests: commands[0] | flake8 ceph_detect_init tests
tests/test_all.py:121:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:125:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:129:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:133:17: E128 continuation line under-indented for
visual indent
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-13 02:22:15 +08:00
Haomai Wang
e489cd4f07 Merge pull request #10090 from yuyuyu101/wip-remove-async-lock
msg/async: remove file event lock

Reviewed-by: Kefu Chai <kefu@redhat.com>
2016-07-13 00:26:16 +08:00
Haomai Wang
96943eeaad ceph_test_msgr: clean previous sent queue infos
1. A -> B
2. goto standby
3. B mark down
4. A reconnect to B
5. got reset session and dispatch remote reset
6. because remote reset is executed in DispatchQueue, it will be delayed
7. A -> B successfully and begin to send message
8. assert because we found the first message is missing but it's reasonble

if policy.resetcheck is true

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:59:43 +08:00
Haomai Wang
d1268a6aa8 ceph_test_msgr: use ceph log infrastructure to output
because we want to get the right log sequence which mixes ceph logginer and
cerr. Otherwise, cerr output make the logs a little disordered.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
f04f21892c msg/async/AsyncConnection: make sure no STATE_WAIT -> STATE_STANDBY
Otherwise if message in queue, we will continue to reconnect right now,
it won't meet our expectation that we want our connect request delay

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
63d5c252ae test_msgr: add server_msgr cleanup
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
64405c956d msg/async/AsyncConnection: add REPLACING write status to indicate replacing
When replacing, we don't expect any AsyncConnection dispatch new event which
will cause thing chaos

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
97e6850855 msg/async: hold lock avoid state changing
Fixes: http://tracker.ceph.com/issues/16554
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
9a47ea4fe7 test_async_driver: accept new api changes
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
8143fffb30 msg/async: dont delete time event when exchange eventcenter
When replacing and someone called mark_down, it will delete_time_event which
isn't allowed. Because we're exchaning EventCenter now!

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:51:27 +08:00
Haomai Wang
fe5770f500 msg/async/AsyncConnection: simplify shutdown socket flow
Let cleanup resources things all in shutdown_socket

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:50:47 +08:00
Haomai Wang
37629d587d msg/async: add assert to ensure create/delete file event within thread
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:49:13 +08:00
Haomai Wang
24b3c27db1 Event: remove file_lock
Now all EventCenter will exists within one thread, it will let all file events
api changes without locks.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:48:33 +08:00
Haomai Wang
af1d3ed36f msg/async/AsyncMessenger: make sure all connection cleanup handler finished
Fixes: http://tracker.ceph.com/issues/16552
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:48:33 +08:00
Haomai Wang
fdb9c7dba3 msg/async: remove close socket line
because if we are in STATE_CLOSED, fd must be -1

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:48:33 +08:00
Haomai Wang
cbe5ef6385 msg/async: make sure worker started before let msgr ready
When we create event thread, it need a little time to enter event loop(like
calling set_owner), if caller is going to call create_file_event before event
thread enter event loop, it will trigger assert.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:48:33 +08:00
Haomai Wang
f5d1c05b8f msg/async: make EventCenter notify file event creating when set_owner
EventCenter::init is called by other thread instead of event thread, so we
need to move create_file_event to set_owner which is called by event thread.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:48:32 +08:00
Haomai Wang
e66a48f854 msg/async/AsyncConnection: swap eventcenter when replacing
Previously we only exchange fd when replacing, now we will introduce dpdk
plugin in the near future. It needs all fd used locally which not like
kernel socket shared by all cores.

So we need to add EventCenter swapping to let each socket is associated to
EventCenter.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:47:53 +08:00
Kefu Chai
fe98b0b33b Merge pull request #10153 from tchaikov/wip-no-copy-plugins
test: test_{compression_plugin,async_compressor}: do not copy plugins

Reviewed-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:43:24 +08:00
Haomai Wang
96aa327e84 msg/async/Event: fix delete_time_event leak
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:37:43 +08:00
Haomai Wang
478dd90027 AsyncMessenger: make create/delete_file_event within event thread
We are make each AsyncConnection/AsyncMessenger only modify its file event
in event thread. So make sure create/delete_file_event aren't directly called.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-12 23:37:22 +08:00
Casey Bodley
9b6759286c Merge pull request #10174 from theanalyst/test/rgw/multisite-cleanup
rgw: test_multi.py avoid creating mds

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-12 10:12:16 -04:00
Casey Bodley
55f1a34b5b Merge pull request #10211 from oritwas/wip-rgw-fix-zone-modify
rgw: remove unused realm from radosgw-admin zone modify

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-12 10:10:56 -04:00
John Spray
e1f96e2d08 Merge pull request #10256 from ceph/wip-kill-ceph_test_rados_api_tmap_migrate
test: remove ceph_test_rados_api_tmap_migrate

Reviewed-by: John Spray <john.spray@redhat.com>
2016-07-12 11:10:59 +01:00
Kefu Chai
05120d17db Merge pull request #10229 from badone/wip-libgmock_main.la-build-failures
makefile: change librgw_file_* as check_PROGRAMS

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-12 16:25:37 +08:00
Kefu Chai
53af28db6e test: remove ceph_test_rados_api_tmap_migrate
see also: http://tracker.ceph.com/issues/16144

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-12 16:08:09 +08:00
Jason Dillaman
8ad36cab7c rbd-mirror: include local pool id in resync throttle unique key
Fixes: http://tracker.ceph.com/issues/16536
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2016-07-12 00:11:22 -04:00
Casey Bodley
77143c6511 Merge pull request #10252 from jdurgin/wip-rgw-build
rgw: fix compilation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-11 23:27:48 -04:00
Josh Durgin
874de58024 rgw: fix compilation
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-11 17:38:43 -07:00
Casey Bodley
aa28d8175d Merge pull request #10205 from oritwas/wip-rgw-fix-upgrade
rgw: add missing master_zone when running with old default region config

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2016-07-11 16:44:32 -04:00
Sage Weil
dc02b37a97 Merge pull request #10040 from xiexingguo/xxg-wip-bit-alloc-tree-configurable
os/bluestore: make zone/span size of bitmap-allocator configurable

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
2016-07-11 15:18:45 -05:00
Matt Benjamin
16854d7557 Merge pull request #10075 from cbodley/wip-rgw-tenant-sync
rgw: add tenant support to multisite sync
2016-07-11 15:24:22 -04:00
Sage Weil
e0c4bb31f9 Merge pull request #8964 from songbaisen/song24
global: log which process/command sent a signal

Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-11 14:07:14 -05:00
Sage Weil
af6ececa04 Merge pull request #10195 from stiopaa1/msg_asyncmess_removeIncFile
msg/AsyncMessenger: remove unneeded include file
2016-07-11 12:29:46 -05:00
Sage Weil
c6dc92d3e5 Merge pull request #10224 from stiopaa1/msg_asyncmess_event_addConst
msg/async/Event.h: add const to member function

Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-11 12:19:08 -05:00
Sage Weil
2b370e95ed Merge pull request #10230 from stiopaa1/msg_asyncmess_Processor_accept_changeToVoid
msg/AsyncMessenger: change return type to void

Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-11 12:17:42 -05:00
Kefu Chai
be4ec38d75 Merge pull request #10152 from tchaikov/wip-16277
osd: increment stats on recovery pull also

Reviewed-by: Samuel Just <sjust@redhat.com>
2016-07-12 00:17:03 +08:00
Matt Benjamin
42785a7927 Merge pull request #10222 from ceph/wip-cmake
cmake changes for pypi

verfied f23
2016-07-11 11:57:53 -04:00
Haomai Wang
2ee2652379 Event: no need to delete_file_Event when deconstruct
Since we are going to close all epoll and cleanup resources, no need to delete
notify fd resource. And another reason is "delete_file_event" doesn't expect
other threads to call

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-11 22:43:00 +08:00
Haomai Wang
f6879fea78 async: change set_socket_option caller
Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-11 22:42:43 +08:00
Haomai Wang
2d1f7aaf40 net_handler: adjust set_socket_options to avoid read from conf
We don't want net_handler rely on config value, caller may expect to pass
different value to set_socket_options according to different socket type
like heartbeat, client socket or server socket.

Signed-off-by: Haomai Wang <haomai@xsky.com>
2016-07-11 22:41:50 +08:00
Kefu Chai
5e58249362 cmake: fix the detection of curl_multi_wait()
it was broken by b5bb77d5

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-11 21:29:11 +08:00
John Spray
3b5a6a4fa0 mds: add health warning for oversized cache
Various issues can lead to an oversized cache, which
is a precursor to an OOM condition: let's give users
a clearer message with some useful counts that might
hint at what is wrong.

Fixes: http://tracker.ceph.com/issues/16570
Signed-off-by: John Spray <john.spray@redhat.com>
2016-07-11 14:11:31 +01:00
Kefu Chai
0f55402401 pybind/rados: remove rados.c from MANIFEST.in
this fixes the warning of

```
running egg_info
creating
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info
writing
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/PKG-INFO
writing top-level names to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/top_level.txt
writing dependency_links to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/dependency_links.txt
writing manifest file
'/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/SOURCES.txt'
warning: no files found matching 'rados.c'
```

in the out-of-source tree build, rados.c is not located in the same
directory of setup.py, rados.pyx, and rados.pyd. it is in the build_dir,
for example, it could be:

  ceph/build/src/pybind/rados/pyrex/rados.c

where ceph/build is the build directory of cmake.
and sdist will include it when running egg_info, because cython's
build_ext module lists `rados.c` as one of the "source_files".

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-11 16:23:20 +08:00
Kefu Chai
c57dbd3451 cmake: pass necessary cflags to build_ext
this fixes:
```
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex
/usr/bin/gcc-6 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-fno-strict-aliasing -iquote /var/ceph/ceph/src/include -fPIC
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-
gnu/python2.7 -I/usr/include/python2.7 -c
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c -o
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex/ra
dos.o -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g
-fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -g
-fwrapv -O2 -Wall -Wstrict-prototypes -lpython2.
7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1
-Wl,-Bsymbolic-functions
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c:239:28: fatal error:
rados/librados.h: No such file or directory
 #include "rados/librados.h"
                            ^
compilation terminated.
```

seems we can not pass the CFLAGS with space in it to setup.py using env variable

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-11 16:23:20 +08:00
Kefu Chai
6a99eec653 cmake: put cythonized .c files into ${CMAKE_CURRENT_BINARY_DIR}
it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
  the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
  --cython-include-dirs to "build_ext" command

Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-11 16:23:20 +08:00
Kefu Chai
b3bea59b7e Merge pull request #9138 from cxwshawn/wip-cl-fix
osd: replace ceph:atomic_t with std::atomic in osd module.

Reviewed-by: Samuel Just <sjust@redhat.com>
2016-07-11 13:36:42 +08:00
Kefu Chai
1d7aed6911 Merge pull request #9871 from odivlad/fix-init-el7-master
master: remove SYSTEMD_RUN from initscript

Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-11 13:22:05 +08:00