* CMakeLists.txt:
always pass "EXACT" to find_package(Python3).
because per cmake document, "EXACT" only takes effect when
<Package>_FIND_VERSION_COUNT is greater than 1, where <Package>
is "Python3". see also cmake/modules/FindPython/Support.cmake
* cmake/modules/AddCephTest.cmake:
drop redundant find_package(Python3) calls. since Python3 is
a mandatory requirement for building Ceph, we only need a
single call of find_package(Python3..) in the top of the source
tree. the only possible case to repeat it is to ensure that we
have the correct version of Python3 used in following CMake
script. but there is no need to repeat it if we just want to
ensure that we have a python3 interpretor in place.
* cmake/modules/Distutils.cmake:
always pass "EXACT" to find_package(Python3).
we should always pass EXACT to find_package() when finding python3,
this is a follow-up of e2babdfae8
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
y2c.py is like a compiler which translates .yaml to .cc and .h files,
it does not have access to all .yaml files. to validate the dangling
see-also issue, we need to do this with a "linker".
in this change, validate-options.py is introduced to check if any of
option name included by the see-also property is valid.
Fixes: https://tracker.ceph.com/issues/51483
Signed-off-by: Kefu Chai <kchai@redhat.com>
this allows us to pass extra argument to the script or executable to
further customize its behavior when testing.
also fix all the callers of add_ceph_unittest(), as there is no need to
repeat the executable, which is also used as the name of the test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change should allow us to decouple "ninja tests" from "ctest".
in other words, we can just run
ctest -R run-tox-python-common -V
without running "ninja tests" first. before this change
${name}-venv is added as a dependency of "tests" target.
after this change,
setup-venv-for-${name} is added as a test, which is in turn a test of
run-tox-${name}, so we can just
ctest -R run-tox-${name}
now for preparing the venv and then testing the tox test of ${name}.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* GIT_SHALLOW=TRUE, so we don't pull the full git history,
as we don't care about it.
* UPDATE_DISCONNECTED=TRUE, to skip the UPDATE step, this change
somehow works around
https://gitlab.kitware.com/cmake/cmake/-/issues/19703. otherwise
cmake keeps building liburing.
Signed-off-by: Kefu Chai <kchai@redhat.com>
before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.
after this change, py3 is added only if the caller does not specify any
TOX_ENVS.
this change helps with the readability.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
python version for python binding and ceph-mgr embedded python
interpreter anymore. as before switching to python3-only build,
we can build python3 and python2 python bindings, and ceph-mgr
can use either of them. but after switching to python3-only
build, ceph-mgr has to use whatever python version used to
build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
parameter anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Otherwise it will result in:
[ 59%] preparing venv for mgr-dashboard
New python executable in /home/jenkins/workspace/ceph-master/build/mgr-dashboard-virtualenv/bin/python2.7
Also creating executable in /home/jenkins/workspace/ceph-master/build/mgr-dashboard-virtualenv/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter /usr/local/bin/python2.7
Even though the Cmake options contained:
'-DWITH_PYTHON2=OFF' '-DWITH_PYTHON3=3' '-DMGR_PYTHON_VERSION=3'
Mostly due to the fact that
src/tools/setup-virtualenv.sh
default to:
PYTHON_BINARY="python2.7"
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Refactor CMake add_tox_test to automatically add py27 and/or py3 to
provided toxenvs.
Refactor tox.ini:
- Remove requirements-{py27,py3}.txt, as python release dependant
packages can be handled with PEP 508 syntax.
- Remove develepment dependencies from requirements.
- Move pycodestyle settings to separate section.
- Add flake8 check and other checkers (rst, naming, etc). Some of them
are commented out for future clean-ups (Ceph trackers have been opened)
- Pycodestyle removed, as flake8 is a wrapper for pycodestyle.
- Add instafail plugin to report failures immediately
- Add timeout plugin to limit max run time (sometimes test_tasks hangs)
- Remove unused dependencies (lru_cache, pluggy)
Test and code linting fixes:
- Unused imports
- Fixes to HACKING.rst
Doc:
- Update HACKING.rst
Add conftest.py to mock imported modules (rados, rbd, cephfs), and mock
also rados Error and OSError Exceptions.
Fixes: https://tracker.ceph.com/issues/40487
Fixes: https://tracker.ceph.com/issues/41152
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
* add two optional keyword parameters to `add_tox_test()`
* use the default `TOX_PATH` parameter instead passing it explicitly,
as it is always `${CMAKE_CURRENT_SOURCE_DIR}`
* drop the code to guess the tox envs in `run_tox.sh`, and always pass
them explicitly in CMake script using the `--tox-envs` argument.
Signed-off-by: Kefu Chai <kchai@redhat.com>
so we can prepare the venv for tox with `make tests`. and it's optional,
as run_tox.sh will check for it and setup a venv if it's not around.
Signed-off-by: Kefu Chai <kchai@redhat.com>
use the ones shiped from the latest cmake. which
* enables us to find the recent python intepreter and development files,
* find intepreter and development in a single `find_pacakge()` command,
simpler this way and less error prone.
and to accomodate this change:
* all `PYTHON${PYTHON_VERSION}_*` variables are renamed to
`Python${PYTHON_VERSION}_*` if we use `find_package(Python2...)` or
`find_package(Python3...)` to find python2 or python3 instead of using
`find_package(Python...)`.
* use "2" explicitly when using python2, as `Python_*` variables are not
defined anymore
* when compiling python support of ceph-mgr, continue using `Python_*`
variables. because we find the python interpreter and development
files using `find_pacakge(Python...)` for ceph-mgr.
Signed-off-by: Kefu Chai <kchai@redhat.com>
It is rather had to overide a once set timeout during CTesting.
This is the best next compromise. Disadvantage is that a rebuild
is required.
But it helps in my Jenkins testing timely catching run-away programs.
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
* add FindGMock.cmake which allows user to use the libgtest-dev
shipped by distro
* add GMock::{GMock,Main}, GTest::{GTest,Main} targets to be
compatible with FindGTest.cmake and FindGMock.cmake, which
expose the built libraries with properties adhered to
them. so the consumer of them can import them in a better way.
* update tests to drop the commands like
set_target_properties(foo PROPERTIES COMPILE_FLAGS ${UNITTEST_CXX_FLAGS}),
as they are already linked against gmock and gtest.
Signed-off-by: Kefu Chai <kchai@redhat.com>
and remove src/test/gtest-parallel submodule, because gtest-parallel is
only useful for running tests. and not all end-users are interested in
running test not to mention running them in parallel. so, to avoid
including gtest-parallel scripts in the dist tarball. it'd be better to
make it optional, and an external project.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Unittests are run sequentially and could take a long while to run.
This commit is about using gtest-parallel on some of them which are
known to be very slow due to this sequentiality.
To enable the parallel features, the 'parallel' argument just have to be
added to the add_ceph_unittest() call like in :
-add_ceph_unittest(unittest_throttle)
+add_ceph_unittest(unittest_throttle parallel)
This commit impact the following tests :
Test name Before After (in seconds)
unittest_erasure_code_shec_all: 212 43
unittest_throttle 15 5
unittest_crush 9 6
unittest_rbd_mirror 79 21
Total 315 75
This commit saves 240 seconds (4 minutes) per build.
Note it exist several other long tests but can't be parallelized since
there is explicit dependencies in the order to run the subtests.
Those stay sequential.
Signed-off-by: Erwan Velu <erwan@redhat.com>
so cmake will error on "non-existent dependency in add_dependencies", we
add dependency on CONFIGURE_FILE() output, which is wrong, as its output
is not a target, and is generated when cmake runs. so remove them from
dependencies.
regarding to ceph_ver.h, its path is also wrong. it is created under
${CMAKE_BINARY_DIR}/src/include. so this is another reason to remove it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
distutils' install command installs all stuff under $build-base, which
includes the temporary files. and rpm packaging does not like::
Installed (but unpackaged) file(s)
and to support both py2 and py3, we should encode the python version
into the path, otherwise the built files will be overwritten.
Signed-off-by: Kefu Chai <kchai@redhat.com>
please note "make test" is used by cmake to run tests, so we cannot just
repurpose it to *build* them.
* AddCephTest.cmake: depends on "tests"
* CMakeLists.txt: let "check" depend on "tests"
* src/CMakeLists.txt: update the run-tox tests
* run-make-check.sh: use "make tests" and "ctest" instead of "make check"
* ceph-detect-init/CMakeLists.txt: let "tests" depend on
"ceph-detect-init"
* ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk"
Signed-off-by: Kefu Chai <kchai@redhat.com>
Replaced relative paths in shell scripts in
test/ceph_objectstore_tool.py and init-ceph.in to
work with CEPH_FOO environment variables set in cmake.
Also added CEPH_BUILD_DIR environment variable set
to CMAKE_BINARY_DIR. It is used in init-ceph and
ceph_objectstore_tool.py.
Signed-off-by: Ali Maredia <amaredia@redhat.com>
make check targets now use add_ceph_test, or
add_ceph_unittest to be added to CTest. Renamed and
reordered targets in tests/CMakeLists.txt
Signed-off-by: Ali Maredia <amaredia@redhat.com>
