* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
python version for python binding and ceph-mgr embedded python
interpreter anymore. as before switching to python3-only build,
we can build python3 and python2 python bindings, and ceph-mgr
can use either of them. but after switching to python3-only
build, ceph-mgr has to use whatever python version used to
build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
parameter anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
if `distutils_install_module("foo" ...)` is called mutiple times with
different python version, `foo-clone` will be added multiple times as a
custom target. which is not allowed:
add_custom_target cannot create target "foo-clone" because another
target with the same name already exists.
Signed-off-by: Kefu Chai <kchai@redhat.com>
allow `distutils_*()` functions to choose the python according to this
parameter, instead of implicitly using the variable of ${PYTHON_VERSION}
inherited from the caller.
Signed-off-by: Kefu Chai <kchai@redhat.com>
use the ones shiped from the latest cmake. which
* enables us to find the recent python intepreter and development files,
* find intepreter and development in a single `find_pacakge()` command,
simpler this way and less error prone.
and to accomodate this change:
* all `PYTHON${PYTHON_VERSION}_*` variables are renamed to
`Python${PYTHON_VERSION}_*` if we use `find_package(Python2...)` or
`find_package(Python3...)` to find python2 or python3 instead of using
`find_package(Python...)`.
* use "2" explicitly when using python2, as `Python_*` variables are not
defined anymore
* when compiling python support of ceph-mgr, continue using `Python_*`
variables. because we find the python interpreter and development
files using `find_pacakge(Python...)` for ceph-mgr.
Signed-off-by: Kefu Chai <kchai@redhat.com>
use the combination of add_custom_command() and add_custom_target() to
build cython extensions to avoid rebuilding the cython extension
everytime we run "make"
Signed-off-by: Kefu Chai <kchai@redhat.com>
so add_custom_target() has better idea what is generated. if any target
will use it, it can be used as a dependency.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Commit 3bde34af8a removed cython 0.29's subinterpreter check when
building the various python modules during `make`, but unforunately
they're *rebuilt* during `make install`, with the rebuild overwriting
the original build. The original fix was of course missing from the
install stage...
Fixes: https://tracker.ceph.com/issues/38788
Signed-off-by: Tim Serong <tserong@suse.com>
cython 0.29 introduced a check which prevents multiple python
subinterpreters from loading the same module:
https://github.com/cython/cython/commit/7e27c7c
Unfortunately, this completely breaks ceph-mgr. Until we can
figure out a better long term solution, this commit removes
cython's subinterpreter check, via some careful abuse of the
C preprocessor.
This works because when cython is invoked, it first generates
some C code, then compiles it. We know it's going to generate
C code including:
int __Pyx_check_single_interpreter(void) { ... }
and:
if (__Pyx_check_single_interpreter())
return NULL;
So, we can do the following:
#define void0 dead_function(void)
#define __Pyx_check_single_interpreter(ARG)=ARG ## 0
This replaces the call to __Pyx_check_single_interpreter()
with a literal 0, removing the subinterpreter check.
The void0 dead_function(void) thing is necessary because
the __Pyx_check_single_interpreter() macro also clobbers
that function definition, so we need to make sure it's
replaced with something that works as a function definition.
Fixes: https://tracker.ceph.com/issues/37472
Signed-off-by: Tim Serong <tserong@suse.com>
--single-version-externally-managed is used for packaging python eggs
to deb. so the metadata can be stored in the egg, and can be accessed
by user if necessary, see [1]. but this does not apply to users who
just want to "make install" without packaging Ceph.
the problem is, if we pass --single-version-externally-managed to
setup.py, we need to pass --record or --root also. otherwise setuptools
complains:
error: You must specify --record or --root when building system packages
so in this change, we choose not to pass
"--single-version-externally-managed", unless $DESTDIR is set.
both "ceph.spec.in" and "debian/rules" set $DESTDIR when running "make
install". so we can always check $DESTDIR to tell if current "make
install" is launched by the packaging script or not. this might not be
accurate, but good enough. assuming enduser does not use DESTDIR when
running plain "make install".
--
[1] https://wiki.debian.org/Python/FAQ#How_should_we_package_Python_eggs.3F
Signed-off-by: Kefu Chai <kchai@redhat.com>
in python's distutils.ccompiler, linker_exe is composed using CC instead
of LDFLAGS. the latter only effects how it builds (shared) library.
and put CMAKE_C_FLAGS into the cflags for the compiler for building
python C extensions, it's more consistent this way. more importantly,
if we build with ASan enabled, the canary program, a.k.a. rados_dummy.c,
won't link without proper CFLAGS.
without this change, rados.so fails to build with errors like:
/usr/bin/ld: /var/ssd/ceph/build/lib/librados.so: undefined reference to
`__asan_stack_free_10'
/usr/bin/ld: /var/ssd/ceph/build/lib/librados.so: undefined reference to
`__asan_report_exp_store8'
...
...
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
Link Error: RADOS library not found
make[3]: ***
[src/pybind/rados/CMakeFiles/cython_rados.dir/build.make:57:
src/pybind/rados/CMakeFiles/cython_rados] Error 1
Signed-off-by: Kefu Chai <kchai@redhat.com>
When building with ccache, distcc, and other compiler wrappers (such
as STLFilt):
CC='ccache gcc' CXX='ccache g++' cmake /path/to/ceph
make
python modules fail to compile since distutils try to execute the
wrapper itself without specifying the actual compiler.
Although cmake has a special magic switch for compiling with ccache
(cmake -DWITH_CCACHE=ON) other tools (distcc) are not supported, and
specifying the compiler as
CC=/whatever/compiler/is
used to work for decades, and it's a good idea to keep it working
Signed-off-by: Alexey Sheplyakov <asheplyakov@mirantis.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise, the default gcc will be used, and the $CMAKE_C_COMPILER
passed to the outer CMakeLists.txt won't kick in. moreover, if the
building script (ceph.spec for instance) could set the $PATH, and
expect that the CMakeLists.txt will use the toolchain executables
in the $PATH to build Ceph, distutils will continue using the default
$CC for linking the python bindings, on UNIX it will be gcc in the
new shell's $PATH, because we are using `install(CODE "... execute_process(
...))` for installing the python bindings. apparently, this is not
expected. because the new shell's $PATH is very likely different
from the one changed by the building script. to address this, we
should always specify the `$LDSHARED` env var explicitly.
also, pass env vars using `ENV{}` instead of the `env` command to
workaround the issue of https://cmake.org/pipermail/cmake/2015-December/062216.html,
because it's not straightforward to set environment variables with
spaces in the them using cmake. and because one cannot use add_custom_target()
in the script mode of cmake. this leave me only limited options to
fix this issue.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Skip nonexistent 'expected' Python packaging files, and add bin/ as
a possible file for symlinking during build (for Python projects with
slightly different forms, currently ceph-volume)
Signed-off-by: Dan Mick <dan.mick@redhat.com>
distutils' install command installs all stuff under $build-base, which
includes the temporary files. and rpm packaging does not like::
Installed (but unpackaged) file(s)
and to support both py2 and py3, we should encode the python version
into the path, otherwise the built files will be overwritten.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the empty .so file does not compile at all, so we need to bypass the
sanity check, if we are building docs. and what sphinx needs is just the
docstrings.
Fixes: http://tracker.ceph.com/issues/16940
Signed-off-by: Kefu Chai <kchai@redhat.com>
it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
--cython-include-dirs to "build_ext" command
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like
+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...
Signed-off-by: Kefu Chai <kchai@redhat.com>
* Distutils.cmake:
set --prefix=${CMAKE_INSTALL_PREFIX} for python packages installed using
setuptools. it was set to --prefix=/user only when $DESTDIR is set. so
if user installs ceph using -DCMAKE_INSTALL_PREFIX, these python
packages still go to /usr, which is unexpected.
* ceph-disk/CMakeLists.txt:
install script into ${CMAKE_INSTALL_SBINDIR} instead of /usr/sbin
Signed-off-by: Kefu Chai <kchai@redhat.com>
* add keyword "INSTALL_SCRIPT" to distutils_install_module(), so we can
override the install path of ceph-disk script.
* refactor the Distutils.cmake module a little bit, the ${option}
variable out lives the CODE snipplet. so we need to reset it
at the beginning.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
with existing targets like "rbd" or "rados". so we can not reuse the
name in ${name}.pyx. and instead, we should specify the target name
explicitly.
* add distutils_install_cython_module() function to build and install
cython modules.
* we can split build and install of cython module, but the install phase
always tries to build the module. so keep it this way. will look at it
later on.
* move the variables initializations into the Distutils.cmake module.
Signed-off-by: Kefu Chai <kchai@redhat.com>
