Upon re-executing `install-deps.sh` recently, I got this error message:
```
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
```
The issue was that pip was not up to date within the virtualenv that `install-deps.sh` sets up. Adding the line `python3 -m pip install --upgrade pip` right after the virtualenv is activated should help to ensure that pip is up to date in that context.
Signed-off-by: Laura Flores <lflores@redhat.com>
so we don't need to use virtualenv python package for creating a
virtualenv, the "venv" module in Python3 would suffice.
see also https://docs.python.org/3/library/venv.html
Signed-off-by: Kefu Chai <kchai@redhat.com>
instead of inventing our way for defining "make check" dependencies, use
build profile for adding "make check" specific dependencies. see
https://wiki.debian.org/BuildProfileSpec
Signed-off-by: Kefu Chai <kchai@redhat.com>
1) make reference to python3 indepedant of explicit path
2) add required py-yaml module to install list
fixes: #40731
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Using subscription-manager will fail in a container so use dnf
config-manager which should work on bare metal as well as in a
container.
Fixes: https://tracker.ceph.com/issues/50118
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
we install different versions of precompiled ceph-libboost packages
for different branches when building and testing them on ubuntu test
nodes. for instance,
- nautilus: v1.72
- octopus, pacific: v1.73
they share the same set of test nodes. and these ceph-libboost packages
conflict with each other, because they install files to the same places.
in order to avoid the confliction, we should uninstall existing packages
before installing a different version of ceph-libboost packages.
ceph-libboost${version}-dev is a package providing the shared headers of
boost library, so, in this change we check if it is installed before
returning or removing the existing packages.
Signed-off-by: Kefu Chai <kchai@redhat.com>
before this change, we use docker for running promtools offered by
a docker image, but this is not efficient, and quite a few developers
do not want to use docker for running "make check". this change was
introduced by #39246, the reason was that, in Ceph's CI process, we
are using Ubuntu/Bionic for running "make check" jobs, but prometheus
packaged by Bionic does not offer the "test rules" command. so, to
address problem, we are using "dnanexus/promtool:2.9.2" docker image
for verifying monitoring/prometheus/alerts/test_alerts.yml.
after this change, we use prometheus packaged by debian derivatives
instead of pulling a docker image.
* debian/control: add prometheus as a "make check" dependency
* install-deps.sh: partially revert
53a5816ded, as we don't need to
pull docker or start docker service for using promtool anymore.
* cmake: check if promtool is capable of running "test rules"
command, bail out if it is not.
see also: https://tracker.ceph.com/issues/49653
Signed-off-by: Kefu Chai <kchai@redhat.com>
we bump boost on regular basis. let's take the opportunity of moving to
focal to use boost v1.75.
v1.73 was used before this change. since both boost 1.75 and boost 1.73
install some files at the same places, we need to remove boost 1.73
before installing boost 1.75.
Signed-off-by: Kefu Chai <kchai@redhat.com>
WITH_ZBD is enabled for testing the build of zbd bluestore backend, and
we plan to migrate to Ubuntu/Focal for testing "make check", so need to
install libzbd when the distro version is focal.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This PR intends to add unit testing for prometheus rules using promtool. To run the tests run 'run-promtool-unittests.sh' file.
Fixes: https://tracker.ceph.com/issues/45415
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
java packaging tools are now offered by "javapackages-tools" module in
el8. so let's re-enable it. this change partially reverts
dd133840b9
Signed-off-by: Kefu Chai <kchai@redhat.com>
this was an oversight in e7e3b86762
also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.
and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ubuntu disco and ubuntu focal do not ship libboost 1.72 and up, and
we depend on libboost 1.72 or up, so it does not help to install
liboost 1.67 or libboost 1.71 anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* if WITH_JAEGER flag is specified, install-deps should mangle and update
debian/control + ceph.spec the way we do for adding crimson dependencies
with WITH_SEASTAR flag.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
* This commit introduces Jaegertracing library as package libjaeger,
pickwhich would be consumed by other ceph pacakges such as ceph-common0
* adds the following dependencies, which would be build from source
using ExternalProjectHelper.cmake +IncludeJaeger.cmake +
Build<package>.cmake scripts:
jaegertracing: v0.6.0 [added as a submodule]
opentracing: v1.6.0 [added as a submodule]
thrift: 0.13.0 [added as a submodule]
yaml-cpp: 0.6.0
json(optional)
* updates Boost to be installed instead of being build only, because
jaegertracing them during their build process.
* ceph.spec.in: introduces a default enabled jaeger packaging option,
which could be disabled using --without-jaeger flag during rpmbuild
* note: libjaeger package if enabled will be a dependency on ceph-common, ceph-mon, rgw_common and transitively will be a dependency for modules that have them as a dependency.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
It looks like CentOS 8.3 will see all repos converted to lower case and
this has been pre-empted in the CentOS stream repos so we need to be
able to enable a repo called 'PowerTools' or 'powertools'
See https://git.centos.org/rpms/centos-repos/c/b759b17
Fixes: https://tracker.ceph.com/issues/48174
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
nasm support build isa-l:AVX512 algorithm implementation while yasm
doens't support it. Install nasm assembler to build isa-l
refer to: https://github.com/yasm/yasm/issues/101
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
just like WITH_SEASTAR, WITH_ZBD is passed to install-deps.sh as an env
var, and it's disabled by default, as libzbd-devel is not yet packaged
in any distro supported by this script. we packaged and uploaded it
to http://apt-mirror.front.sepia.ceph.com/lab-extras/8/ for build test
the libzbd bluestore backend. so "with_zbd" is always false on non-el8
distros. and should only be enabled for upstream "make check" test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
alpine build recipe is stale and does not work with the latest Ceph,
also the APKBUILD for Ceph can be found at alphine's aports repo, see
https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/ceph.
so instead keeping a stale version, let's drop it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
python-sphinx creates a symlink pointing ../share/sphinx/scripts/python2/sphinx-build
to /usr/bin/sphinx-build even if python3-sphinx is already installed.
in that case, if python-sphinx is installed after python3-sphinx,
sphinx-build is in python2. and it breaks the build of master, as we are
using python3 syntax in conf.py since
e9e17b9cef.
and we are still using python2 as well as "python-sphinx" when building nautilus,
so if a build slave happen to compile nautilus before it is scheduled to
build master or a wip- branch, the doc build fails.
in this change, python-sphinx is uninstalled, if /usr/bin/sphinx-build
is found to be written in python2.
Signed-off-by: Kefu Chai <kchai@redhat.com>
since GCC offered by RHEL8/CentOS8 is able to compile ceph, there is no
need to add repos providing devtoolset anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
we needed DTS for ensuring centos 7 had decent GCC version, since we won't be
supporting el7 anymore this check becomes outdated.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
should be specific when enabling codeready-builder repos, there is
chance that some repos are just not available, while the required one
is. for instance, "codeready-builder-for-rhel-8-x86_64-eus-debug-rpms"
might not be available. and in that case, `install-deps.sh` just fails.
so in this change, only the required one is enabled. see also
https://fedoraproject.org/wiki/EPEL
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise the copr repo won't be enabled. and dnf will bail out with
following message:
Do you really want to enable copr.fedorainfracloud.org/ktdreyer/ceph-el8? [y/N]: Error: Safe and good
answer. Exiting.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Also install the correct dnf-utils package rather than yum-utils.
Fixes: https://tracker.ceph.com/issues/42504
Co-Authored-By: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.
Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>
The binary /usr/bin/rpmspec was recently moved to rpm-build, breaking
install-deps.sh on openSUSE Tumbleweed. The package is not strictly
needed for SLE-15-SP* and openSUSE Leap 15.*, but it doesn't hurt to
have it, and will future-proof these distros from this regression.
Putting the dependency in the spec file does not address the issue,
because /usr/bin/rpmspec must be available before install-deps.sh runs it to
determine the dependencies, but it's nice to have it explicitly listed there,
since it *is* a dependency of "make check" on SUSE distros.
SUSE versions < 15 are of no interest in master/octopus+.
Fixes: https://tracker.ceph.com/issues/42612
Signed-off-by: Nathan Cutler <ncutler@suse.com>
it was used for removing the kitware repo, and in
7265b55d09, we switched to a rebuilt
version hosted in chacra, and it was more than two months ago,
presumably, none builder has kitware repo anymore. so, let's remove
the cleanup code.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ld 2.26 segment faults when linking librados.so in
https://github.com/ceph/ceph/pull/16715 .
while ld 2.28 always survives.
binutils 2.28 is rebuilt and uploaded to chacra
binutils/master/7fa393306ed8b93019d225548474c0540b8928f7/ubuntu/xenial,
so let's use it instead.
the source deb package comes from
http://ppa.launchpad.net/jonathonf/binutils/ubuntu
i tried to use the binutils 2.30 source package from bionic,
but it has build dependency of dpkg-dev (>= 1.19.0.5) which cannot be
fulfilled on xenial. so without more changes, we cannot get
binutils 2.30 built on xenial using this source package.
Fixes: https://tracker.ceph.com/issues/42596
Signed-off-by: Kefu Chai <kchai@redhat.com>
some build dependencies are still missing in PowerTools and EPEL8, so we
built and pushed them to sepia so it can be used before they are ready
in these repos.
Signed-off-by: Kefu Chai <kchai@redhat.com>
python3-devel depends on python-rpm-macros, which in turn depends on
python-srpm-macros. the latter offers `python3_pkgversion` macro, which
is necessary for prepare the build dependencies of rpm packages.
this change avoids the unnecessary `$yumdnf` calls, and to simplify the
code.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the reason why we need to install these macros is to solve the
egg-chicken problem -- to set `_python_buildid` and `python3_pkgversion`
so that we can prepare the build dependencies and install them. in which,
`_python_buildid` is defined using `python3_pkgversion`. this macro is
offered by python-srpm-macros.
the other macros, like `python3_sitelib` and `__python3` are offered by
`python3-rpm-macros`.
this change also avoid the issue if we install `*rpm-macros` on CentOS8:
Error:
Problem: package R-rpm-macros-1.1.0-2.el8.noarch requires /usr/bin/Rscript, but none of the providers can be installed
- package R-rpm-macros-1.1.0-2.el8.noarch requires R-core, but none of the providers can be installed
- conflicting requests
- nothing provides libRblas.so()(64bit) needed by R-core-3.6.1-1.el8.x86_64
- nothing provides openblas-Rblas needed by R-core-3.6.1-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Signed-off-by: Kefu Chai <kchai@redhat.com>
there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.
Signed-off-by: Kefu Chai <kchai@redhat.com>
in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.
but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.
it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".
as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.
once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7
Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>
* move `for_make_check` to the beginning of script, as FreeBSD will also
use this variable
* extract `preload_wheels_for_tox()` function out to improve readability
* call `preload_wheels_for_tox()` only if `for_make_check` is true
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise we will fail to install the build dependencies of
`lazy-object-proxy` from the wheelhouse. as `lazy-object-proxy` does not
add `setuptools_scm` in its `setup.py`, instead it lists
`setuptools_scm` in `setup.cfg` and `pyproject.toml` as a `build-system`
requires. but unfortunately, `pip download` only downloads the
install/run-time dependencies at this moment. and `lazy-object-proxy`
does not offer binary package for at least python2.7.
ideally, `pip download` should collects its dependencies like
Collecting setuptools_scm>=3.3.1 (from lazy-object-proxy->astroid<3,>=2.2.0->pylint->-r requirements-lint.txt (line 1))
so we need to use `pip wheel` do download build-time dependencies
see also https://github.com/pypa/pip/issues/6222
Signed-off-by: Kefu Chai <kchai@redhat.com>
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Refactor CMake add_tox_test to automatically add py27 and/or py3 to
provided toxenvs.
Refactor tox.ini:
- Remove requirements-{py27,py3}.txt, as python release dependant
packages can be handled with PEP 508 syntax.
- Remove develepment dependencies from requirements.
- Move pycodestyle settings to separate section.
- Add flake8 check and other checkers (rst, naming, etc). Some of them
are commented out for future clean-ups (Ceph trackers have been opened)
- Pycodestyle removed, as flake8 is a wrapper for pycodestyle.
- Add instafail plugin to report failures immediately
- Add timeout plugin to limit max run time (sometimes test_tasks hangs)
- Remove unused dependencies (lru_cache, pluggy)
Test and code linting fixes:
- Unused imports
- Fixes to HACKING.rst
Doc:
- Update HACKING.rst
Add conftest.py to mock imported modules (rados, rbd, cephfs), and mock
also rados Error and OSError Exceptions.
Fixes: https://tracker.ceph.com/issues/40487
Fixes: https://tracker.ceph.com/issues/41152
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
two reasons:
- do not rely on kitware repo so we have better control of the
cmake3: the `debian` directory is tracked by
https://github.com/tchaikov/ceph-cmake
- chacra repo also offers aarch64 build of cmake3
Signed-off-by: Kefu Chai <kchai@redhat.com>
to enable us to build on xenial, install newer cmake.
cmake 3.10.2 is the version offered by bionic.
with this change, we can safely require cmake 3.10.2 in our cmake
script. as EPEL7 offers cmake 3.13
Signed-off-by: Kefu Chai <kchai@redhat.com>
two reasons
* GCC-9 features more checks. so let's use it!
* crimson targets the hardware + toolchain + kernel after 1+ years,
so it would be great if we can compile and test crimson using
the toolchain which is ubiquitous on most mainstream distos.
Signed-off-by: Kefu Chai <kchai@redhat.com>
because `install-deps.sh` is executed using `source`, we have to pass
these options using env variables. but before this change, `WITH_SEASTAR` is used directly,
while `FOR_MAKE_CHECK` is checked and translated to a local variable
`for_make_check`. which, in my opinion, has better readability.
so, in this change, `WITH_SEASTAR` is translated to `with_seastar`
variable in `install-deps.sh`.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The FreeBSD dependency install specifies devel/kbr5 instead of devel/krb5
Fixes: https://tracker.ceph.com/issues/40050
Signed-off-by: Thomas Johnson <NTmatter@gmail.com>
Prebuilt boost-1.67 packages are only available for limited arch on Ubuntu
(x86_64 for bionic, x86_64 and arm64 for xenial). This will cause install
failure on missing platforms.
Add a new env variable to allow user to build and install from source
while the default option is to install pkgs from ceph repo.
Change-Id: I4259733dd40a638d1bd5d1578a64ecaaa6490121
Signed-off-by: Jun He <jun.he@arm.com>
so `yum-builddep` can have access to the latest macros for preparing the
build dependencies
Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
