should be specific when enabling codeready-builder repos, there is
chance that some repos are just not available, while the required one
is. for instance, "codeready-builder-for-rhel-8-x86_64-eus-debug-rpms"
might not be available. and in that case, `install-deps.sh` just fails.
so in this change, only the required one is enabled. see also
https://fedoraproject.org/wiki/EPEL
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise the copr repo won't be enabled. and dnf will bail out with
following message:
Do you really want to enable copr.fedorainfracloud.org/ktdreyer/ceph-el8? [y/N]: Error: Safe and good
answer. Exiting.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Also install the correct dnf-utils package rather than yum-utils.
Fixes: https://tracker.ceph.com/issues/42504
Co-Authored-By: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.
Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>
The binary /usr/bin/rpmspec was recently moved to rpm-build, breaking
install-deps.sh on openSUSE Tumbleweed. The package is not strictly
needed for SLE-15-SP* and openSUSE Leap 15.*, but it doesn't hurt to
have it, and will future-proof these distros from this regression.
Putting the dependency in the spec file does not address the issue,
because /usr/bin/rpmspec must be available before install-deps.sh runs it to
determine the dependencies, but it's nice to have it explicitly listed there,
since it *is* a dependency of "make check" on SUSE distros.
SUSE versions < 15 are of no interest in master/octopus+.
Fixes: https://tracker.ceph.com/issues/42612
Signed-off-by: Nathan Cutler <ncutler@suse.com>
it was used for removing the kitware repo, and in
7265b55d09, we switched to a rebuilt
version hosted in chacra, and it was more than two months ago,
presumably, none builder has kitware repo anymore. so, let's remove
the cleanup code.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ld 2.26 segment faults when linking librados.so in
https://github.com/ceph/ceph/pull/16715 .
while ld 2.28 always survives.
binutils 2.28 is rebuilt and uploaded to chacra
binutils/master/7fa393306ed8b93019d225548474c0540b8928f7/ubuntu/xenial,
so let's use it instead.
the source deb package comes from
http://ppa.launchpad.net/jonathonf/binutils/ubuntu
i tried to use the binutils 2.30 source package from bionic,
but it has build dependency of dpkg-dev (>= 1.19.0.5) which cannot be
fulfilled on xenial. so without more changes, we cannot get
binutils 2.30 built on xenial using this source package.
Fixes: https://tracker.ceph.com/issues/42596
Signed-off-by: Kefu Chai <kchai@redhat.com>
some build dependencies are still missing in PowerTools and EPEL8, so we
built and pushed them to sepia so it can be used before they are ready
in these repos.
Signed-off-by: Kefu Chai <kchai@redhat.com>
python3-devel depends on python-rpm-macros, which in turn depends on
python-srpm-macros. the latter offers `python3_pkgversion` macro, which
is necessary for prepare the build dependencies of rpm packages.
this change avoids the unnecessary `$yumdnf` calls, and to simplify the
code.
Signed-off-by: Kefu Chai <kchai@redhat.com>
the reason why we need to install these macros is to solve the
egg-chicken problem -- to set `_python_buildid` and `python3_pkgversion`
so that we can prepare the build dependencies and install them. in which,
`_python_buildid` is defined using `python3_pkgversion`. this macro is
offered by python-srpm-macros.
the other macros, like `python3_sitelib` and `__python3` are offered by
`python3-rpm-macros`.
this change also avoid the issue if we install `*rpm-macros` on CentOS8:
Error:
Problem: package R-rpm-macros-1.1.0-2.el8.noarch requires /usr/bin/Rscript, but none of the providers can be installed
- package R-rpm-macros-1.1.0-2.el8.noarch requires R-core, but none of the providers can be installed
- conflicting requests
- nothing provides libRblas.so()(64bit) needed by R-core-3.6.1-1.el8.x86_64
- nothing provides openblas-Rblas needed by R-core-3.6.1-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Signed-off-by: Kefu Chai <kchai@redhat.com>
there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.
Signed-off-by: Kefu Chai <kchai@redhat.com>
in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.
but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.
it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".
as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.
once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7
Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>
* move `for_make_check` to the beginning of script, as FreeBSD will also
use this variable
* extract `preload_wheels_for_tox()` function out to improve readability
* call `preload_wheels_for_tox()` only if `for_make_check` is true
Signed-off-by: Kefu Chai <kchai@redhat.com>
otherwise we will fail to install the build dependencies of
`lazy-object-proxy` from the wheelhouse. as `lazy-object-proxy` does not
add `setuptools_scm` in its `setup.py`, instead it lists
`setuptools_scm` in `setup.cfg` and `pyproject.toml` as a `build-system`
requires. but unfortunately, `pip download` only downloads the
install/run-time dependencies at this moment. and `lazy-object-proxy`
does not offer binary package for at least python2.7.
ideally, `pip download` should collects its dependencies like
Collecting setuptools_scm>=3.3.1 (from lazy-object-proxy->astroid<3,>=2.2.0->pylint->-r requirements-lint.txt (line 1))
so we need to use `pip wheel` do download build-time dependencies
see also https://github.com/pypa/pip/issues/6222
Signed-off-by: Kefu Chai <kchai@redhat.com>
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Refactor CMake add_tox_test to automatically add py27 and/or py3 to
provided toxenvs.
Refactor tox.ini:
- Remove requirements-{py27,py3}.txt, as python release dependant
packages can be handled with PEP 508 syntax.
- Remove develepment dependencies from requirements.
- Move pycodestyle settings to separate section.
- Add flake8 check and other checkers (rst, naming, etc). Some of them
are commented out for future clean-ups (Ceph trackers have been opened)
- Pycodestyle removed, as flake8 is a wrapper for pycodestyle.
- Add instafail plugin to report failures immediately
- Add timeout plugin to limit max run time (sometimes test_tasks hangs)
- Remove unused dependencies (lru_cache, pluggy)
Test and code linting fixes:
- Unused imports
- Fixes to HACKING.rst
Doc:
- Update HACKING.rst
Add conftest.py to mock imported modules (rados, rbd, cephfs), and mock
also rados Error and OSError Exceptions.
Fixes: https://tracker.ceph.com/issues/40487
Fixes: https://tracker.ceph.com/issues/41152
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
two reasons:
- do not rely on kitware repo so we have better control of the
cmake3: the `debian` directory is tracked by
https://github.com/tchaikov/ceph-cmake
- chacra repo also offers aarch64 build of cmake3
Signed-off-by: Kefu Chai <kchai@redhat.com>
to enable us to build on xenial, install newer cmake.
cmake 3.10.2 is the version offered by bionic.
with this change, we can safely require cmake 3.10.2 in our cmake
script. as EPEL7 offers cmake 3.13
Signed-off-by: Kefu Chai <kchai@redhat.com>
two reasons
* GCC-9 features more checks. so let's use it!
* crimson targets the hardware + toolchain + kernel after 1+ years,
so it would be great if we can compile and test crimson using
the toolchain which is ubiquitous on most mainstream distos.
Signed-off-by: Kefu Chai <kchai@redhat.com>
because `install-deps.sh` is executed using `source`, we have to pass
these options using env variables. but before this change, `WITH_SEASTAR` is used directly,
while `FOR_MAKE_CHECK` is checked and translated to a local variable
`for_make_check`. which, in my opinion, has better readability.
so, in this change, `WITH_SEASTAR` is translated to `with_seastar`
variable in `install-deps.sh`.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The FreeBSD dependency install specifies devel/kbr5 instead of devel/krb5
Fixes: https://tracker.ceph.com/issues/40050
Signed-off-by: Thomas Johnson <NTmatter@gmail.com>
Prebuilt boost-1.67 packages are only available for limited arch on Ubuntu
(x86_64 for bionic, x86_64 and arm64 for xenial). This will cause install
failure on missing platforms.
Add a new env variable to allow user to build and install from source
while the default option is to install pkgs from ceph repo.
Change-Id: I4259733dd40a638d1bd5d1578a64ecaaa6490121
Signed-off-by: Jun He <jun.he@arm.com>
so `yum-builddep` can have access to the latest macros for preparing the
build dependencies
Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
seems devtools-8 is not packaged on aarch64. and we are suffering from
ICE on centos7 when building with GCC-7.3. so let's try GCC-8.2!
Fixes: http://tracker.ceph.com/issues/38892
Signed-off-by: Kefu Chai <kchai@redhat.com>
There is a few essentail differences between the way versioning
and a version-map is interpreted for binutils ld and LLVM ld.
The easy way out for the time being is to use the one that works.
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
this change reverts "install-deps.sh: workaround setuptools' dependency on six"
This reverts commit 30ce5e5546.
tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.
long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.
Signed-off-by: Kefu Chai <kchai@redhat.com>
as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.
Fixes: http://tracker.ceph.com/issues/37707
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Remove the continuous release repos for CentOS and Virtuozzo 7 as they
should no longer be needed since http://tracker.ceph.com/issues/13997 is
no longer relevant and the newer versions of selinux packages pulled in
by the build system are causing problems for systems without CR repos
enabled.
Fixes: https://tracker.ceph.com/issues/37335
Signed-off-by: Alfredo Deza <adeza@redhat.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Fedora 29 still ships a Python 2 binary, but some of Ceph's build
dependencies are only available in py3 versions there. In other
words, from F29 on, it is no longer possible to do a py2 Ceph build
on Fedora, even if a python2 binary exists on the system.
If that were not enough, the Python 2 that ships with Fedora 29 is
linked against a non-compatible version of OpenSSL.
Before this commit, install-deps.sh was overriding the spec file's
Python build setting based on the presence or absence of a python2
binary. As the bug cited below indicates, this was not a good idea.
It's better for the spec file to be explicit about which OS versions
are py2 and which are py3, and just stick to that.
Fixes: http://tracker.ceph.com/issues/37301
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
* extract boost install function into a helper, so we don't need to
repeat it when adding support for new distros.
* install ceph-libboost-* for bionic build also. we check for the
existence in run-make-check.sh, so there is no need to update
the build script elsewhere.
Signed-off-by: Kefu Chai <kchai@redhat.com>
On ubuntu 18.04, "gcc -dumpversion" output "7", it result in
that below check will return false:
dpkg --compare-versions 7 ge 7.0
Then, this script will install other gcc.
Actully, the full gcc version "gcc -dumpfullversion" output is
"7.3.0", than below check will turn true:
dpkg --compare-version 7.3.0 ge 7.0
So, there's no need to install other gcc.
In case of hitting error on ubuntu16.04, use below parameter:
gcc -dumpfullversion -dumpversion
Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
python-saml depends on dm.xmlsec.binding. which links against ltdl. so
without libtool-ltdl-devel, we will have
/opt/rh/devtoolset-7/root/usr/libexec/gcc/aarch64-redhat-linux/7/ld:
cannot find -lltdl
Signed-off-by: Kefu Chai <kchai@redhat.com>
this should address the failures when running install-deps.sh, like
Downloading/unpacking virtualenv
Running setup.py egg_info for package virtualenv
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
warnings.warn(msg)
error in virtualenv setup command: 'extras_require' must be a
dictionary whose values are strings or lists of strings containing valid
project/version requirement specifiers.
Complete output from command python setup.py egg_info:
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
warnings.warn(msg)
error in virtualenv setup command: 'extras_require' must be a dictionary
whose values are strings or lists of strings containing valid
project/version requirement specifiers.
this only happens on very old virtualenv shipped with RHEL7.4
Signed-off-by: Kefu Chai <kchai@redhat.com>
Running `npm install` for the dashboard pulls dependenies with git.
Under some setups, git was not installed when building the frontend.
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
ceph-libboost1.67 repo is setup to satisfy the libboost dependencies
used when building ceph. now it offers arm64 and amd64 builds for xenial.
* install-deps.sh: add ceph-libboost1.67 repo hosted on chacra if the
building host does not have any of the listed ceph-libboost1.67
packages installed
* run-make-check.sh: add "-DWITH_SYSTEM_BOOST=ON
-DBOOST_ROOT=/opt/ceph" if include/boost/config.hpp is found
under /opt/ceph
See-also: http://tracker.ceph.com/issues/25186
Signed-off-by: Kefu Chai <kchai@redhat.com>
awk uses some tests that the native FreeBSD awk does not support:
like: BEGIN{print 0 < 90}
And TESTDIR is not set when calling ceph-helpers from smoke.sh
So fix with keeping the archive in /tmp
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Unfortunately the mapping between release number and codename (which is only
relevant for Debian and Ubuntu btw) is not available from /etc/os-release.
In that one respect, lsb_release was "better".
However, when I weigh the advantages of obtaining that mapping from an external
tool, with the (substantial) risk that the external dependency might cause
trouble on one or more supported distros (to say nothing of the non- or
semi-/pseudo-supported ones), against the work involved in maintaining a
hard-coded mapping (negligible), the needle on my scale immediately swings
toward eliminating the dependency.
Also, I see this commit as part of the longer-term effort to completely expunge
lsb_release from our codebase. See git log --grep lsb_release.
For another example of an external distro-detection tool (albeit one that was
included in Python 2) gone awry, see http://tracker.ceph.com/issues/18163.
Signed-off-by: Nathan Cutler <ncutler@suse.com>
The $() form is preferable to `` because folks (like me) might be using
` as a keyboard shortcut to GNU Screen, causing havoc to ensue whenever
copy-pasting the ` character.
Signed-off-by: Nathan Cutler <ncutler@suse.com>
* ceph.spec.in: c-ares-devel and fmt-devel are installed as
dependencies fo seastar.
* install-deps.sh: libfmt-dev is missing in xenial, so we only
install libc-ares-dev for ubuntu in install-deps.sh .
* cmake: build libfmt if it's not found.
Signed-off-by: Kefu Chai <kchai@redhat.com>
25d5122b55 introduced code into
install-deps.sh which was intended to set a variable "with_seastar" to the
string "true" or "false" depending on whether or not an environment
variable WITH_SEASTAR is defined.
As originally implemented, the conditional is always true, so with_seastar
is set to "true" unconditionally. Fix this by putting $WITH_SEASTAR in
double quotes.
Signed-off-by: Nathan Cutler <ncutler@suse.com>
we are running into following error recently
gpg: requesting key BA9EF27F from hkp server keyserver.ubuntu.com
gpgkeys: key 1E9377A2BA9EF27F can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver communications error: keyserver helper general error
gpg: keyserver communications error: unknown pubkey algorithm
gpg: keyserver receive failed: unknown pubkey algorithm
so use the local copy of the pubkey instead
Signed-off-by: Kefu Chai <kchai@redhat.com>
in some case, the $builddepcmd will failed without any "error:" output.
so we should check the exit status to handle it.
Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>
