Add another workaround for changes to the ceph container such that
the ceph.repo file is deleted from /etc/yum.repos.d. Now
we look for a new env var CEPH_IS_DEVEL (which I hope is not going
to get changed any time soon) and uses that to decide if we should
reinstall the release rpm from the ci system or from the official
releases download site.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Recent changes to the ceph ci images are removing the repo files, as
well as some other minor changes. We're working with ceph maintainers
to come up with a clean solution for all, but in the meantime
re-installing the ceph-release rpm recreates the file we need to
continue.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Now that we have removed the `yum update` step, it doesn't make sense to
install the matching development packages based on the version already
present with the base container image. This is due to the fact that
their availability is not always guaranteed. Instead leave it up to DNF
to figure out if higher versions are available with the repositories.
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
In general it is not desirable to blindly update packages as a whole
while building another from a base container image.
Historically this step was required due to the introduction of version
specific installation[1] of packages i.e, we extract the package version
that comes with the base container image and try to install the matching
development libraries which might be unavailable close to a new release
happening in upstream. In order to overcome this short gap we came up
with the idea of `yum update`[2] to fetch whatever is the latest and
then extract the version for further installation of development
libraries.
This seemed to work until we discovered a different issue where updated
versions for particular dependencies are pushed to standard repositories
causing problems[3] during `yum update`.
Ceph repositories(and packages) are now more robust and DNF is capable
of handling such situations to figure out the new/updated versions for
packages even if a match is not found with the already installed package
versions. Ideally it can never be the case that matching packages for
each version are missing from a particular repository directory(only the
links to the directories is supposed to change).
Thus in our best interest we avoid running `yum update`.
[1] https://github.com/ceph/go-ceph/pull/331
[2] https://github.com/ceph/go-ceph/pull/510
[3] https://github.com/ceph/go-ceph/pull/1038
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
Work around errors related to protobuf package locations on centos.
This is triggered by our squid jobs failing with errors like:
```
22.40 Problem: protobuf-3.14.0-13.el9.i686 from appstream does not
belong to a distupgrade repository
22.40 - package protobuf-compiler-3.14.0-13.el9.x86_64 from @System
requires protobuf = 3.14.0-13.el9, but none of the providers can be
installed
22.40 - cannot install both protobuf-3.14.0-14.el9.x86_64 from
appstream and protobuf-3.14.0-13.el9.x86_64 from @System
22.40 - cannot install both protobuf-3.14.0-14.el9.x86_64 from
appstream and protobuf-3.14.0-13.el9.x86_64 from appstream
22.40 - cannot install the best update candidate for package
protobuf-3.14.0-13.el9.x86_64
22.40 - problem with installed package
protobuf-compiler-3.14.0-13.el9.x86_64
22.40 (try to add '--allowerasing' to command line to replace
conflicting packages or '--skip-broken' to skip uninstallable packages
or '--nobest' to use not only best candidate packages)
------
```
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Add libradosstriper devel and debug packages to the test container
image. This is in preparation for adding a future `rados/striper`
package to go-ceph that will wrap these APIs.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Reorganize the RUN command in the Dockerfile that sets up distro based
repositories and ceph packages. Move the `&&` to the beginning of the
lines to highlight when commands begin (and end). Change the indent
level of items that are arguments to commands to further clarify
what is part of what.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The centos 8 stream distribution is now EOL and the yum repo mirrors are
now offline. However, our CI is designed to test older released versions
of ceph where the container images are still using centos 8 stream.
Borrow and adapt a line from the ceph-csi project in order to use the
centos 8 archives (vault) until either we stop testing these versions of
ceph or the ceph project changes the base distro for these release
versions.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
With CentOS Stream 8 reaching end of builds phase most of our ceph
images are migrating their bases to CentOS Stream 9. Since they come
with curl by default in a minimal rpm package we switch to attempt the
install by directly linking the binary than the package name. This is
due to the conflict between curl and curl-minimal packages in CentOS
Stream 9 images.
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
This bumps the Go version of the test container to 1.21.8, which is the
oldest supported Go version as of now.
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
Currently the tests on the released reef image fails due to an RPM
transaction error when updating nfs-ganesha. Since go-ceph doesn't
need ganesha packages to be up-to-date to ensure that
lib{cephfs,rados,rbd}-devel are installed we _temporarily_ disable
ganesha repo when running yum update.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Previously, commit 102616b158 tried to
expose GOPROXY env var from developer environment by specifying it as
environment variable or build argument to container build process. But
using --build-arg to container build command would require the same to
be referenced inside Dockerfile/Containerfile so that the following
warning can be avoided:
[Warning] one or more build args were not consumed: [GOPROXY]
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
This bumps the Go version of the test container to 1.19.5, which is
the oldest supported Go version as of now.
The go.mod file (including an upgrade of its dependencies) and the
revive environment is bumped to 1.19 as well.
Signed-off-by: Sven Anderson <sven@redhat.com>
Use the GOARCH variable to define to proper version of go binaries to
download. Without this, the end result image can not run properly on
local host machine.
Signed-off-by: Shachar Sharon <ssharon@redhat.com>
Ceph version is already available from CEPH_VERSION env variable.
CEPH_TAG on the other hand is only defined to determine the tag for
ceph base image and seems to be unused in the resulting go-ceph
container image.
Signed-off-by: Anoop C S <anoopcs@cryptolab.net>
The Go 1.18 release was done in march 2022 and so we should
continue to use the older of the two supported releases which
is now Go 1.17.x.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
This issue should have been fixed in ceph-containers already but for
whatever reason we don't seem to be getting the fix.
For now, disable the apache-arrow-centos repo.
Based on: 8b4c8dc730
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Return to trying to install castxml into the container image when
possible. This now appears to work (and function) on the centos 8 based
images.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
With the recent release of go1.17 the oldest supported go version is
1.16. Update to the newest 1.16.z and clean up the structure of
the go install process a bit.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
First off, ceph is no longer using docker hub for the canonical image
hosting. We have also been recommended to use ceph/ceph instead of
ceph/daemon-base. Thus we switch to 'quay.io/ceph/ceph'.
However, ceph/ceph doesn't use codenames for tags so we also have to use
version number tags. These are set as build args now. To retain
compatibility with the way we were working previously we have add a
best-effort "assertion" to ensure the code name we provide matches with
the codename embedded in the base image.
While working on this I discovered that our previous build args were
probably not working as intended, but only by accident. See:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
This is why we now name our build arg GO_CEPH_VERSION and it is located
after the 'FROM' line.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
If the make variable USE_CACHE is set, the test container will mount
the /go directory from a named volume that is reused in following
test runs, so that go dependencies don't have to be installed again.
Signed-off-by: Sven Anderson <sven@redhat.com>
Sometimes the version in the RPM repository don't match the version
of the base image. This change updates the packages before installing
the libraries in order to make them match.
Signed-off-by: Sven Anderson <sven@redhat.com>
We're adding a fair amount of octopus (and later) only features now, and
at the time of this writing, pacific is already out. Stop defaulting to
nautilus and default to octopus instead.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The current version we are testing with is no longer supported.
This updates Go to a supported released version.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Bump the golang version to 1.14.7. Go 1.14 is now the older supported
version, with the recent release of Go.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The castxml install from centos8+epel has been rather fraught with
errors including broken/failed installs and, at the time of this commit,
it installs but can not run due to llvm integration/cli issues.
On top of this it was not available on centos7 anyway and so would,
in theory, only have worked on the octopus container image.
At this point it seem that running the implements tool is not worth
all the failures that trying to install castxml causes.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Now that ceph/daemon-base:latest-octopus is on v15.2.4 we can
stop the need to update already installed packages in the container.
To make things "safer" and fail early with errors rather than mysterious
compatibility problems, I am adding explicit versions to the
lib{cephfs,rados,rbd}-devel-* packages. Now if the container can't find
these packages it will fail rather than partially upgrade the ceph
packages.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
On octopus taking in the -devel packages causes *some* of the packages
to be updated but not all. This apparently makes ceph-fuse fail to
start correctly by blocking forever. This works around that issue.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Currently, this tool is only readily available on centos8 based
containers (octopus). But as it is being requested by binary name
it will be taken in if any repo used by the container makes it
available.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Ceph containers [1] are built by the ceph team and for our purposes,
only lack build tools and Go making the delta we need to do smaller.
1: https://github.com/ceph/ceph-container
Signed-off-by: John Mulligan <jmulligan@redhat.com>
