In preparation for running builds and tests in containers, make some
organizational changes to the run-make-*.sh scripts.
Original version of the patch can be found at https://github.com/ceph/ceph/pull/46071
Original-version-by: Ernesto Puerta <epuertat@redhat.com>
Co-authored-by: Ernesto Puerta <epuertat@redhat.com>
Co-authored-by: John Mulligan <jmulligan@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>
This was attempted in commit 69a7ed4eab ("run-make-check: enable
WITH_RBD_RWL when WITH_PMEM is true") but never completed. We soon
bumped the requirement on libpmem, so WITH_SYSTEM_PMDK=ON wouldn't
have worked anyway.
Enable the RWL mode conditionally based on WITH_RBD_RWL variable.
Enable the SSD mode unconditionally as it has no special dependencies
and can be built on any architecture.
Fixes: https://tracker.ceph.com/issues/55285
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Having a unique string like "CI_DEBUG" will help me know where we are in the build process in Jenkins logs.
Signed-off-by: David Galloway <dgallowa@redhat.com>
To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.
To connect to Motr cluster, add the following configuration
parameters to ceph.conf:
[client]
...
rgw backend store = motr
motr profile fid = 0x7000000000000001:0x4f
motr ha endpoint = inet:tcp:10.0.0.1@2001
...
[client.rgw.8000]
...
motr my endpoint = inet:tcp:10.0.0.1@5001
motr my fid = 0x7200000000000001:0x29
The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:
Profile:
0x7000000000000001:0x4f
Services:
centos8n1
[started] hax 0x7200000000000001:0x6 inet:tcp:10.0.0.1@5001
...
[unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001
Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.
Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
This commit is a try to balance crimson's debug facilities with overwhelming
some hosts the `make check` bot runs on with huge number of log entries;
see https://jenkins.ceph.com/job/ceph-pull-requests-arm64/17127/ for an
exemplification of the problem.
This is achieved by differentiating debug levels depending upon the
`FOR_MAKE_CHECK` env variable. It's a hack / a makeshift solution
only. IMHO ultimately we should introduce a dedicated target for
the bot while leaving `ninja test` for humans which would allow for more
debugs and maybe also for running `vstart.sh`-dependent tests.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
run-make-check.sh uses run-make.sh to `prepare`(install dependencies)
and `configure` cmake options, without quotes these options containing
special characters(hypens mostly) are skipped, hence we see not all
options supplied at cmake configure step.
Resolves(focused to solve issues in jenkins build env):
- missing cmake options:
cmake_opts+=" -DCMAKE_CXX_COMPILER=$cxx_compiler -DCMAKE_C_COMPILER=$c_compiler"
cmake_opts+=" -DCMAKE_CXX_FLAGS_DEBUG=\-Werror"
- Ninja not being used as cmake generator
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
should leave it to do_cmake.sh to decide which python3 version to use,
there is case that we have multiple python3 installed, but only one of them
is fully supported by the distro, in the sense that python3-* packages
are packaged for that python3.
Signed-off-by: Kefu Chai <kchai@redhat.com>
so we can at least build-test the rwl cache plugin in "make check" run.
this is an opt-in option which is only enabled if WITH_PMEM is "true",
we will set it in the corresponding ceph-build job.
Signed-off-by: Kefu Chai <kchai@redhat.com>
to enable XUnit plugin of jenkins to consume the ctest output and
publish it in the dashboard, we need to
* let ctest generate XML output instead of plain text output
* do not fail the test if any test case fails. this allows the publisher
to do its job by checking the XML output.
* prevent ctest from compressing the output. see
https://issues.jenkins.io/browse/JENKINS-21737
Signed-off-by: Kefu Chai <kchai@redhat.com>
without this change the seastar based tests fail on host with 48 cores,
because the /proc/sys/fs/aio-nr used by the tests is greater than
1048576. if run-make-check.sh is used to launch the test, the default
job number is `$(nproc) / 2`, and the peak number of /proc/sys/fs/aio-nr
when running ctest was 3190848 when testing on the 48-core host.
so we need to increase fs.aio-max-nr accordingly to the available cores
on the host.
Signed-off-by: Kefu Chai <kchai@redhat.com>
We 'set -e' but that is ignored because 'build tests' is executed in a
'&&' list (see 'man set') so move the echo to the following line.
Follow-up to 03ff2146f9
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
run-make-check.sh is used by ceph-build/ceph-pull-requests/build/build
to run "make check". this change prepares run-make-check.sh so that we
can enable ZBD build on demand once ceph-build is updated accordingly.
Signed-off-by: Kefu Chai <kchai@redhat.com>
With 5e9a1d95c9 ("run-make-check.sh: extract run-make.sh"), this log
message is actively confusing. At this point we just have the binaries,
no tests have run yet (and if -N is passed through CHECK_MAKEOPTS, none
will).
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
python version for python binding and ceph-mgr embedded python
interpreter anymore. as before switching to python3-only build,
we can build python3 and python2 python bindings, and ceph-mgr
can use either of them. but after switching to python3-only
build, ceph-mgr has to use whatever python version used to
build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
parameter anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
so we can reuse run-make.sh for building the artifact used by other
tests than "make check", for instance, dashboard's E2E test and
crimson's performance test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
but we still enable it in `run-make-check.sh`
* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency
Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>
Before this patch, run-make-check.sh was setting the cmake options for
py3 build based only on the absence of a python2 binary in the system.
This was not sufficiently deterministic, and it didn't help folks who
run do_cmake.sh directly.
With this patch, we explicitly force the py3 build in do_cmake.sh, for
those OS types/versions we know need it.
Fixes: https://tracker.ceph.com/issues/37330
Signed-off-by: Nathan Cutler <ncutler@suse.com>
ceph-libboost1.67 repo is setup to satisfy the libboost dependencies
used when building ceph. now it offers arm64 and amd64 builds for xenial.
* install-deps.sh: add ceph-libboost1.67 repo hosted on chacra if the
building host does not have any of the listed ceph-libboost1.67
packages installed
* run-make-check.sh: add "-DWITH_SYSTEM_BOOST=ON
-DBOOST_ROOT=/opt/ceph" if include/boost/config.hpp is found
under /opt/ceph
See-also: http://tracker.ceph.com/issues/25186
Signed-off-by: Kefu Chai <kchai@redhat.com>
The actual code is resetting the statistics before doing the actual compilation and prints them after the build.
That is nice to understand how much the cache was used but doesn't help understanding how much it _could_ have been used.
This patch is adding a reporting (-s) when cleaning the statistics so we can estimate :
- the actual number of files in cache
- the actual size of the cache
With this two missing information, its now possible estimate if there is some miss-usage of the cache.
Signed-off-by: Erwan Velu <erwan@redhat.com>
Since we source install-deps.sh, it replaces our trap with its own,
so we have to reinstate our trap after it finishes.
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Since run-make-check.sh already ensures that ccache is installed,
it makes sense to let everyone benefit from the ccache
tweaks introduced by 4cb5a59053
Note 1: The previous solution using "date" would cause build tools to reset
their timestamps after 24 hours, on subsequent runs of run-make-check.sh.
In order to maximize ccache effectiveness, this commit sets SOURCE_DATE_EPOCH
to a fixed value: the number of seconds elapsed since the Unix epoch as at
January 1, 2000 (chosen to commemorate Y2K armageddon).
Note 2: this commit introduces "set -e". This was actually in effect
before, via "source install-deps.sh". Better to make it explicit.
Fixes: http://tracker.ceph.com/issues/24777
Signed-off-by: Nathan Cutler <ncutler@suse.com>
When run-make-check is run by the CI, some tuning can be performed to
speedup the build.
This commit :
- Detect jenkins by searching JENKINS_HOME env variable
- Defines the SOURCE_DATE_EPOCH to enforce a stable date across builds
- Ask cmake not to use the git versioning which adds useless entropy for a temporary build (ENABLE_GIT_VERSION=OFF)
- Define the ccache slopiness to increase efficiency
- Increase the ccache size to save multiple builds to maximise cache hit between PRs
- Print ccache statistics to evaluate ccache efficiency
Signed-off-by: Erwan Velu <erwan@redhat.com>
