Having a unique string like "CI_DEBUG" will help me know where we are in the build process in Jenkins logs.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.
To connect to Motr cluster, add the following configuration
parameters to ceph.conf:
[client]
...
rgw backend store = motr
motr profile fid = 0x7000000000000001:0x4f
motr ha endpoint = inet:tcp:10.0.0.1@2001
...
[client.rgw.8000]
...
motr my endpoint = inet:tcp:10.0.0.1@5001
motr my fid = 0x7200000000000001:0x29
The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:
Profile:
0x7000000000000001:0x4f
Services:
centos8n1
[started] hax 0x7200000000000001:0x6 inet:tcp:10.0.0.1@5001
...
[unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001
Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.
Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
so we don't need to use virtualenv python package for creating a
virtualenv, the "venv" module in Python3 would suffice.
see also https://docs.python.org/3/library/venv.html
Signed-off-by: Kefu Chai <kchai@redhat.com>
instead of inventing our way for defining "make check" dependencies, use
build profile for adding "make check" specific dependencies. see
https://wiki.debian.org/BuildProfileSpec
Signed-off-by: Kefu Chai <kchai@redhat.com>
1) make reference to python3 indepedant of explicit path
2) add required py-yaml module to install list
fixes: #40731
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Using subscription-manager will fail in a container so use dnf
config-manager which should work on bare metal as well as in a
container.
Fixes: https://tracker.ceph.com/issues/50118
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
we install different versions of precompiled ceph-libboost packages
for different branches when building and testing them on ubuntu test
nodes. for instance,
- nautilus: v1.72
- octopus, pacific: v1.73
they share the same set of test nodes. and these ceph-libboost packages
conflict with each other, because they install files to the same places.
in order to avoid the confliction, we should uninstall existing packages
before installing a different version of ceph-libboost packages.
ceph-libboost${version}-dev is a package providing the shared headers of
boost library, so, in this change we check if it is installed before
returning or removing the existing packages.
Signed-off-by: Kefu Chai <kchai@redhat.com>
before this change, we use docker for running promtools offered by
a docker image, but this is not efficient, and quite a few developers
do not want to use docker for running "make check". this change was
introduced by #39246, the reason was that, in Ceph's CI process, we
are using Ubuntu/Bionic for running "make check" jobs, but prometheus
packaged by Bionic does not offer the "test rules" command. so, to
address problem, we are using "dnanexus/promtool:2.9.2" docker image
for verifying monitoring/prometheus/alerts/test_alerts.yml.
after this change, we use prometheus packaged by debian derivatives
instead of pulling a docker image.
* debian/control: add prometheus as a "make check" dependency
* install-deps.sh: partially revert
53a5816ded, as we don't need to
pull docker or start docker service for using promtool anymore.
* cmake: check if promtool is capable of running "test rules"
command, bail out if it is not.
see also: https://tracker.ceph.com/issues/49653
Signed-off-by: Kefu Chai <kchai@redhat.com>
we bump boost on regular basis. let's take the opportunity of moving to
focal to use boost v1.75.
v1.73 was used before this change. since both boost 1.75 and boost 1.73
install some files at the same places, we need to remove boost 1.73
before installing boost 1.75.
Signed-off-by: Kefu Chai <kchai@redhat.com>
WITH_ZBD is enabled for testing the build of zbd bluestore backend, and
we plan to migrate to Ubuntu/Focal for testing "make check", so need to
install libzbd when the distro version is focal.
Signed-off-by: Kefu Chai <kchai@redhat.com>
This PR intends to add unit testing for prometheus rules using promtool. To run the tests run 'run-promtool-unittests.sh' file.
Fixes: https://tracker.ceph.com/issues/45415
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
java packaging tools are now offered by "javapackages-tools" module in
el8. so let's re-enable it. this change partially reverts
dd133840b9
Signed-off-by: Kefu Chai <kchai@redhat.com>
this was an oversight in e7e3b86762
also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.
and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ubuntu disco and ubuntu focal do not ship libboost 1.72 and up, and
we depend on libboost 1.72 or up, so it does not help to install
liboost 1.67 or libboost 1.71 anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* if WITH_JAEGER flag is specified, install-deps should mangle and update
debian/control + ceph.spec the way we do for adding crimson dependencies
with WITH_SEASTAR flag.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
* This commit introduces Jaegertracing library as package libjaeger,
pickwhich would be consumed by other ceph pacakges such as ceph-common0
* adds the following dependencies, which would be build from source
using ExternalProjectHelper.cmake +IncludeJaeger.cmake +
Build<package>.cmake scripts:
jaegertracing: v0.6.0 [added as a submodule]
opentracing: v1.6.0 [added as a submodule]
thrift: 0.13.0 [added as a submodule]
yaml-cpp: 0.6.0
json(optional)
* updates Boost to be installed instead of being build only, because
jaegertracing them during their build process.
* ceph.spec.in: introduces a default enabled jaeger packaging option,
which could be disabled using --without-jaeger flag during rpmbuild
* note: libjaeger package if enabled will be a dependency on ceph-common, ceph-mon, rgw_common and transitively will be a dependency for modules that have them as a dependency.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
It looks like CentOS 8.3 will see all repos converted to lower case and
this has been pre-empted in the CentOS stream repos so we need to be
able to enable a repo called 'PowerTools' or 'powertools'
See https://git.centos.org/rpms/centos-repos/c/b759b17
Fixes: https://tracker.ceph.com/issues/48174
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
nasm support build isa-l:AVX512 algorithm implementation while yasm
doens't support it. Install nasm assembler to build isa-l
refer to: https://github.com/yasm/yasm/issues/101
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
just like WITH_SEASTAR, WITH_ZBD is passed to install-deps.sh as an env
var, and it's disabled by default, as libzbd-devel is not yet packaged
in any distro supported by this script. we packaged and uploaded it
to http://apt-mirror.front.sepia.ceph.com/lab-extras/8/ for build test
the libzbd bluestore backend. so "with_zbd" is always false on non-el8
distros. and should only be enabled for upstream "make check" test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
alpine build recipe is stale and does not work with the latest Ceph,
also the APKBUILD for Ceph can be found at alphine's aports repo, see
https://gitlab.alpinelinux.org/alpine/aports/-/tree/master/community/ceph.
so instead keeping a stale version, let's drop it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
