When run on focal and bionic, install-deps ends early
due to this extra debug message that was added to the
end of `ensure_decent_gcc_on_ubuntu`. The debug message
prints when the script is run in a jenkins environment.
When the script is not run in a jenkins environment, the
value returned there is "false" or "0", which acts as
an early return. This stops the script from completing.
We can remove this line, as `ensure_decent_gcc_on_ubuntu`
is only called for focal and bionic, and most of the jenkins
nodes are running jammy. Also, there is a debug message at the
beginning of the function that should suffice.
Fixes: https://tracker.ceph.com/issues/57466
Signed-off-by: Laura Flores <lflores@redhat.com>
install-deps.sh: ensure that pip ugrades to most recent version within virtualenv
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
this is an intermediate solution before cotx-motr upstream builds the
packages for jammy and upload them to github.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Install libpmem and libpmemobj under focal ubuntu. the version of apt
list can meet the current requirements. libpmemobj require >=1.8.
Libpmem has no version requirements.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
this change reverts 17d2bc3707, before
we recreate a chacra repo hosting libpmem packages, we are not able
to query the repo from shaman or pull the dependencies from chacra.
in future, we should be able to get the libpmem dependencies from
offical ubuntu package repo and fedora, CentOS Stream and RHEL repos.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Having a unique string like "CI_DEBUG" will help me know where we are in the build process in Jenkins logs.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.
To connect to Motr cluster, add the following configuration
parameters to ceph.conf:
[client]
...
rgw backend store = motr
motr profile fid = 0x7000000000000001:0x4f
motr ha endpoint = inet:tcp:10.0.0.1@2001
...
[client.rgw.8000]
...
motr my endpoint = inet:tcp:10.0.0.1@5001
motr my fid = 0x7200000000000001:0x29
The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:
Profile:
0x7000000000000001:0x4f
Services:
centos8n1
[started] hax 0x7200000000000001:0x6 inet:tcp:10.0.0.1@5001
...
[unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001
Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.
Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
Upon re-executing `install-deps.sh` recently, I got this error message:
```
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
```
The issue was that pip was not up to date within the virtualenv that `install-deps.sh` sets up. Adding the line `python3 -m pip install --upgrade pip` right after the virtualenv is activated should help to ensure that pip is up to date in that context.
Signed-off-by: Laura Flores <lflores@redhat.com>
so we don't need to use virtualenv python package for creating a
virtualenv, the "venv" module in Python3 would suffice.
see also https://docs.python.org/3/library/venv.html
Signed-off-by: Kefu Chai <kchai@redhat.com>
instead of inventing our way for defining "make check" dependencies, use
build profile for adding "make check" specific dependencies. see
https://wiki.debian.org/BuildProfileSpec
Signed-off-by: Kefu Chai <kchai@redhat.com>
1) make reference to python3 indepedant of explicit path
2) add required py-yaml module to install list
fixes: #40731
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Using subscription-manager will fail in a container so use dnf
config-manager which should work on bare metal as well as in a
container.
Fixes: https://tracker.ceph.com/issues/50118
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
we install different versions of precompiled ceph-libboost packages
for different branches when building and testing them on ubuntu test
nodes. for instance,
- nautilus: v1.72
- octopus, pacific: v1.73
they share the same set of test nodes. and these ceph-libboost packages
conflict with each other, because they install files to the same places.
in order to avoid the confliction, we should uninstall existing packages
before installing a different version of ceph-libboost packages.
ceph-libboost${version}-dev is a package providing the shared headers of
boost library, so, in this change we check if it is installed before
returning or removing the existing packages.
Signed-off-by: Kefu Chai <kchai@redhat.com>
before this change, we use docker for running promtools offered by
a docker image, but this is not efficient, and quite a few developers
do not want to use docker for running "make check". this change was
introduced by #39246, the reason was that, in Ceph's CI process, we
are using Ubuntu/Bionic for running "make check" jobs, but prometheus
packaged by Bionic does not offer the "test rules" command. so, to
address problem, we are using "dnanexus/promtool:2.9.2" docker image
for verifying monitoring/prometheus/alerts/test_alerts.yml.
after this change, we use prometheus packaged by debian derivatives
instead of pulling a docker image.
* debian/control: add prometheus as a "make check" dependency
* install-deps.sh: partially revert
53a5816ded, as we don't need to
pull docker or start docker service for using promtool anymore.
* cmake: check if promtool is capable of running "test rules"
command, bail out if it is not.
see also: https://tracker.ceph.com/issues/49653
Signed-off-by: Kefu Chai <kchai@redhat.com>
