build scripts: various cleanups and minor changes preparing for containerized build and test
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Copy the logic from run-make.sh into install-deps.sh so that we can later
remove it from run-make.sh. It helps prevent breakage when apt-get is
interrupted.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Instead of requiring other scripts to install packages independently,
teach install-deps.sh to install additional packages from the variable
INSTALL_EXTRA_PKGS. Now, other scripts should just set
INSTALL_EXTRA_PKGS and call install-deps.sh.
In particular, this fixes an issue installing packages in a clean (ex.
container) system that doesn't yet have repositories set up. Since this
task is performed by install-deps.sh already we avoid a chicken-and-egg
issue (or doing redundant work of setting up repos) in other scripts.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Previously, the main part (top level body) of the script started and
then some function definitions occurred and then the main part of the
script resumed after that. I, and others, find this confusing so this
change moves the function definitions to occur before the main body of
the install-deps.sh script.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
More specifically, if we get e.g. a 504 while attempting to download a
.list file, fail the build and log the status code.
Signed-off-by: Zack Cerza <zack@redhat.com>
The en_US.UTF-8 locale is not always going to present on all systems.
Specifically, setting en_US.UTF-8 causes numerous warnings when running
this script on a CentOS Stream 8 container. Rather than install
additional packages to suppress the warning, use the C.UTF-8 locale.
Additionally, try to update the comment explaining the line to something
based on the commit history rather than the confusing term "vulnerable".
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The majority of the install-deps.sh file was already using spaces to
indent, convert the remaining hard tabs to 8 spaces.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
The majority of the lines in this shell script uses spaces to indent,
but I *think* this modeline tells emacs to indent with tabs. In a very
near future change I'm going to make all indents use spaces. So we first
remove the emacs modeline that tells it to use the opposite.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
When run on focal and bionic, install-deps ends early
due to this extra debug message that was added to the
end of `ensure_decent_gcc_on_ubuntu`. The debug message
prints when the script is run in a jenkins environment.
When the script is not run in a jenkins environment, the
value returned there is "false" or "0", which acts as
an early return. This stops the script from completing.
We can remove this line, as `ensure_decent_gcc_on_ubuntu`
is only called for focal and bionic, and most of the jenkins
nodes are running jammy. Also, there is a debug message at the
beginning of the function that should suffice.
Fixes: https://tracker.ceph.com/issues/57466
Signed-off-by: Laura Flores <lflores@redhat.com>
install-deps.sh: ensure that pip ugrades to most recent version within virtualenv
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>
this is an intermediate solution before cotx-motr upstream builds the
packages for jammy and upload them to github.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Install libpmem and libpmemobj under focal ubuntu. the version of apt
list can meet the current requirements. libpmemobj require >=1.8.
Libpmem has no version requirements.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
this change reverts 17d2bc3707, before
we recreate a chacra repo hosting libpmem packages, we are not able
to query the repo from shaman or pull the dependencies from chacra.
in future, we should be able to get the libpmem dependencies from
offical ubuntu package repo and fedora, CentOS Stream and RHEL repos.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Having a unique string like "CI_DEBUG" will help me know where we are in the build process in Jenkins logs.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.
To connect to Motr cluster, add the following configuration
parameters to ceph.conf:
[client]
...
rgw backend store = motr
motr profile fid = 0x7000000000000001:0x4f
motr ha endpoint = inet:tcp:10.0.0.1@2001
...
[client.rgw.8000]
...
motr my endpoint = inet:tcp:10.0.0.1@5001
motr my fid = 0x7200000000000001:0x29
The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:
Profile:
0x7000000000000001:0x4f
Services:
centos8n1
[started] hax 0x7200000000000001:0x6 inet:tcp:10.0.0.1@5001
...
[unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001
Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.
Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
Upon re-executing `install-deps.sh` recently, I got this error message:
```
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
```
The issue was that pip was not up to date within the virtualenv that `install-deps.sh` sets up. Adding the line `python3 -m pip install --upgrade pip` right after the virtualenv is activated should help to ensure that pip is up to date in that context.
Signed-off-by: Laura Flores <lflores@redhat.com>
so we don't need to use virtualenv python package for creating a
virtualenv, the "venv" module in Python3 would suffice.
see also https://docs.python.org/3/library/venv.html
Signed-off-by: Kefu Chai <kchai@redhat.com>
