we use dh_python3 to define subvar of ${python3:Depends} as a part
of the runtime dependencies of python3 packages, like,
ceph-mgr modules named "ceph-mgr-*", python3 bindings named "python3-*".
but unlike python3 bindings of Ceph APIs, the ceph-mgr modules are
not packaged in a typical python way. in other words, they do not
ship a "dist-info" or an "egg-info" directory. instead, we just
install the python scripts into a directory which can be found by
ceph-mgr, by default it is /usr/share/ceph/mgr/dashboard/plugins.
this does not follow the convention of python packaging or
debian packaging policies related to python package. but it
still makes to put these files in this non-convention place, as
they are not supposed to be python packages consumed by the
outer world -- they are but plugins. and should always work
with the same version of ceph-mgr.
the problem is, despite that we have ${python3:Depends} in
the "Depends" field of packages like ceph-mgr-dashboard, dh_python3
is not able to figure out the dependencies by looking at the
installed files. for instance, we have following "Depends" of
ceph-mgr-dashboard:
Depends: ceph-mgr (= 17.0.0-12481-g805d2320-1focal), python3-cherrypy3, python3-jwt, python3-bcrypt, python3-werkzeug, python3-routes
and in the debian/control file we have:
Depends: ceph-mgr (= ${binary:Version}),
python3-cherrypy3,
python3-jwt,
python3-bcrypt,
python3-werkzeug,
python3-routes,
${misc:Depends},
${python:Depends},
${shlibs:Depends},
apparently, none of the subvar is materialized to
a non-empty string.
to improve the packaging, in this change:
* drop all subvars from ceph-mgr-*, as they
are all implemented in pure python.
* add debian/ceph-mgr-*.requires, it's content
is replicated with the corresponding requirements.txt
files.
* add python3-distutils for distutils, as debian
and its derivatives package non-essetial part of
distutils into a separate package, see
https://packages.debian.org/stable/python3-distutils
* add ${python3:Depends} so dh_python3
can extract the deps from debian/ceph-mgr-*.pydist
* update the rule for "override_dh_python3" target,
so dh_python3 can pick up the dependencies specified
in .requires file.
* remove the python3 dependencies not used by
ceph-mgr from ceph-mgr's "Depends"
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
${python:Depends} is added by dh_python2. but we've migrated to
python3 and Ceph is not compatible with python2 anymore. let's
replace all references of python2 with python3.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
we use ninja build as default build now, having it installed only with
make check enabled may make builds fail, if ran without make check.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
With the direct calls of nose methods removed, we no longer need to
BuildRequire it.
Fixes: https://tracker.ceph.com/issues/54252
Signed-off-by: Steve Kowalik <steven@wedontsleep.org>
this change addresses the regression introduced by
637dd7b404
dist-packages is a debian specific directory for holding 3rd party
python modules. so install the pure python packages into it.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
The trailing '3' was missed in one instance, ceph-mgr-cephadm, leading to:
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
ceph-mgr-cephadm : Depends: python3-cherrypy but it is not installable
Which makes the installation fail.
Fixes: 78983ad0d0
Signed-off-by: Koen Kooi <koen@softiron.com>
change the default value of WITH_SYSTEM_UTF8PROC from ON to OFF, so that
centos/rhel users can build with the default cmake configuration. no other
WITH_SYSTEM_* variable in ceph defaults to ON, so this is consistent
with other bundled libraries like boost and rocksdb
unfortunately, this also means that users that do have system packages
must opt-in to using them with -DWITH_SYSTEM_UTF8PROC=ON
both deb and rpm builds dependended on the previous default value, so
their logic was negated to match the new default
Fixes: https://tracker.ceph.com/issues/55114
Signed-off-by: Casey Bodley <cbodley@redhat.com>
adds an arrow submodule. when WITH_RADOSGW_SELECT_PARQUET is enabled,
the submodule is built as an external project and rgw links against its
imported Arrow::Parquet target
Signed-off-by: Casey Bodley <cbodley@redhat.com>
After https://github.com/ceph/ceph/pull/44059 the monitoring/prometheus
and monitoring/grafana/dashboards directories are changed to
monitoring/ceph-mixins. That broke the shared_folders in the cephadm
bootstrap script.
Changed all the instances of monitoring/prometheus and
monitoring/grafana/dashboards to monitoring/ceph-mixins
Also, renaming all the instances of prometheus_alerts.yaml to
prometheus_alerts.yml.
Fixes: https://tracker.ceph.com/issues/54176
Signed-off-by: Nizamudeen A <nia@redhat.com>
As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Mixin is a way to bundle dashboards, prometheus rules and alerts into
jsonnet package. Shifting to mixin will allow easier integration with
monitoring automation that some users may use.
This commit moves `/monitoring/grafana/dashboards` and
`/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts
was also converted to Jsonnet using an automated way (from yaml to json
to jsonnet). This commit minimises any change made to the generated files
and should not change neithers the dashboards nor the Prometheus alerts.
In the future some configuration will also be added to jsonnet to add
more functionalities to the dashboards or alerts (i.e.: multi cluster).
Fixes: https://tracker.ceph.com/issues/53374
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Add golang as a build dependency to build golang project in the test
for monitoring/ceph-mixin.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
migrate from using opentracing-cpp to opentelemetry-cpp static as distributed tracing API
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
we no longer need to package libjaeger, as we would be using
libopentelemetry static libraries(until we support shared).
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
Whereas new LevelDB packages are build with -fno-rtti and break our
attempts to compile against them,
and whereas LevelDB has been deprecated for some time, with the Ceph
team expending great effort to migrating OSDs off of it,
let it be therefore removed.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Creates an http endpoint in mgr/cephadm to receive
http requests and an agent that can be deployed on
each host that will gather metadata on the host and
send it to the mgr/cephadm http endpoint. Should save the
cephadm mgr module a lot of time it would have to spend
repeatedly ssh-ing into each host to gather the metadata
and help performance on larger clusters.
Fixes: https://tracker.ceph.com/issues/51004
Signed-off-by: Adam King <adking@redhat.com>
This reverts commit 707edc0b2d.
This commit was somewhat premature in removing the xmlstarlet
dependency.
Fixes: https://tracker.ceph.com/issues/52681
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
xmlstarlet is not used anymore while performing "make check", so drop it
from the build dependency list.
it is still used by the teuthology test though, so keep it as a
dependency of ceph-test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change makes util-linux, xfsprogs and e2fsprogs runtime deps of ceph-volume
ceph-volume uses blkid and lsblk, which are in turn packaged by
util-linux.
util-linux were added as build dependency to fulfill the needs of
ceph-disk. and we tested ceph-disk as part of "make check", since
ceph-disk was dropped, there is no need to have util-linux as
build dependency anymore.
the same applies to e2fsprogs and xfsprogs
see also a991778341
Signed-off-by: Kefu Chai <kchai@redhat.com>
was originally added by 6a45b8ed04 and
cfcaceac44 as a dependency of ceph-disk.
although, ceph-disk was replaced by ceph-volume, cryptsetup-bin is still
used.
but it should be a runtime dependency of ceph-volume. and it is not a
build dependency of ceph, so it's removed from the build dependency list
as well.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.
in this change
* ceph-volume is extracted out into a separated package
* ceph-volume depends on ceph-osd, as it deploys it and relies on
an already-installed ceph-osd in the system.
* ceph-osd recommends ceph-volume. as ceph-osd can be used as a
standalone package. but ceph-volume enhances it. also, to ensure
the existing users to get ceph-volume installed along with
ceph-osd.
Signed-off-by: Kefu Chai <kchai@redhat.com>
we don't run parted when building ceph, instead, it is a runtime
dependency of ceph-volume, which is packaged in ceph-osd.
so in this change
* parted is removed from build dependency list
* parted is added as a runtime depedency of ceph-osd.
Signed-off-by: Kefu Chai <kchai@redhat.com>
We wish to be able to scrape SMART and NVMe metrics from OSD and MON
nodes. For this we require / recommend smartmontools and nvme-cli
dependencies for both the ceph-osd and ceph-mon packages. However, the
sudoers file (which is required for invoking `smartctl` by user 'ceph')
was installed only in the ceph-osd package. Since different packages
cannot own the same file, and because we want to be able to scrape from
every daemon, we move the dependencies and the sudoers installation to
ceph-base. For generalization, we rename:
sudoers.d/ceph-osd-smartctl -> sudoers.d/ceph-smartctl
Fixes: https://tracker.ceph.com/issues/50657
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
A tool to test the effect (number of pgs, objects, bytes moved)
of a crushmap change. This is a wrapper around osdmaptool, hardly
relying on its --test-map-pgs-dump option to get the list of
changed pgs. Additionally it uses pg stats to calculate the
numbers of objects and bytes moved.
Signed-off-by: Mykola Golub <mykola.golub@clyso.com>
