This enhances the script to both process versioned buckets correctly
and to handle object names that begin with underscore.
If the bucket is versioned it submits each version chronologically
(based on mtime) to be reindexed in order to "replay" the modification
of objects. However mtime is not a perfect indicator. So additionally
it looks at the OLH object to determine the most recent version and
the script makes sure that it is replayed last. The order of previous
versions is likely correct, but not guaranteed to be so.
Additional logic is added to handle objects with names that begin with
underscore ('_') since that's used as a delimiter and needs to be
escaped and rados object locators are also used.
A man page for the script is added.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
The issue this is addressing is that if somebody has
docker-ce installed but not docker.io with the
current recommends it will install podman. If
the user has docker-ce we don't want it to do that.
Signed-off-by: Adam King <adking@redhat.com>
Update ceph.spec.in and debian install files so
rgw-restore-bucket-index, rgw-orphan-list, rgw-gap-list,
rgw-gap-list-comparator are installed with common files.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Adds an experimental script that allows a bucket index of a
non-versioned bucket to be restored by applying `radosgw-admin object
reindex ...` to all objects in the specified bucket. The objects in
the bucket are determined by scanning the data pool for head objects
containing the bucket's marker.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
pytest isn't listed in ceph.spec.in for rpm builds, so it probably isn't
really a requirement for debian
Signed-off-by: Casey Bodley <cbodley@redhat.com>
instead of using the top-level "packaging" module, use the one
vendored by setuptools.
packaging python module provides versioning defined by PEP-440.
but python3-packaging is provided by CentOS8 powertools repo,
which is not enabled by default. and in CentOS9, this package
is provided by AppStream instead of BaseOS.
as prometheus mgr module is included by ceph-mgr-module-core,
it would be desirable if our user can install ceph-mgr-module-core
without enabling powertools or AppStream repo on a CentOS or
its derivative distros.
fortunately, setuptools vendors packaging module. and both
CentOS8 and CentOS9 provide python3-setuptools in their BaseOS
repos.
in this change, instead of using "packging" module, we use the
venderored one, which is in turn embedded in pkg_resources.
this python module is provided by python3-setuptools on CentOS
distros, and python3-pkg-resources on Debian and its derivatives.
the packaging recipes are updated accordingly to reflect the
new runtime dependency.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
rgw-policy-check - a program to do syntax checking on bucket policy.
This program just reads the policy into memory, so it is not
checking anything except syntax.
Signed-off-by: Marcus Watts <mwatts@redhat.com>
rgw: Fix return value of `rgw-policy-check`
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
rgw: Use ceph initialization in `rgw-policy-check`
Specifically so we can pull in the options from `ceph.conf` and similar.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
The current VDO support implementation is buried inside the common/blkdev.cc
with a simple interface used by KernelDevice. It is not easily extendable
and can not be easily used for other devices providing similar capabilities.
This patch adds a plugin system that is based in its structure on the
erasure code plugin system and moves the VDO support code into a VDO plugin.
Signed-off-by: Martin Ohmacht <mohmacht@us.ibm.com>
in ubuntu 22.04 and debian unstable, the layout (scheme) for system
python module is named "deb_system", the default one is 'posix_local'.
and 'posix_local' installs python modules into paths like
usr/local/lib/python3.10/dist-packages/. hence dh_install fails
when it tries to find the files to be packaged under directory of
usr/lib/python3*/site-packages/.
in this change, the "deb_system" scheme is used if it is available,
and fall back to "posix_prefix" to be backward compatible with older
debian (derivative) distros.
also, update the source directories of pure python's installation
from `site-packages` to `*-packages`, to be compatible with ubuntu focal
and ubuntu jammy. as we are now using the specified scheme instead of
the default one.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Without this, upgrades will keep older versions of the libcephsqlite3
module.
Fixes: https://tracker.ceph.com/issues/56274
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
In order to support the character device of pmem usage in bluestore via
libpmem built by Ceph itself, we need to enable daxctl and ndctl
dependency. add the installation of ndctl and find it. the version of
ndctl and daxctl library requires >63. "apt-get install" meet the version
under ubuntu focal.
the installation of ndctl-devel in ceph.spec.in has not been verified.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
Install libpmem and libpmemobj under focal ubuntu. the version of apt
list can meet the current requirements. libpmemobj require >=1.8.
Libpmem has no version requirements.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
we use dh_python3 to define subvar of ${python3:Depends} as a part
of the runtime dependencies of python3 packages, like,
ceph-mgr modules named "ceph-mgr-*", python3 bindings named "python3-*".
but unlike python3 bindings of Ceph APIs, the ceph-mgr modules are
not packaged in a typical python way. in other words, they do not
ship a "dist-info" or an "egg-info" directory. instead, we just
install the python scripts into a directory which can be found by
ceph-mgr, by default it is /usr/share/ceph/mgr/dashboard/plugins.
this does not follow the convention of python packaging or
debian packaging policies related to python package. but it
still makes to put these files in this non-convention place, as
they are not supposed to be python packages consumed by the
outer world -- they are but plugins. and should always work
with the same version of ceph-mgr.
the problem is, despite that we have ${python3:Depends} in
the "Depends" field of packages like ceph-mgr-dashboard, dh_python3
is not able to figure out the dependencies by looking at the
installed files. for instance, we have following "Depends" of
ceph-mgr-dashboard:
Depends: ceph-mgr (= 17.0.0-12481-g805d2320-1focal), python3-cherrypy3, python3-jwt, python3-bcrypt, python3-werkzeug, python3-routes
and in the debian/control file we have:
Depends: ceph-mgr (= ${binary:Version}),
python3-cherrypy3,
python3-jwt,
python3-bcrypt,
python3-werkzeug,
python3-routes,
${misc:Depends},
${python:Depends},
${shlibs:Depends},
apparently, none of the subvar is materialized to
a non-empty string.
to improve the packaging, in this change:
* drop all subvars from ceph-mgr-*, as they
are all implemented in pure python.
* add debian/ceph-mgr-*.requires, it's content
is replicated with the corresponding requirements.txt
files.
* add python3-distutils for distutils, as debian
and its derivatives package non-essetial part of
distutils into a separate package, see
https://packages.debian.org/stable/python3-distutils
* add ${python3:Depends} so dh_python3
can extract the deps from debian/ceph-mgr-*.pydist
* update the rule for "override_dh_python3" target,
so dh_python3 can pick up the dependencies specified
in .requires file.
* remove the python3 dependencies not used by
ceph-mgr from ceph-mgr's "Depends"
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
${python:Depends} is added by dh_python2. but we've migrated to
python3 and Ceph is not compatible with python2 anymore. let's
replace all references of python2 with python3.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
we use ninja build as default build now, having it installed only with
make check enabled may make builds fail, if ran without make check.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
With the direct calls of nose methods removed, we no longer need to
BuildRequire it.
Fixes: https://tracker.ceph.com/issues/54252
Signed-off-by: Steve Kowalik <steven@wedontsleep.org>
this change addresses the regression introduced by
637dd7b404
dist-packages is a debian specific directory for holding 3rd party
python modules. so install the pure python packages into it.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
The trailing '3' was missed in one instance, ceph-mgr-cephadm, leading to:
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
ceph-mgr-cephadm : Depends: python3-cherrypy but it is not installable
Which makes the installation fail.
Fixes: 78983ad0d0
Signed-off-by: Koen Kooi <koen@softiron.com>
change the default value of WITH_SYSTEM_UTF8PROC from ON to OFF, so that
centos/rhel users can build with the default cmake configuration. no other
WITH_SYSTEM_* variable in ceph defaults to ON, so this is consistent
with other bundled libraries like boost and rocksdb
unfortunately, this also means that users that do have system packages
must opt-in to using them with -DWITH_SYSTEM_UTF8PROC=ON
both deb and rpm builds dependended on the previous default value, so
their logic was negated to match the new default
Fixes: https://tracker.ceph.com/issues/55114
Signed-off-by: Casey Bodley <cbodley@redhat.com>
adds an arrow submodule. when WITH_RADOSGW_SELECT_PARQUET is enabled,
the submodule is built as an external project and rgw links against its
imported Arrow::Parquet target
Signed-off-by: Casey Bodley <cbodley@redhat.com>
After https://github.com/ceph/ceph/pull/44059 the monitoring/prometheus
and monitoring/grafana/dashboards directories are changed to
monitoring/ceph-mixins. That broke the shared_folders in the cephadm
bootstrap script.
Changed all the instances of monitoring/prometheus and
monitoring/grafana/dashboards to monitoring/ceph-mixins
Also, renaming all the instances of prometheus_alerts.yaml to
prometheus_alerts.yml.
Fixes: https://tracker.ceph.com/issues/54176
Signed-off-by: Nizamudeen A <nia@redhat.com>
As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Mixin is a way to bundle dashboards, prometheus rules and alerts into
jsonnet package. Shifting to mixin will allow easier integration with
monitoring automation that some users may use.
This commit moves `/monitoring/grafana/dashboards` and
`/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts
was also converted to Jsonnet using an automated way (from yaml to json
to jsonnet). This commit minimises any change made to the generated files
and should not change neithers the dashboards nor the Prometheus alerts.
In the future some configuration will also be added to jsonnet to add
more functionalities to the dashboards or alerts (i.e.: multi cluster).
Fixes: https://tracker.ceph.com/issues/53374
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Add golang as a build dependency to build golang project in the test
for monitoring/ceph-mixin.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
