When running test-dedup-tool.sh on Ubuntu, the binary for ceph-dedup-daemon
is not installed, resulting in the failure of test_sample_dedup due to its
exclusion from ceph-test debian package.
To fix this issue, this commit includes the ceph-dedup-daemon in ceph-test debian
package.
Fixes: https://tracker.ceph.com/issues/66368
Signed-off-by: Sungmin Lee <sung_min.lee@samsung.com>
in 844b66de, we stopped using pkg_resources for import packaging.
and the exact reason why we introduced pkg_resources was for using
the packaging python module, see cf608920.
so, let's partially revert cf608920.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
in 110db72e, we added the rgw mgr module to ceph-mgr-modules-core
rpm package. but we didn't add this module to the corresponding
debian package.
rgw mgr module provides a simple interface to deploy RGW multisite
setup. so it would be nice to have it in ceph's debian packages as
well.
despite that rgw is not part of the core features, since this module
is already in ceph-mgr-modules-core rpm package, and it is relatively
small and does not pulling extra dependencies, let's added to the
debian packge with the same name. we can revisit this decision and
extract it out in a following up change if it is necessary in future.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
instead of relying on the internal implementation of pkg_resources,
let's import packaging directly.
in this change, we also add the dependency to the packaging module
in the packaging recipes.
See also https://github.com/pypa/setuptools/issues/4385
Fixes https://tracker.ceph.com/issues/66201
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
For this, the grafana dashboards are installed directly to the
containers and later, cephadm picks it up to mount the dashboards to the
grafana container
Signed-off-by: Nizamudeen A <nia@redhat.com>
debian/*.postinst: add adduser as a dependency and specify --home when adduser
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Adam King adking@redhat.com
It is hard for Debian/Ubuntu users to use ceph-exporter
because it is not included in any deb packages.
This commit adds a new deb package for ceph-exporter.
Fixes: https://tracker.ceph.com/issues/64095
Signed-off-by: Shinya Hayashi <shinya-hayashi@cybozu.co.jp>
--gecos option of adduser is deprecated in debian/bookworm, and
will be removed in debian/trixie,
see https://manpages.debian.org/bookworm/adduser/adduser.8.en.html.
so to be future-proof, let's switch to `usermod --comment`. please
note, since we still need to support ubuntu/jammy which is used in
our CI, and `adduser` shipped by ubuntu/jammy does not support
`--comment` yet, so we cannot use this option.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
for better readability, and to be more consistent with the rest
of this file, and other .postinst scripts of this project.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
quote from adduser/NEWS.Debian.gz:
> System user home defaults to /nonexistent if --home is not specified.
> Packages that call adduser to create system accounts should explicitly
> specify a location for /home (see Lintian check
> maintainer-script-lacks-home-in-adduser).
so let's follow this change in adduser. otherwise "cephadm"
would have a $HOME at `/nonexistent`.
Fixes: https://tracker.ceph.com/issues/64069
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
now that adduser allows us to set its home directory, we can do
this using adduser instead of using usermod. this change also
silences the warning from lintian
"maintainer-script-lacks-home-in-adduser". lintian complains if
`adduser --system` is called without passing `--home` option.
also, take this opportunity to s/-c/--comment/ in the command line
of `usermod`, for better readability.
Fixes: https://tracker.ceph.com/issues/64069
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
in `debian/ceph-common.postinst` and `debian/cephadm.postinst`, we
use `adduser --system` to create the system user when configuring
the corresponding package.
before this change, the dependency is not listed in the runtime
`Depends` section of ceph-common and cephadm.
in this change, the dependency is added. this is also suggested
by Securing Debian Manual, see
https://www.debian.org/doc/manuals/securing-debian-manual/bpp-lower-privs.en.html
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
Move the JWT requirement to the test requirements file. Also remove JWT from ceph specification and debian build.
Signed-off-by: Daniel Persson <mailto.woden@gmail.com>
The `ceph-mgr` package lost its dependency on `python3-bcrypt` when
the dependencies got moved from d/control to a dh_python3 compatible
requires file. Add it again as the bcrypt module is still used there.
Otherwise one gets errors when, e.g., calling `ceph -s` after a fresh
installation:
> 13 mgr modules have failed dependencies
> Module 'balancer' has failed dependency: No module named 'bcrypt'
> Module 'crash' has failed dependency: No module named 'bcrypt'
> Module 'devicehealth' has failed dependency: No module named 'bcrypt'
> Module 'iostat' has failed dependency: No module named 'bcrypt'
> Module 'nfs' has failed dependency: No module named 'bcrypt'
> Module 'orchestrator' has failed dependency: No module named 'bcrypt'
> Module 'pg_autoscaler' has failed dependency: No module named 'bcrypt'
> Module 'progress' has failed dependency: No module named 'bcrypt'
> Module 'rbd_support' has failed dependency: No module named 'bcrypt'
> Module 'restful' has failed dependency: No module named 'bcrypt'
> Module 'status' has failed dependency: No module named 'bcrypt'
> Module 'telemetry' has failed dependency: No module named 'bcrypt'
> Module 'volumes' has failed dependency: No module named 'bcrypt'
Fixes: https://tracker.ceph.com/issues/63637
Fixes: ef19547e83 ("debian: add .requires for specifying python3 deps")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up
that these packages contain python scripts and adds a necessary
python3 dependency. In the case of ceph-volume it additionally parses
the requirements.txt file.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Installation of init scripts properly belongs with dh_installinit, so
move the installation there.
That means we no longer need the override of dh_auto_build, which
simplifies the rules file.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
debian/ceph-base.docs only referred to a README that doesn't exist, so
remove it. Because dpkg-source doesn't reflect deletions from debian/
cf the orig.tar.gz, also remove the file in dh_auto_clean.
Then do away with the removal of the empty override of dh_installdocs;
the main benefit of which here is that debian/copyright gets installed
in all of the built packages, which otherwise lack a copyright
file.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
This means that debian/control matches changelog entries, and that the
Maintainer address is up to date.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Bring the dh compat level to 12, the most recent supported by the
oldest supported Ubuntu LTS release, 20.04. This necessitates changes
to how initscripts & systemd packaging are done.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Update the header paragraph to link to the canonical URL for the
format, and point to dev@ceph.io as the Contact.
Also add License: stanzas to reflect the licences in use (and refer to
fuller versions in /usr/share/common-licenses/ as appropriate).
This means that packages containing this copyright file are better in
compliance with the licences concerned.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:
https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
These were previously missing. The requirement for interpreters is in
Debian policy section 10.4:
https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts
Debian's packaging already adds the #! to these two postinsts. In
practice, a text executible without a #! line will likely be executed
by the calling shell, so a lot of the time we'd get away with it
unless the administrator is using an incompatible shell like tcsh.
This behaviour of shells is documented in POSIX section 1(e)(i)(b)
here:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.
This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.
Fixes: https://tracker.ceph.com/issues/61845
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
This is the MVP for a driver for RGW that operates on top of a POSIX
filesystem. It supports get, put, list, copy, multipart, external
access via the filesystem itself, and ordered bucket listings via an
LRU-based cache.
Note that this is currently a Filter, indended to run on top of dbstore.
This is because it currently doesn't have any User implementation, so it
depends on dbstore's User. Everything else is implemented in
POSIXDriver. Once there is a User implementation, this will become a
Store, instead of a Filter.
Commit messages from bucket listing cache:
rgw/posixdriver: recycle lmdb database handles as required
While LMDB workflows often do not close/return database handles,
ours continually reuses them. This requires us to close each
handle (atomically) when a cache entry is recycled.
rgw/posixdriver: don't instantiate bucket cache entries from notify events
rgw/posixdriver: incorporate lmdb-safe for now
The current inclusion is based on https://github.com/Martchus/lmdb-safe,
which is actively maintained but currently has some packaging issues the
author has agreed to accept fixes for.
For now, skip the submodule to save time and remove an external dependency.
rgw/posixdriver: fix listing of cached, empty bucket
* check lmdb enumeration result in all cases and w/better style
* add unit test for enumeration of an empty cached directory
rgw/posixdriver: nest lmdbs in a directory under the dbroot path to avoid cleanup issues
rgw/posixdriver: refactor for posix integration
* Derive BucketCache types as templates on a SAL driver and SAL
bucket pair.
* Integrate cache fills as callbacks into SAL layer (or mock, for
tests)
* Renaming and cleanups
rgw/posixdriver: add bucket cache implementation and tests
Adds free-standing cache of buckets and object names, with
bucket names (and listing attributes, upcoming) managed in
a hashed set of lmdb databases, which provides ordering and
a high-performance listing cache.
An framework for notification on new object creation (e.g.,
outside S3 workflow) is provided, and a Linux implementation
using inotify.
FindLMDB.cmake taken with attribution and license.
rgw/posixdriver: add zpp_bits serialization (FAST)
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
This enhances the script to both process versioned buckets correctly
and to handle object names that begin with underscore.
If the bucket is versioned it submits each version chronologically
(based on mtime) to be reindexed in order to "replay" the modification
of objects. However mtime is not a perfect indicator. So additionally
it looks at the OLH object to determine the most recent version and
the script makes sure that it is replayed last. The order of previous
versions is likely correct, but not guaranteed to be so.
Additional logic is added to handle objects with names that begin with
underscore ('_') since that's used as a delimiter and needs to be
escaped and rados object locators are also used.
A man page for the script is added.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
The issue this is addressing is that if somebody has
docker-ce installed but not docker.io with the
current recommends it will install podman. If
the user has docker-ce we don't want it to do that.
Signed-off-by: Adam King <adking@redhat.com>
Update ceph.spec.in and debian install files so
rgw-restore-bucket-index, rgw-orphan-list, rgw-gap-list,
rgw-gap-list-comparator are installed with common files.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Adds an experimental script that allows a bucket index of a
non-versioned bucket to be restored by applying `radosgw-admin object
reindex ...` to all objects in the specified bucket. The objects in
the bucket are determined by scanning the data pool for head objects
containing the bucket's marker.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
pytest isn't listed in ceph.spec.in for rpm builds, so it probably isn't
really a requirement for debian
Signed-off-by: Casey Bodley <cbodley@redhat.com>
instead of using the top-level "packaging" module, use the one
vendored by setuptools.
packaging python module provides versioning defined by PEP-440.
but python3-packaging is provided by CentOS8 powertools repo,
which is not enabled by default. and in CentOS9, this package
is provided by AppStream instead of BaseOS.
as prometheus mgr module is included by ceph-mgr-module-core,
it would be desirable if our user can install ceph-mgr-module-core
without enabling powertools or AppStream repo on a CentOS or
its derivative distros.
fortunately, setuptools vendors packaging module. and both
CentOS8 and CentOS9 provide python3-setuptools in their BaseOS
repos.
in this change, instead of using "packging" module, we use the
venderored one, which is in turn embedded in pkg_resources.
this python module is provided by python3-setuptools on CentOS
distros, and python3-pkg-resources on Debian and its derivatives.
the packaging recipes are updated accordingly to reflect the
new runtime dependency.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
