Move the JWT requirement to the test requirements file. Also remove JWT from ceph specification and debian build.
Signed-off-by: Daniel Persson <mailto.woden@gmail.com>
cephadm is a compressed zipapp, and dh3_python3 doesn't understand
this sort of binary file, so fails to produce the required python3
dependency. So specify this explicitly in debian/control
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
This means that debian/control matches changelog entries, and that the
Maintainer address is up to date.
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Unless there's a version requirement (which there isn't here),
packages should not declare a Build-Depends: or Depends: relationship
on essential packages. Policy link:
https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Rely on the packaging system to provide a suitable g++ of version 11
or greater, and removing the corresponding hard-coding from
debian/rules, since cmake will then find a suitable version. This
seems better than trying to hard-code a particular version in
debian/rules, and Debian package building tools like e.g. sbuild will
then do the right thing.
This enables Reef (v18.2.0) to build on Debian bookworm in a clean
chroot.
Fixes: https://tracker.ceph.com/issues/61845
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
This is the MVP for a driver for RGW that operates on top of a POSIX
filesystem. It supports get, put, list, copy, multipart, external
access via the filesystem itself, and ordered bucket listings via an
LRU-based cache.
Note that this is currently a Filter, indended to run on top of dbstore.
This is because it currently doesn't have any User implementation, so it
depends on dbstore's User. Everything else is implemented in
POSIXDriver. Once there is a User implementation, this will become a
Store, instead of a Filter.
Commit messages from bucket listing cache:
rgw/posixdriver: recycle lmdb database handles as required
While LMDB workflows often do not close/return database handles,
ours continually reuses them. This requires us to close each
handle (atomically) when a cache entry is recycled.
rgw/posixdriver: don't instantiate bucket cache entries from notify events
rgw/posixdriver: incorporate lmdb-safe for now
The current inclusion is based on https://github.com/Martchus/lmdb-safe,
which is actively maintained but currently has some packaging issues the
author has agreed to accept fixes for.
For now, skip the submodule to save time and remove an external dependency.
rgw/posixdriver: fix listing of cached, empty bucket
* check lmdb enumeration result in all cases and w/better style
* add unit test for enumeration of an empty cached directory
rgw/posixdriver: nest lmdbs in a directory under the dbroot path to avoid cleanup issues
rgw/posixdriver: refactor for posix integration
* Derive BucketCache types as templates on a SAL driver and SAL
bucket pair.
* Integrate cache fills as callbacks into SAL layer (or mock, for
tests)
* Renaming and cleanups
rgw/posixdriver: add bucket cache implementation and tests
Adds free-standing cache of buckets and object names, with
bucket names (and listing attributes, upcoming) managed in
a hashed set of lmdb databases, which provides ordering and
a high-performance listing cache.
An framework for notification on new object creation (e.g.,
outside S3 workflow) is provided, and a Linux implementation
using inotify.
FindLMDB.cmake taken with attribution and license.
rgw/posixdriver: add zpp_bits serialization (FAST)
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Signed-off-by: Ali Maredia <amaredia@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
The issue this is addressing is that if somebody has
docker-ce installed but not docker.io with the
current recommends it will install podman. If
the user has docker-ce we don't want it to do that.
Signed-off-by: Adam King <adking@redhat.com>
The current VDO support implementation is buried inside the common/blkdev.cc
with a simple interface used by KernelDevice. It is not easily extendable
and can not be easily used for other devices providing similar capabilities.
This patch adds a plugin system that is based in its structure on the
erasure code plugin system and moves the VDO support code into a VDO plugin.
Signed-off-by: Martin Ohmacht <mohmacht@us.ibm.com>
Without this, upgrades will keep older versions of the libcephsqlite3
module.
Fixes: https://tracker.ceph.com/issues/56274
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
In order to support the character device of pmem usage in bluestore via
libpmem built by Ceph itself, we need to enable daxctl and ndctl
dependency. add the installation of ndctl and find it. the version of
ndctl and daxctl library requires >63. "apt-get install" meet the version
under ubuntu focal.
the installation of ndctl-devel in ceph.spec.in has not been verified.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
Install libpmem and libpmemobj under focal ubuntu. the version of apt
list can meet the current requirements. libpmemobj require >=1.8.
Libpmem has no version requirements.
Signed-off-by: Yin Congmin <congmin.yin@intel.com>
we use dh_python3 to define subvar of ${python3:Depends} as a part
of the runtime dependencies of python3 packages, like,
ceph-mgr modules named "ceph-mgr-*", python3 bindings named "python3-*".
but unlike python3 bindings of Ceph APIs, the ceph-mgr modules are
not packaged in a typical python way. in other words, they do not
ship a "dist-info" or an "egg-info" directory. instead, we just
install the python scripts into a directory which can be found by
ceph-mgr, by default it is /usr/share/ceph/mgr/dashboard/plugins.
this does not follow the convention of python packaging or
debian packaging policies related to python package. but it
still makes to put these files in this non-convention place, as
they are not supposed to be python packages consumed by the
outer world -- they are but plugins. and should always work
with the same version of ceph-mgr.
the problem is, despite that we have ${python3:Depends} in
the "Depends" field of packages like ceph-mgr-dashboard, dh_python3
is not able to figure out the dependencies by looking at the
installed files. for instance, we have following "Depends" of
ceph-mgr-dashboard:
Depends: ceph-mgr (= 17.0.0-12481-g805d2320-1focal), python3-cherrypy3, python3-jwt, python3-bcrypt, python3-werkzeug, python3-routes
and in the debian/control file we have:
Depends: ceph-mgr (= ${binary:Version}),
python3-cherrypy3,
python3-jwt,
python3-bcrypt,
python3-werkzeug,
python3-routes,
${misc:Depends},
${python:Depends},
${shlibs:Depends},
apparently, none of the subvar is materialized to
a non-empty string.
to improve the packaging, in this change:
* drop all subvars from ceph-mgr-*, as they
are all implemented in pure python.
* add debian/ceph-mgr-*.requires, it's content
is replicated with the corresponding requirements.txt
files.
* add python3-distutils for distutils, as debian
and its derivatives package non-essetial part of
distutils into a separate package, see
https://packages.debian.org/stable/python3-distutils
* add ${python3:Depends} so dh_python3
can extract the deps from debian/ceph-mgr-*.pydist
* update the rule for "override_dh_python3" target,
so dh_python3 can pick up the dependencies specified
in .requires file.
* remove the python3 dependencies not used by
ceph-mgr from ceph-mgr's "Depends"
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
${python:Depends} is added by dh_python2. but we've migrated to
python3 and Ceph is not compatible with python2 anymore. let's
replace all references of python2 with python3.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
we use ninja build as default build now, having it installed only with
make check enabled may make builds fail, if ran without make check.
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
With the direct calls of nose methods removed, we no longer need to
BuildRequire it.
Fixes: https://tracker.ceph.com/issues/54252
Signed-off-by: Steve Kowalik <steven@wedontsleep.org>
The trailing '3' was missed in one instance, ceph-mgr-cephadm, leading to:
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
ceph-mgr-cephadm : Depends: python3-cherrypy but it is not installable
Which makes the installation fail.
Fixes: 78983ad0d0
Signed-off-by: Koen Kooi <koen@softiron.com>
adds an arrow submodule. when WITH_RADOSGW_SELECT_PARQUET is enabled,
the submodule is built as an external project and rgw links against its
imported Arrow::Parquet target
Signed-off-by: Casey Bodley <cbodley@redhat.com>
As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Add golang as a build dependency to build golang project in the test
for monitoring/ceph-mixin.
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
migrate from using opentracing-cpp to opentelemetry-cpp static as distributed tracing API
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
we no longer need to package libjaeger, as we would be using
libopentelemetry static libraries(until we support shared).
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
Whereas new LevelDB packages are build with -fno-rtti and break our
attempts to compile against them,
and whereas LevelDB has been deprecated for some time, with the Ceph
team expending great effort to migrating OSDs off of it,
let it be therefore removed.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Creates an http endpoint in mgr/cephadm to receive
http requests and an agent that can be deployed on
each host that will gather metadata on the host and
send it to the mgr/cephadm http endpoint. Should save the
cephadm mgr module a lot of time it would have to spend
repeatedly ssh-ing into each host to gather the metadata
and help performance on larger clusters.
Fixes: https://tracker.ceph.com/issues/51004
Signed-off-by: Adam King <adking@redhat.com>
This reverts commit 707edc0b2d.
This commit was somewhat premature in removing the xmlstarlet
dependency.
Fixes: https://tracker.ceph.com/issues/52681
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
xmlstarlet is not used anymore while performing "make check", so drop it
from the build dependency list.
it is still used by the teuthology test though, so keep it as a
dependency of ceph-test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change makes util-linux, xfsprogs and e2fsprogs runtime deps of ceph-volume
ceph-volume uses blkid and lsblk, which are in turn packaged by
util-linux.
util-linux were added as build dependency to fulfill the needs of
ceph-disk. and we tested ceph-disk as part of "make check", since
ceph-disk was dropped, there is no need to have util-linux as
build dependency anymore.
the same applies to e2fsprogs and xfsprogs
see also a991778341
Signed-off-by: Kefu Chai <kchai@redhat.com>
was originally added by 6a45b8ed04 and
cfcaceac44 as a dependency of ceph-disk.
although, ceph-disk was replaced by ceph-volume, cryptsetup-bin is still
used.
but it should be a runtime dependency of ceph-volume. and it is not a
build dependency of ceph, so it's removed from the build dependency list
as well.
Signed-off-by: Kefu Chai <kchai@redhat.com>
ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.
in this change
* ceph-volume is extracted out into a separated package
* ceph-volume depends on ceph-osd, as it deploys it and relies on
an already-installed ceph-osd in the system.
* ceph-osd recommends ceph-volume. as ceph-osd can be used as a
standalone package. but ceph-volume enhances it. also, to ensure
the existing users to get ceph-volume installed along with
ceph-osd.
Signed-off-by: Kefu Chai <kchai@redhat.com>
we don't run parted when building ceph, instead, it is a runtime
dependency of ceph-volume, which is packaged in ceph-osd.
so in this change
* parted is removed from build dependency list
* parted is added as a runtime depedency of ceph-osd.
Signed-off-by: Kefu Chai <kchai@redhat.com>
