mgr/cephadm: use jinja2 to render templates
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
The librbd parent cache is an initial testbed for librbd plugin
support. The shared library will be packaged with librbd for the
time being.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
* refs/pull/34596/head:
test: add a new program for testing ino_release_cb
client: add a new inode release request callback
client: expose ceph_ll_register_callbacks via libcephfs
client: move callback typedefs and arg struct into ceph_ll_client.h
client: rename ceph_statx.h to ceph_ll_client.h
client: make client_dentry_callback_t more friendly for C
client: only override umask_cb with non-NULL values
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Adds a `radosgw-admin` subcommand and walks the associated bucket
indices and manifests to generate the list of rados objects that
represent the rgw objects in the bucket(s).
Also adds a tool named `rgw-orphan-list`, which uses the radoslist
subcommand, that produces a list in a local file of what appear to be
rgw orphans.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
...so we can use it to include other definitions and types that need to
be shared with in-tree code that doesn't want to include libcephfs.h.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
in 589626464d, the core modules are moved
into ceph-mgr-modules-core. but the upgrade path from older ceph-mgr was
broken by this change.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Remove the files from ceph-mgr that are added to ceph-mgr-modules-core.
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout:dpkg: error processing archive /tmp/apt-dpkg-install-YpwLHV/044-ceph-mgr_15.1.0-1037-ga6b324b-1bionic_amd64.deb (--unpack):
2020-02-21T17:59:56.031 INFO:teuthology.orchestra.run.smithi060.stdout: trying to overwrite '/usr/share/ceph/mgr/alerts/__init__.py', which is also in package ceph-mgr-modules-core 15.1.0-1037-ga6b324b-1bionic
Broken by 589626464d
Signed-off-by: Sage Weil <sage@redhat.com>
ceph-mgr depends on it in the sense that these plugins are
a hard dependency of ceph-mgr. like cmake depends on cmake-data,
even if the later cannot function on its own without the former.
the reason why we need a separated package is that mgr modules
are written in python, and should be arch independent. this change
helps the downstream to understand this and help to minimize the
divergence between upstream and downstream, because distros like
debian and its derivatives encourage splitting architecture independent
data into its own package.
see
https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#architecture-independent-data
also, the python related runtime dependencies are also moved into
ceph-mgr-modules-core.
if a python dependency is shared by another ceph-mgr-* package and
ceph-mgr-modules-core. it's only added to ceph-mgr-modules-core for
simplicity and consistency. as the name of dependency might vary
on different distros, so it's tedious and error-prone to repeat
these conditons and checks.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Majority of radosgw is contained in libradosgw.so. (/usr)/bin/radosgw
is now a few lines that calls radosgw_Main() in libradosgw.so.
The "zipper" work to modularize storage back-ends that will require
linking to the shared library to resolve the methods that they reference.
Putting the bulk of the implementation in a shared lib also allows for
unit testing to link with the shared lib for testing.
radosgw_Main() is the C++ implementation. For C linkage, radosgw_main()
is provided.
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
python version for python binding and ceph-mgr embedded python
interpreter anymore. as before switching to python3-only build,
we can build python3 and python2 python bindings, and ceph-mgr
can use either of them. but after switching to python3-only
build, ceph-mgr has to use whatever python version used to
build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
parameter anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
cephadm mounts `/run/lvm` into OSD containers, which only exists,
if `lvm2` is installed on the container host. Otherwise we cannot
start ceph-volume or osd containers.
Fixes: https://tracker.ceph.com/issues/43368
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Moving ceph-daemon into src/ceph-daemon/ makes it simpler to add extra
code (eg. tox.ini, README, unittests, ...) specific to ceph-daemon.
That way related files are in a single directory.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
* refs/pull/30738/head:
mgr/alerts: raise health alert if smtplib has a problem
mgr/alerts: simple module to send health alerts
Reviewed-by: Tim Serong <tserong@suse.com>
Initialy SMTP support is implemented; nothing else.
This is just smart enough for me to get emails from my home cluster when
something goes wrong. No bells and whistled at this point.
Signed-off-by: Sage Weil <sage@redhat.com>
as a C++ project, we should depends on g++. actually, GCC-7 is able to
support C++17. but we use C++17 features extensively, and have been
running into GCC bugs every now and then. so would be ideal if we can
use newer GCC if possible.
we've dropped the support of ubuntu/xenial. while on ubuntu/bionic, only
GCC-7.4 is available. so let's require g++ (>= 7) then.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
When a secret and/or the mon addrs are not specified by the admin,
then mmap a MAP_SHARED buffer and spawn a child process to get that
info. For safety reasons, the child drops all capabilities other than
CAP_DAC_READ_SEARCH (to ensure that it'll be able to read the keyring,
should one be found). To achieve this, we add a new dependency on
libcap-ng.
Add a new C++ file with a single routine that will create a CephContext,
get a list of monitor addresses and scrape the keyring for a secret for
the specified cephx user.
If that info is found, then it is copied to fixed-length buffers in the
MAP_SHARED area and the child exits successfully.
The parent will then vet the returned info and copy it into the
appropriate fields if they are currently blank.
Fixes: https://tracker.ceph.com/issues/16656
Signed-off-by: Jeff Layton <jlayton@redhat.com>
Adds a mgr module to provide integration between
Ceph and the kubernetes events API within the rook-ceph
namespace. It provides several commands to view event
status including;
ceph k8sevents list .... show all k8s related events
ceph k8sevents ceph .... show events generated from this
module
ceph k8sevents status .. show status of the tracker threads
and counts of tracked events
Events sent to kubernetes are sourced from the
a) clog: ceph healtchecks and admin commands
b) explicit checks: hosts, pools and OSD states are checked
every 'n' seconds
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.
Signed-off-by: Kefu Chai <kchai@redhat.com>
but we still enable it in `run-make-check.sh`
* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency
Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>
the dashboard requires python-routes via cherrypy/_cpdispatch.py during runtime
but the cherrypy debian package only recommends it and doesn't depend on it
Fixes: https://tracker.ceph.com/issues/24420
Signed-off-by: Paul Emmerich <paul.emmerich@croit.io>
what we need is "libnl-genl-3-dev", libnl-3-dev should be a dependency
of "libnl-genl-3-dev". the only reason we would have added it, is if
the package maintainer of "libnl-genl-3-dev" fails to do so. and
the packages of "libnl-genl-3-dev" on both debian and ubuntu just look
correct in this perspective.
so let's remove this.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* debian/: remove ceph-crimson-osd package.
* debian/control: set `CEPH_OSD_BASENAME` env variable, which
will be consumed by `ceph-osd.install`. alternatively, we could
rename crimson-osd to ceph-osd in `override_dh_auto_install`,
but let's go with this way at this moment, unless `mv` in
`override_dh_auto_install` is proved to be better.
* ceph-osd.install: replace ceph-osd with crimson-osd if
`CEPH_EXTRA_CMAKE_ARGS` has `WITH_SEASTAR` in it. this only
happens when we are packaging the "crimson" flavor packages from
jenkins.
* ceph-osd.install: `chmod +x` this file, as we need to use
`/usr/bin/dh-exec` as the interpreter of it to perform variable
substitution and install.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/22446/head:
relicense LGPL-2.1 code as LGPL-2.1 or LGPL-3.0
COPYING: add reference to a few Apache-2.0 licensed source files
debian/copyright: sync with COPYING
COPYING: LGPL2.1 -> LGPL-2.1
COPYING: extra bits from debian/copyright
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.
The "patch" command is used by the BuildBoost cmake module.
Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>
this package is commented out using `# Crimson`. and it will be enabled
only if we are building "crimson" flavor builds.
this package depends on ceph-osd, because it'll basically be a drop-in
replacement of ceph-osd. and it shares all the dependencies of ceph-osd.
so let's just reuse ceph-osd package for now.
Signed-off-by: Kefu Chai <kchai@redhat.com>
The next patch adds netlink support to rbd-nbd. This patch just adds the
build related changes to bring in the netlink lib and test and build
against it.
Signed-off-by: Mike Christie <mchristi@redhat.com>
Instead of self-written validation methods to validate IPv4 and IPv6 addresses.
Use Python's standard library functions `ipaddress`.
Signed-off-by: Ashish Singh <assingh@redhat.com>
The primary motivation to relicense is a desire to integrate with projects
that are licensed under the Apache License version 2.0. Although opinions
vary, there are some who argue the the LGPL-2.1 and Apache-2.0 licenses
are not fully compatible. We would like to avoid the ambiguity and
potential for controversy.
Projects we would like to consume that are Apache-2.0 licensed include
Seastar, OpenSSL (which is in the process of relicensing to Apache-2.0),
and Swagger (swagger.io). Note that some of these are dynamically linked
or consumed via a high-level language and may or may not require a change
to LGPL-3.0, but providing the option for LGPL-3.0 certainly avoids any
uncertainty.
A few other source files are already incorporated into Ceph that claim an
Apache-2.0 license:
src/common/deleter.h
src/common/sstring.h
src/include/cpp-btree
The Ceph developers would further like to provide a license option that is
more modern than the current LGPL-2.1. LGPL-3.0 includes updated,
clarified language around several issues and is widely considered
more modern, superior license.
Signed-off-by: Sage Weil <sage@redhat.com>
ceph-common depends on librados2, but ceph-immutable-object-cache only
uses librados2's C++ API, so librados2 would suffice.
rpmbuild is able to figure out the .so dependencies, and let
ceph-immutable-object-cache depend on the package which offers
librados2, but we'd better make sure immutable-object-cache is linked
against the librados2 of the same version, to ensure that librados2
offers the correct C++ symbols. as we only version the C++ API in the
librados2 after nautilus.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/26237/head:
Revert "qa: update test_envlibrados_for_rocksdb.sh for libradospp split"
doc/librados: explicitly mention that the C++ API is not stable
ceph.spec: force use of upgrade devtoolset-gcc under RHEL 7
librados: add symbol versioning to the C++ API
librados: add symbol versioning to the C API
librados: revert librados3/libradoscc back to librados2
Reviewed-by: Kefu Chai <kchai@redhat.com>
The future goal would be to change the version for each Ceph major
release to ensure C++ applications will need to be recompiled against
the librados C++ API since we don't guarentee ABI stability.
Fixes: https://tracker.ceph.com/issues/38177
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
* refs/pull/26526/head:
rpm,deb: remove btrfs-tools as a build-dependency
rpm,deb: remove bc as a build-dependency
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
For backwards compatibility and upgrade reasons, the librados2
API needs to be preserved and it needs to continue to be compatible
with dependent libraries like librbd1.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
this change reverts 7d48d21a
we are using bluestore for testing by default on GNU/Linux now. and on
FreeBSD, install-deps.sh does not look at spec.in or d/control.
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/24805/head:
qa/suite: add dedup test
src/tools: fix compile error (master version issue)
src/tools: add stats (fixed objects,total objects)
src/tools: make room for cdc
src/tools: make enhacned stats and interface class
src/tools: set timelimit and add signal handler to check progress
src/tools: use the slice thing and make parallel (chunk_scrub)
src/test: add max-thread test in test_dedup_tool.sh
src/tools: use the slice thing and make parallel
src/test: add chunk-scrub test in test_dedup_tool.sh
src/tools: add chunk-scrub op in dedup tool
src/cls/cas: add has_chunk op
src/test: add test_dedup_tool.sh
src/tools: initial works for dedup tool
Reviewed-by: Sage Weil <sage@redhat.com>
mark dependencies used for `make check` run with `Make-Check`
in which, socat is used only by `qa/standalone/mon/mon-bind.sh`, we
could mark it with QA-Run, but since there only a single dependency in
this bucket, let's wait for more of them before adding the QA-Run
category.
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change reverts 7d48d21a
we are using bluestore for testing by default on GNU/Linux now. and on
FreeBSD, install-deps.sh does not look at spec.in or d/control.
Signed-off-by: Kefu Chai <kchai@redhat.com>
to make ceph-mgr-dashboard a separated package
- helps to reduce the repo size of downstream. because
ceph-mgr-dashboard is an architecture independent package. by
making it separated package avoids needless duplication of
the same data in mutiple .debs.
- gives user a fine grained control of selection.
Signed-off-by: Kefu Chai <kchai@redhat.com>
python-jinja2 dependency was added for the dashboard v1 in 87399bea. but
dashboard v2, which is now known as "dashboard" has taken the place of
the old dashboard. and dashboard now does not use jinja2, so we should
drop this dependency.
Signed-off-by: Kefu Chai <kchai@redhat.com>
change following plugins' Architecture to "all", as they are pure python
plugins, and are architecture-independent packages.
- ceph-mgr-diskprediction-cloud
- ceph-mgr-diskprediction-local
- ceph-mgr-rook
this also matches their couterparts in ceph.spec.in, where we have
```
BuildArch: noarch
```
Signed-off-by: Kefu Chai <kchai@redhat.com>
* refs/pull/25977/head:
qa/suites: exclude new packages when installing old versions
rpm: add dependency on python-kubernetes module to ceph-mgr-rook package
rpm,deb: add rbd_support module to ceph-mgr
packaging: split ceph-mgr diskprediction and rook plugins into own packages
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
We can re-enable after we branch. For N, though, there is no point in dragging
in the dependencies and implying there is something that works.
Signed-off-by: Sage Weil <sage@redhat.com>
This provides an 'iostat' and 'top'-like IO monitor for all
RBD images
Fixes: http://tracker.ceph.com/issues/37913
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
The ambiguous shebang now produces an error in rawhide and halts the
build. In f29 this was a warning. Add python3 as a dependency for
ceph-fuse.
Fixes: http://tracker.ceph.com/issues/37787
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
Cython version 0.29 removed the support for python subinterpreters,
which completely breaks ceph-mgr funcionality.
See cython repo commit:
7e27c7cd51
Fixes: http://tracker.ceph.com/issues/37472
Signed-off-by: Ricardo Dias <rdias@suse.com>
Fixes: https://tracker.ceph.com/issues/367419fd30b93f7 moved
/etc/bash_completion.d/radosgw-admin from radosgw to ceph-common. This
means that if you try and install a newer ceph-common over an older
radosgw, there's a conflict, and the install fails:
```
Unpacking ceph-common (12.2.8-1xenial) over (10.2.9-0ubuntu0.16.04.1) ...
dpkg: error processing archive ceph-common_12.2.8-1xenial_amd64.deb (--install):
trying to overwrite '/etc/bash_completion.d/radosgw-admin', which is also in package radosgw 10.2.9-0ubuntu0.16.04.1
```
Per Debian policy (
https://www.debian.org/doc/debian-policy/ch-relationships.html#overwriting-files-in-other-packages
) the correct way to handle a package taking over a file is for a
versioned Replaces and Breaks.
The change went into 12.0.3, so this commit adds Replaces and Breaks
against radosgw less than that version. It should be backported to
Luminous to avoid issues with upgrades from older versions (Jewel and
Kraken).
Signed-off-by: Matthew Vernon <mv3@sanger.ac.uk>
Currently /var/lib/ceph/bootstrap-rbd-mirror is absent, which means we
need to create it manually in order to pool the
client.bootstrap-rbd-mirror key.
Signed-off-by: Sébastien Han <seb@redhat.com>
be more explicit on what we are packaging. because only
libceph-common.so.${soversion} will be packaged, since libceph-common.so
won't be installed by cmake anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
