Commit Graph

1169 Commits

Author SHA1 Message Date
Sage Weil
5f15829174 Merge PR #30738 into master
* refs/pull/30738/head:
	mgr/alerts: raise health alert if smtplib has a problem
	mgr/alerts: simple module to send health alerts

Reviewed-by: Tim Serong <tserong@suse.com>
2019-10-09 14:13:40 -05:00
Sage Weil
6fd67f19e3 Merge PR #30603 into master
* refs/pull/30603/head:
	ceph-daemon: -n type.id instead of -i id
	ceph-daemon: drop unused VERSION
	ceph-daemon: clean up dir helpers, tighten up permissions
	ceph-daemon: fchmod before writing to keyring file
	test_ceph_daemon.sh: skip ssh until container image has remoto
	ceph-daemon: decode utf-8 in run() helper
	mgr/ssh: clean up debug cruft
	mgr/ssh: clean up bare except: block
	ceph-daemon: clean up bare except: blocks
	ceph-daemon: all imports to top
	ceph-volume: no_tmpfs -> tmpfs
	doc/bootstrap: add new bootstrap documentation
	ceph-daemon: add --output-pub-ssh-key for bootstrap
	ceph-daemon: make 'shell' easier to use
	ceph-daemon: support docker; prefer podman
	qa: add ceph-daemon
	debian: ceph-daemon package, required by ceph-mgr-ssh
	ceph.spec.in: ceph-daemon package, required by ceph-mgr
	common/options: cleanup whitespace
	mgr/ssh: simplify getting the cluster fsid
	mgr/ssh: pipe ceph-daemon script to stdin of python3
	ceph-daemon: add support for args and/or stdin from top of script
	ceph-daemon: make ceph-volume use get_config_and_keyring
	ceph-daemon: ls: behave if /var/log/ceph doesn't exist
	ceph-daemon: implement 'adopt' for legacy style daemons
	ceph-daemon: fix fsid detection for legacy osds
	ceph-daemon: make rm-cluster clean up system-ceph*.slice too
	ceph-daemon: configure ssh orchestrator
	ceph-daemon: be more restrictive with file permissions
	mgr/ssh: create osd with ceph-daemon
	mgr/ssh: pass daemon id separately to _create_daemon
	ceph-daemon: add --config-and-keyring to ceph-volume command
	ceph-daemon: create log path for shell (if needed)
	mgr/ssh: use _run_ceph_daemon for _create_daemon
	mgr/ssh: factor _run_ceph_daemon out of _get_device_inventory
	mon/ConfigMonitor: allow entity type only for 'config get'
	ceph-daemon: add ceph-volume subcommand
	ceph-daemon: remove unused CephContainer dname property
	ceph-daemon: drop useless uid/gid checks
	mgr/ssh: deploy new mgrs with ceph-daemon
	mgr/ssh: factor _create_daemon out of create_mon
	mon/MonCap: allow mgr to create new auth keys
	mgr/ssh: run c-v with podman when getting inventory
	mgr/ssh: simplify ssh connection management
	mgr/ssh: use ceph-daemon for deploying mon
	ceph-daemon: allow --mon-network for deploying new mon (vs specifying IP)
	ceph-daemon: --config-and-keyring (not key)
	common/options: add 'image' config option
	test_ceph_daemon: specify image name
	vstart.sh: add --ssh to enable+configure ssh orchestrator
	mgr/ssh: use ssh identity from config-key, if present
	mgr/ssh: hardcode default ssh_config
	ceph-daemon: store ssh identity in mon config-key store
	ceph-daemon: --privileged arg for 'exec'
	ceph-daemon: make deploy work for osd (do a c-v prepare)
	ceph-daemon: make shell privileged
	ceph-daemon: move get_container_mounts to a helper
	ceph-daemon: pass full path for entrypoint
	ceph-daemon: make id portion of 'shell' optional
	ceph-volume: accept --no-tmpfs argument for bluestore
	ceph-daemon: 'unit' command
	ceph-daemon: fix run command to use call(), not check_output()
	src/ceph-daemon: whitespace
	ceph-daemon: add 'enter', 'exec' commands
	ceph-daemon: bind config to default location
	test_ceph_daemon.sh: test deploy mds too
	ceph-daemon: generate ssh keys
	ceph-daemon: --config, not --conf
	ceph-daemon: long lines
	ceph-daemon: add --config to bootstrap
	ceph-daemon: add 'shell' command
	ceph-daemon: do not import subprocess symbols directly
	ceph-daemon: add mons with 'deploy mon.x ...'
	ceph-daemon: add 'ls'
	ceph-daemon: simplify uid/gid a bit
	ceph-daemon: fix libudev
	ceph-daemon: autodetect uid/gid from container image
	ceph-daemon: default to empty log files, log to stderr (systemd journal)
	ceph-daemon: rm-{daemon,cluster}
	ceph-daemon: fix bootstrap config
	ceph-daemon: fix args.fsid usage
	ceph-daemon: be careful overwriting live files
	ceph-daemon: slurp some options over from the standard systemd unit
	ceph-daemon: add ceph.target and ceph-$fsid.target units
	test_ceph_daemon.sh: stupid test script
	ceph-daemon: bootstrap and deploy (mgr) work
	ceph-daemon: initial checkin
	ceph-mon: fix debug print of public_addr
2019-10-07 15:31:14 -05:00
Sage Weil
d7223938f8 mgr/alerts: simple module to send health alerts
Initialy SMTP support is implemented; nothing else.

This is just smart enough for me to get emails from my home cluster when
something goes wrong.  No bells and whistled at this point.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-07 09:04:39 -05:00
Kefu Chai
73ee20ea26
Merge pull request #30656 from tchaikov/wip-install-deps-for-el8
install-deps.sh: enable PowerTool repo for EL8

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-10-07 16:06:30 +08:00
Sage Weil
171fecc38a ceph.spec.in: ceph-daemon package, required by ceph-mgr
Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-04 20:33:35 -05:00
Kefu Chai
0098b42927 ceph.spec.in: install python3-Cython on el8
EL8 has python3 support. no need to wait for python3-Cython to populate
to its mirrors.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-10-01 18:28:17 +08:00
Kefu Chai
e92cb7a033 ceph.spec.in: disable "make check" for rhel8
following packages are not available on RHEL/CentOS 8, they are for
dashboard unit tests. so disable all python packages for dashboard
tests for now:

python-cherrypy
python-coverage
python-pecan
python-routes
python-tox
xmlsec1-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-28 22:20:35 +08:00
Kefu Chai
dd133840b9 ceph.spec.in: disable cephfs_java and amqp_endpoint if RHEL8
RHEL/CentOS 8 does not offer junit and sharutils yet, so disable
cephfs_java on rhel8 at this moment.

RHEL/CentOS 8 does not offer librabbitmq-devel yet, so disable
amqp_endpoint on rhel8 at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-28 22:20:35 +08:00
Jeff Layton
40c5a18f81 ceph.spec.in: fix Cython package dependency for Fedora
Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-09-26 08:20:57 -04:00
Kefu Chai
b43cbefc05
Merge pull request #30420 from runsisi/wip-remove-requests
rpm,deb: python-requests is not needed for ceph-common

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-25 16:31:13 +08:00
Kefu Chai
6d2f66846d ceph.spec.in: s/pkgversion/version_nodots/
`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-24 23:23:35 +08:00
luo.runbing
998a3025c0
rpm,deb: python-requests is not needed for ceph-common
Signed-off-by: luo.runbing <luo.runbing@zte.com.cn>
2019-09-23 09:56:19 +08:00
Jeff Layton
eae0127513 mount.ceph: fork a child to get info from local configuration
When a secret and/or the mon addrs are not specified by the admin,
then mmap a MAP_SHARED buffer and spawn a child process to get that
info. For safety reasons, the child drops all capabilities other than
CAP_DAC_READ_SEARCH (to ensure that it'll be able to read the keyring,
should one be found). To achieve this, we add a new dependency on
libcap-ng.

Add a new C++ file with a single routine that will create a CephContext,
get a list of monitor addresses and scrape the keyring for a secret for
the specified cephx user.

If that info is found, then it is copied to fixed-length buffers in the
MAP_SHARED area and the child exits successfully.

The parent will then vet the returned info and copy it into the
appropriate fields if they are currently blank.

Fixes: https://tracker.ceph.com/issues/16656
Signed-off-by: Jeff Layton <jlayton@redhat.com>
2019-09-13 08:14:48 -04:00
Kefu Chai
4d72fba539
Merge pull request #30126 from dvanders/dvanders_spec
ceph.spec.in: reserve more memory per build job

Reviewed-By: Nathan Cutler <nculter@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-09-10 16:30:09 +08:00
Sebastian Wagner
199d6418f4
Merge pull request #29520 from pcuzner/add-events-mgr-module
mgr/k8sevents: Add mgr module for kubernetes event integration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>
2019-09-06 17:09:21 +02:00
Dan van der Ster
e466c43b23 ceph.spec.in: reserve 2500MB per build job
On our 14GB build machines, 8 parallel jobs results in the compiler
going OOM. Reserving 2500MB per job allows it to succeed with 5 jobs.

Fixes: https://tracker.ceph.com/issues/41659
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
2019-09-06 08:52:21 +02:00
Paul Cuzner
2166676288 mgr/k8sevents: Initial ceph -> k8s events integration
Adds a mgr module to provide integration between
Ceph and the kubernetes events API within the rook-ceph
namespace. It provides several commands to view event
status including;

ceph k8sevents list .... show all k8s related events
ceph k8sevents ceph .... show events generated from this
                         module
ceph k8sevents status .. show status of the tracker threads
                         and counts of tracked events

Events sent to kubernetes are sourced from the
a) clog: ceph healtchecks and admin commands
b) explicit checks: hosts, pools and OSD states are checked
   every 'n' seconds

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>
2019-09-05 14:45:52 +12:00
Kefu Chai
33dfffc841 ceph.spec.in: use g++ >= 8.3.1-3.1
since https://bugzilla.redhat.com/show_bug.cgi?id=1726630 has been
fixed. and we have devtoolset-8-gcc-c++-8.3.1-3.1.el7.x86_64.rpm.

no reason to stick with devtoolset-8-gcc-c++-8.2.1-3 anymore.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-09-03 14:19:31 +08:00
Kefu Chai
5d6d770e2f install-deps.sh,deb,rpm: move python-saml deps into debian/control and ceph.spec.in
these dependencies are only used for building python-saml which is in
turn used for the SAML support. this feature is tested using
`test_sso.py` while performing dashboard tests. we do not package or
ship python-saml along with other Ceph packages. so let's move these
dependencies to the "make check" sections in ceph.spec.in and
debian/control for simplifying install-deps.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-24 02:11:51 +08:00
Kefu Chai
3234fb3982 cmake,run-make-check.sh,deb,rpm: disable SPDK by default
but we still enable it in `run-make-check.sh`

* cmake: disable SPDK by default
* run-make-check.sh: enable WITH_SPDK so at least we can ensure it
  builds
* deb,rpm: add uuid-dev / libuuid-devel as a "make check" dependency

Fixes: https://tracker.ceph.com/issues/41330
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-20 15:38:33 +08:00
Nathan Cutler
b9c7443010 rpm: always build ceph-test package
Fixes: https://tracker.ceph.com/issues/41296
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-08-15 13:28:51 +02:00
Boris Ranto
d9f8a50a89
Merge pull request #28997 from b-ranto/wip-push-dash
Make ceph-dashboard require grafana dashboards

Reviewed-by: Zack Cerza <zcerza@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-08-15 11:02:43 +02:00
Kefu Chai
bbb69fe793 cmake: require CMake v3.10.2
since we dropped the support of xenial, we now have the luxury of using
newer CMake! and by using CMake 3.10.2, we can prevent libfmt from
assuming that we are using C++11, and hence set `CMAKE_CXX_STANDARD` to
11, which will literally append `-std=gnu++11` to `CMAKE_CXX_FLAGS`.
the last `-std` option passed to `g++` takes precendence.
since we've switched over to C++17, and we are using C++17 features.
so, using cmake older than 3.8 breaks the build. because it is CMake 3.8
which stared support `CMAKE_CXX_STANDARD` 17.

- for bionic: https://packages.ubuntu.com/bionic/cmake : 3.10.2
- for CentOS7:
https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/c/ : 3.13.5

so in this change,

* bump up the required version to v3.10.2
* cleanups to wipe out the workaround for lower CMake versions
* use `PROJECT_VERSION` defined by `project()` command instead of
  `VERSION` explicitly defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-08-02 22:09:12 +08:00
Kefu Chai
0e35c6f641
Merge pull request #29349 from smithfarm/wip-40975
rpm: put librgw lttng SOs in the librgw-devel package

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-08-02 11:52:07 +08:00
Sebastian Wagner
475159ec73
Merge pull request #28915 from sebastian-philipp/python-common-package-2
python-common:  Python common package

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>
2019-07-30 09:31:34 +02:00
Kefu Chai
3169da7a1a
Merge pull request #29347 from smithfarm/wip-40974
rpm: improve ceph-mgr plugin package summaries

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-07-30 12:46:37 +08:00
Nathan Cutler
7929af71d9 rpm: put librgw lttng SOs in the librgw-devel package
Shared objects belong in the devel package.

Fixes: http://tracker.ceph.com/issues/40975
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-07-26 13:10:03 +02:00
Nathan Cutler
ea40019c5c rpm: improve ceph-mgr plugin package summaries
Fixes: http://tracker.ceph.com/issues/40974
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-07-26 13:01:05 +02:00
Nathan Cutler
876d6bf6dc rpm: fdupes in SUSE builds to conform with packaging guidelines
Quoting from https://en.opensuse.org/openSUSE:Packaging_Conventions_RPM_Macros

"%fdupes is generally safe for use on /bin, /lib*, /usr (= %_prefix) and /sbin.
Do note however that files with same content but different ownership, when
hardlinked, will get the ownership of any of its content siblings."

Fixes: http://tracker.ceph.com/issues/40973
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-07-26 12:42:09 +02:00
Sebastian Wagner
ea15a67b98 ceph.spec, debian: Add python-common to ceph-common
Co-authored-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2019-07-26 12:10:52 +02:00
Matthias Gerstner
d472e88441
packaging: remove SuSEfirewall2 support
removal of SuSEfirewall2 service, since SuSEfirewall2 has been replaced
by firewalld, see [1].

[1]: https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html

Fixes: http://tracker.ceph.com/issues/40738
Signed-off-by: Matthias Gerstner <matthias.gerstner@suse.de>
2019-07-12 09:55:35 +02:00
Boris Ranto
7b3a01cdfb rpm: Require ceph-grafana-dashboards
The recently introduced 'dashboard grafana dashboards update' command
requires the grafana dashboards for it to work properly. Adding the
dependency.

Signed-off-by: Boris Ranto <branto@redhat.com>
2019-07-11 17:21:55 +02:00
Kefu Chai
00d03f99d0
Merge pull request #28884 from DimStar77/depchain
ceph.spec.in: Drop systemd BuildRequires in case of building for SUSE

Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-07-09 16:04:09 +08:00
Kefu Chai
47d4351d3a ceph.spec.in,install-dep.sh: pin to gcc-c++-8.2.1
as an intermediate solution before
https://bugzilla.redhat.com/show_bug.cgi?id=1726630 gets fixed.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-07-05 23:59:21 +08:00
Dominique Leuenberger
b48ea142de
Packaging: Drop systemd BuildRequires in case of building for SUSE
There is already pkgconfig(systemd) present, which, in SUSE's packaging,
translates to the same package (systemd.pc is shipped as part of the main
systemd package). Not explicitly mentioning 'systemd' as package name allows
the openSUSE Build Service though to find shortcuts by using the
bootstrap packages, i.e. systemd-mini (ABI/API complete).

Signed-off-by: Dominique Leuenberger <dimstar@opensuse.org>
2019-07-05 09:15:20 +02:00
Sage Weil
18eacb6ed8 Merge PR #22446 into master
* refs/pull/22446/head:
	relicense LGPL-2.1 code as LGPL-2.1 or LGPL-3.0
	COPYING: add reference to a few Apache-2.0 licensed source files
	debian/copyright: sync with COPYING
	COPYING: LGPL2.1 -> LGPL-2.1
	COPYING: extra bits from debian/copyright

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
2019-06-12 09:11:49 -05:00
Nathan Cutler
fe632e2d88 build/ops: make "patch" build dependency explicit
This commit adds "patch" as an explicit build dependency. Before, the RPM
build was installing it in the build environment via some mechanism that I
don't entirely understand.

The "patch" command is used by the BuildBoost cmake module.

Fixes: http://tracker.ceph.com/issues/40175
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-06-10 14:15:00 +02:00
Kefu Chai
177956c9f6
Merge pull request #28352 from smithfarm/wip-rpm-obsolete-python
rpm: have pybind RPMs provide/obsolete their python2 predecessors

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-06-08 23:27:39 +08:00
Kefu Chai
5cc1a62fda
Merge pull request #28428 from tchaikov/wip-package-crimson-osd
rpm,debian,install-deps: package crimson-osd

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
2019-06-08 00:03:22 +08:00
Kefu Chai
f03d8e2548 ceph.spec.in: align "Requires" of ceph-osd
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Kefu Chai
34b02f00d0 ceph.spec.in: package crimson-osd
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Mike Christie
8815671553 cmake and build: add netlink lib requirement detection
The next patch adds netlink support to rbd-nbd. This patch just adds the
build related changes to bring in the netlink lib and test and build
against it.

Signed-off-by: Mike Christie <mchristi@redhat.com>
2019-06-04 23:56:53 -05:00
Nathan Cutler
55bc3493f0 rpm: have py3 RPMs provide/obsolete py2 predecessors
When upgrading from a Python 2 build to a Python 3 one, we need to make the
python3-* packages replace their Python 2 counterparts.

Fixes: http://tracker.ceph.com/issues/40099
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2019-06-05 00:15:25 +02:00
Boris Ranto
a30f6eda53
Merge pull request #28228 from jan--f/spec-grafana-dashboards-require-grafana
ceph.spec.in: grafana-dashboards package depends on grafana

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
2019-06-04 22:52:54 +02:00
Jan Fajerski
76204d32fb ceph.spec.in: install grafana dashboards world readable
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-06-02 13:31:27 +02:00
Nathan Cutler
34bd11b992
Merge pull request #28170 from marxin/disable-lto-in-spec-file
rpm: Disable LTO in spec when being used.

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
2019-05-29 18:21:34 +02:00
Martin Liska
6ab109bb9b rpm: Disable LTO in spec when being used
Fixes: http://tracker.ceph.com/issues/39974
Works around: https://tracker.ceph.com/issues/40060

Signed-off-by: Martin Liška <Martin.Liska@suse.com>
2019-05-29 14:26:42 +02:00
Jan Fajerski
c3ead45d1b ceph.spec.in: package prometheus default alerts for SUSE
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-05-10 10:01:18 +02:00
Sage Weil
5ef26e08ca Merge PR #26184 into master
* refs/pull/26184/head:
	mgr/dashboard: Replace IP address validation with Python standard library functions

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
2019-04-24 20:38:19 -05:00
Sage Weil
d64d14a577 Merge PR #26730 into master
* refs/pull/26730/head:
	src/test: remove unused objects to calculate accurate dedup ratio.
	src/tools: fix counting total objects
	src/tools: fix printing wrong offset.
	src/tools: fix dedup tool's name
	qa/workunit: add rabin chunk test
	src/tool: add rabin fingerprint option to cephdeduptool
	src/common: add rabin fingerprint class
	Initail work for rabin fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>
2019-04-24 20:38:02 -05:00