Commit Graph

98796 Commits

Author SHA1 Message Date
Yuri Weinstein
103a4299cd qa/tests: removed 1node and systemd tests as ceph-deploy is not actively developed
Fixes: http://tracker.ceph.com/issues/40207
Fixes: http://tracker.ceph.com/issues/40208
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-06-07 14:39:15 -07:00
Matt Benjamin
b4c7d0faef rgw_file: all directories are virtual with respect to contents
This change causes directory handles to always report an mtime of
"now."  This is not an invalidate per se--it interacts with the
nfs implementation to produce that result when the implementation
updates its cached attributes.  Hence, it can be modulated by timers
or other rules governing attribute caching at the upper layer.

Fixes: http://tracker.ceph.com/issues/40204

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-07 14:59:53 -04:00
Matt Benjamin
167ce67828
Merge pull request #28440 from linuxbox2/wip-rgwfile-invbckt
rgw_file: permit lookup_handle to lookup root_fh
2019-06-07 14:25:48 -04:00
Neha Ojha
2780813667
Merge pull request #28442 from neha-ojha/wip-40120-port
qa/tasks/cbt.py: change port to work with client_endpoints

Reviewed-by: Mark Nelson <mnelson@redhat.com>
2019-06-07 09:36:52 -07:00
Kefu Chai
6e62f5f855 install-deps.sh: update ubuntu-toolchain-r mirrors
the removed two do not have GCC-9, but the newly added mirros do.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-08 00:14:54 +08:00
Kefu Chai
0e5ab6f95a install-deps.sh: use GCC-9 on bionic
two reasons

* GCC-9 features more checks. so let's use it!
* crimson targets the hardware + toolchain + kernel after 1+ years,
  so it would be great if we can compile and test crimson using
  the toolchain which is ubiquitous on most mainstream distos.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-08 00:14:54 +08:00
Yuri Weinstein
dd1ef1081a
Merge pull request #28435 from yuriw/wip-yuriw-smoke-master
qa/tests: reduced distro to run to be random
2019-06-07 09:13:49 -07:00
Kefu Chai
5cc1a62fda
Merge pull request #28428 from tchaikov/wip-package-crimson-osd
rpm,debian,install-deps: package crimson-osd

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
2019-06-08 00:03:22 +08:00
Kefu Chai
f03d8e2548 ceph.spec.in: align "Requires" of ceph-osd
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Kefu Chai
34b02f00d0 ceph.spec.in: package crimson-osd
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Kefu Chai
9bdf2dfa0d debian/: add ceph-crimson-osd package
this package is commented out using `# Crimson`. and it will be enabled
only if we are building "crimson" flavor builds.

this package depends on ceph-osd, because it'll basically be a drop-in
replacement of ceph-osd. and it shares all the dependencies of ceph-osd.
so let's just reuse ceph-osd package for now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Kefu Chai
05899c1b86 cmake: install crimson-osd if WITH_SEASTAR
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Kefu Chai
7f19bf39d6 install-deps.sh: populate WITH_SEASTAR in the same way as for_make_check
because `install-deps.sh` is executed using `source`, we have to pass
these options using env variables. but before this change, `WITH_SEASTAR` is used directly,
while `FOR_MAKE_CHECK` is checked and translated to a local variable
`for_make_check`. which, in my opinion, has better readability.

so, in this change, `WITH_SEASTAR` is translated to `with_seastar`
variable in `install-deps.sh`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-07 22:17:19 +08:00
Neha Ojha
ee02416c65 qa/tasks/cbt.py: change port to work with client_endpoints
hack to work with https://github.com/ceph/cbt/pull/160/

Signed-off-by: Neha Ojha <nojha@redhat.com>
2019-06-06 17:21:38 -07:00
Yuri Weinstein
4c335b31fe
Merge pull request #28439 from yuriw/wip-yuriw-crontab-master
qa/tests:  filtered in only "trusty"
2019-06-06 16:24:19 -07:00
Patrick Donnelly
2b7d75a219
Merge PR #27731 into master
* refs/pull/27731/head:
	qa: use mimic-O upgrade process

Reviewed-by: Sage Weil <sage@redhat.com>
2019-06-06 14:48:44 -07:00
Yuri Weinstein
7939b1a10f qa/tests: filtered in only "trusty"
Fixes: http://tracker.ceph.com/issues/40195
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-06-06 14:36:57 -07:00
Matt Benjamin
547cb9f87b rgw_file: permit lookup_handle to lookup root_fh
Originally, root_fh was reachable from RGWLibFS::fh_cache, but
this was problematic as it has infinite lifetime and so is not
present in RGWLibFS::fs_lru.

To fix the above, RGWLibFS::root_fh came to live unhooked from the
handle caches which works because nfs-ganesha/the application
always has an instance handle.  However, this silently breaks
READDIR event invalidation.

Fixes: https://tracker.ceph.com/issues/40196

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-06 17:03:00 -04:00
Patrick Donnelly
66f18ecd09
qa: use mimic-O upgrade process
Fixes: https://tracker.ceph.com/issues/39436

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-06-06 13:06:56 -07:00
Sage Weil
78122147dc Merge PR #27015 into master
* refs/pull/27015/head:
	OSD: rollforward may need to mark pglog dirty

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-06-06 12:15:50 -05:00
Sage Weil
3d519bd96a Merge PR #28242 into master
* refs/pull/28242/head:
	osdc/Objecter: fix OSDMap leak in handle_osd_map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2019-06-06 12:15:06 -05:00
Sage Weil
5e569e1cf6 Merge PR #28363 into master
* refs/pull/28363/head:
	osd/ClassHandler: use std::variant for storing func
	osd/ClassHandler: no need to add `struct` before struct type
	osd/ClassHandler: use ceph::mutex instead of Mutex
	osd/ClassHandler: remove unused function
	osd/ClassHandler: s/string/std::string/ in header

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2019-06-06 12:14:49 -05:00
Yuri Weinstein
c6c90663c3 qa/tests: reduced distro to run to be random
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2019-06-06 10:08:55 -07:00
Jason Dillaman
163c122941 rbd-mirror: link against the specified alloc library
As a daemon, use the build-time specified alloc library
(tcmalloc/jemalloc) when linking the application.

Fixes: http://tracker.ceph.com/issues/40110
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-06-06 12:17:11 -04:00
Varsha Rao
d141adac9a cephfs-shell: Remove unnecessary empty line
This patch fixes flake8 too many blank lines error.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-06-06 21:25:53 +05:30
Varsha Rao
d063e8b1e0 cephfs-shell: Fix flake8 bare 'except' warnings
Instead of bare 'except', catch OSError and libcephfs exceptions.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-06-06 21:25:53 +05:30
Varsha Rao
6a0a5ed83d cephfs-shell: Remove unused variable 'e'
Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-06-06 21:25:53 +05:30
Varsha Rao
db7b9306fb cephfs-shell: Fix flake8 use != to compare str literals warning
Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-06-06 21:25:53 +05:30
Varsha Rao
e41d467d4f cephfs-shell: Fixe flake8 line too long error
Break the line into multiple lines and add new variable wherever necessary, to
fix the line too long error.

Fixes: https://tracker.ceph.com/issues/40191
Signed-off-by: Varsha Rao <varao@redhat.com>
2019-06-06 21:25:53 +05:30
Jason Dillaman
2964297b9a
Merge pull request #28313 from Songweibin/wip-fix-get-args
rbd/action: fix error getting positional argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-06-06 11:19:35 -04:00
Jason Dillaman
df691487d5
Merge pull request #28253 from Songweibin/wip-39973-rbd
osd/OSDCap: rbd profile permits use of rbd_info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-06-06 11:18:43 -04:00
Jason Dillaman
6cd6f5e607
Merge pull request #27902 from mikechristie/rbd-nbd-netlink-support
rbd-nbd: add netlink map/unmap support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2019-06-06 10:36:48 -04:00
Casey Bodley
8645e24dd4 common: OutputDataSocket retakes mutex on error path
the list of buffers is protected by mutex m_lock. when dump_data() fails
and puts unwritten buffers back into the list, it needs to reaquire the
lock

Fixes: http://tracker.ceph.com/issues/40188

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-06-06 09:55:41 -04:00
myoungwon oh
75ca528b03 test: add unittests
Signed-off-by: Myoungwon Oh <omwmw@sk.com>
2019-06-06 22:36:58 +09:00
Kefu Chai
8066a55b32 mgr/influx: try to call close()
let's at least try to call influxclient.close() which in turn closes the
requests.Session

see
https://github.com/influxdata/influxdb-python/blob/v4.1.1/influxdb/client.py
and
https://github.com/influxdata/influxdb-python/blob/v5.0.0/influxdb/client.py

Fixes: http://tracker.ceph.com/issues/40174
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-06 21:13:38 +08:00
Matt Benjamin
7a38429a08
Merge pull request #28370 from linuxbox2/wip-rgw-tntbucket
rgw_file: include tenant when hashing bucket names
2019-06-06 08:58:29 -04:00
myoungwon oh
f745a7d35f common,osd: add hash algorithms for dedup (sha256, sha512)
Signed-off-by: Myoungwon Oh <omwmw@sk.com>
2019-06-06 21:05:26 +09:00
Kefu Chai
4d363edc59 d/control,install-deps: use d/control for installing seastar deps
prepare for building ceph-crimson-osd deb package

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-06 18:06:43 +08:00
Lenz Grimmer
e7b605ae5b
mgr/dashboard: show degraded/misplaced/unfound objects. (#28104)
mgr/dashboard: show degraded/misplaced/unfound objects.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
2019-06-06 11:46:36 +02:00
Kefu Chai
e7a682ca54
Merge pull request #28039 from majianpeng/make-tcmalloc/jemalloc-work-well
make perf_async_msgr link jemalloc/tcmalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-06-06 17:38:58 +08:00
Xie Xingguo
2cc9324c05
Merge pull request #28373 from xiexingguo/wip-40104
mon, osd: parallel clean_pg_upmaps

Reviewed-by: Sage Weil <sage@redhat.com>
2019-06-06 17:37:15 +08:00
Zhi Zhang
176c24ef50 mds: count purge queue items left in journal
MDS purge queue didn't have a perf counter to record how many items
still left in journal. Even when MDS restarted, there was no any hint to
know how many inodes haven't been really deleted from disks.

Fixes: http://tracker.ceph.com/issues/40121

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
2019-06-06 11:32:34 +08:00
Jianpeng Ma
72c2926bae test/crimson: make perf_async_msgr link tcmalloc/jemalloc.
For 1M performance, perf_async_msgr w/ tcmalloc/jemalloc is better than
libc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2019-06-06 11:26:58 +08:00
Kefu Chai
3e7741aa2d cmake: add crimson-osd as a dependency of "tests" target
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-06 10:54:54 +08:00
xie xingguo
a45112a976 test: add parallel clean_pg_upmaps test
With parallel clean_pg_upmaps feature on, the total time cost
of the performance test which now can utilize up to 8 threads for
parallel upmap validating decreased from:

maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s

to:

clean_pg_upmaps (~10000 pg_upmap_items) latency:7s

Note that by default the mon uses only 4 worker threads for
CPU intensive background work, you could further increase
the "mon_cpu_threads" option value if you decided the
time-consuming of clean_pg_upmaps still matters.

Fixes: http://tracker.ceph.com/issues/40104
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-06 10:10:46 +08:00
xie xingguo
c395f45f1f mon/OSDMonitor: do clean_pg_upmaps the parallel way if necessary
There could definitely be some certain cases we could reliably
skip this kind of checking, but there is no easy way to separate
those out.
However, this is clearly the general way to do the massive pg
upmap clean-up job more efficiently and hence should make sense
in all cases.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-06 10:10:45 +08:00
xie xingguo
4d5cf1e417 osd/OSDMap: split clean_pg_upmaps into smaller helpers
- it's good to read.
- the updating pending_inc part should be made independent
  since it is going to be racy while running in parallel.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-06 10:10:45 +08:00
Kefu Chai
848c40a0aa
Merge pull request #28402 from tchaikov/wip-crimson-cleanup
crimson/osd: some cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>
2019-06-06 09:52:34 +08:00
David Zafman
590b4138ae
Merge pull request #28302 from dzafman/wip-40078
test: Make sure that extra scheduled scrubs don't confuse test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-06-05 14:43:30 -07:00
Boris Ranto
9c6cdf1e50 selinux: Update the policy for RHEL8
We hit a couple more SELinux denials when running ceph on RHEL8. The
dac_read_search change is related to a kernel change where it checks
dac_read_search before dac_override, now.

Signed-off-by: Boris Ranto <branto@redhat.com>
2019-06-05 19:29:23 +02:00