Commit Graph

123747 Commits

Author SHA1 Message Date
Patrick Donnelly
51b60e990b
script: add a few more volume mounts for sepia
We now have a few Ceph file systems with various possible mount points
depending which lab machine you're using.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 09:33:54 -07:00
Patrick Donnelly
9d71fc1bee
script: drop ceph-fuse from docker debugging
Install this on the fly as necessary...

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 09:33:30 -07:00
Patrick Donnelly
efbfd678a2
script: enable centos debuginfo repo for debugging
So we can fetch e.g. the sqlite debuginfo packages.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 09:32:52 -07:00
Patrick Donnelly
472bc091ad
script: update repo url for multi-arch builds
Brad suggested this change based on his commit [1]. Thank you!

[1] 267cce9e83

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 09:31:19 -07:00
Patrick Donnelly
6223ab14cc
script: fetch autobuild.asc key via HTTPS
Rather than relying the key being avaiable on the LRC /ceph file system.
(Someone appears to have deleted it recently.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-04 09:30:04 -07:00
Sage Weil
9b9d90bddf Merge PR #41553 into master
* refs/pull/41553/head:
	ceph-volume: replace __ with _ in device_id

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-03 22:04:55 -04:00
Sage Weil
ce033d629c Merge PR #41636 into master
* refs/pull/41636/head:
	mgr/cephadm/inventory: do not try to resolve current mgr host
	pybind/mgr/mgr_module: make get_mgr_ip() return mgr's IP from mgrmap
	mgr/restful: use get_mgr_ip() instead of hostname

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-03 22:04:32 -04:00
Kefu Chai
b5d9418c11
Merge pull request #41674 from tchaikov/wip-vstart-without-restful
vstart.sh: add an option named --without-restful

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-06-04 09:44:58 +08:00
Kefu Chai
25480e2452
Merge pull request #41670 from tchaikov/wip-op-tracking-spin-off-0
crimson, common: improve const-correctness of Operation::dump()s.

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-06-04 07:50:44 +08:00
Kefu Chai
295afe68a2
Merge pull request #41672 from tchaikov/wip-crimson-test-handle-fut
test/crimson/seastore: always handle returned future<>

Reviewed-by: Samuel Just <sjust@redhat.com>
2021-06-04 07:50:21 +08:00
Patrick Donnelly
e3c4ae389e
Merge PR #41654 into master
* refs/pull/41654/head:
	mds: do not infinitely recursively print a metric

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
2021-06-03 13:34:54 -07:00
Patrick Donnelly
943915f8df
Merge PR #41639 into master
* refs/pull/41639/head:
	mds/scrub: write root inode backtrace at creation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:33:58 -07:00
Patrick Donnelly
a12db7941b
Merge PR #41499 into master
* refs/pull/41499/head:
	qa/tasks/mds_thrash: fix thrash iteration never skip

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:33:27 -07:00
Patrick Donnelly
a52712f955
Merge PR #41443 into master
* refs/pull/41443/head:
	test: update log-ignorelist for fs:mirror test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:23:17 -07:00
Patrick Donnelly
4e1f812461
Merge PR #39910 into master
* refs/pull/39910/head:
	test: Add test for mgr hang when osd is full
	mgr: Set client_check_pool_perm to false
	mds: Add full caps to avoid osd full check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2021-06-03 13:22:23 -07:00
Dan Mick
2544ee6d7f
Merge pull request #41559 from dmick/wip-grafana-container
monitoring/grafana/build/Makefile: revamp for arm64 builds, pushes to docker and quay, jenkins
2021-06-03 11:32:24 -07:00
Sage Weil
45e806676b mgr/cephadm/inventory: do not try to resolve current mgr host
The CNI configuration may set up a private network for the container, which
is mapped to the hostname in /etc/hosts.  For example, my test box sets
up 10.88.0.0/24 because I was using crio + kubeadm on this host earlier
(at least I think that's why):

$ sudo podman run --rm --name test123 --entrypoint /bin/bash -it quay.ceph.io/ceph-ci/ceph:master -c "cat /etc/hosts"
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.8	f9e91bf2478f test123

In any case, we should never trust a lookup of our own hostname from inside
a container!

This isn't quite sufficient, though: if this is a single-host cluster, then
we fall back to using get_mgr_ip(). That value may be distorted by the
public_network option on the mgr, but we don't have any other good
options here, and single-node clusters are unlikely to have complex
network configs.

Refactor a bit to avoid the try/except nesting.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-03 12:11:31 -04:00
Sage Weil
72d72fc5db pybind/mgr/mgr_module: make get_mgr_ip() return mgr's IP from mgrmap
The previous approach was convoluted: we tried to do a DNS lookup on the
hostname, which would fail if /etc/hosts had an entry.  Which, with podman,
it does.  And the IP it has will vary in all sorts of weird ways.  For
example, CNI on my host means that I get a dynamic address in 10.88.0.0/24.

Avoid all of that nonsense and use the IP that is in the mgrmap.  There
may be multiple IPs (v2 + v1, or maybe even IPv4 + v6 in the future); in
that case, use the first one.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-03 12:11:31 -04:00
Sage Weil
df79946174 mgr/restful: use get_mgr_ip() instead of hostname
Now we match dashboard!

Signed-off-by: Sage Weil <sage@newdream.net>
2021-06-03 12:11:31 -04:00
Neha Ojha
11252f6117
Merge pull request #41308 from sseshasa/wip-osd-benchmark-for-mclock
osd: Run osd bench test to override default max osd capacity for mclock

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2021-06-03 08:39:22 -07:00
Casey Bodley
38ae161670
Merge pull request #41316 from cbodley/wip-50785
rgw: parse tenant name out of rgwx-bucket-instance

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
2021-06-03 11:05:00 -04:00
Kefu Chai
a53592e484
Merge pull request #41677 from tchaikov/wip-oom
ceph.spec.in: increase the mem_per_job to 3GiB

Reviewed-by: David Galloway <dgallowa@redhat.com>
2021-06-03 22:40:26 +08:00
Casey Bodley
0f44459bc9
Merge pull request #41668 from pleiadesian/patch-bucket-chown
rgw: require bucket name in bucket chown

Reviewed-by: Or Friedmann <ofriedma@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2021-06-03 10:28:35 -04:00
Casey Bodley
416d60c20d
Merge pull request #41462 from yehudasa/wip-50920
rgw: auth v4 client: don't convert '+' to space

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2021-06-03 10:16:30 -04:00
Kefu Chai
7577fec7fd cmake: increase the MAX_{LINK,COMPILE}_MEM
based on recent observation, quite a few C++ source file take
around more than 3.0GiB to compile. for instance,
test_mock_HttpClient.cc could take up to 6270MiB memory to compile.

so increase MAX_{LINK,COMPILE}_MEM accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 21:03:03 +08:00
Kefu Chai
38be14bc0f ceph.spec.in: increase the mem_per_job to 3GiB
to lower the number of jobs, we are experiencing build failures on
a builder with 48c96t, 193 free mem. the failures were caused by
OOM killer which kills the c++ compiler

[498376.128969] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jenkins.service,task=cc1plus,pid=1387895,uid=1110
[498376.145288] Out of memory: Killed process 1387895 (cc1plus) total-vm:3323312kB, anon-rss:3164568kB, file-rss:0kB, shmem-rss:0kB, UID:1110
[498376.315185] oom_reaper: reaped process 1387895 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[498377.882072] cc1plus invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

before this change, we use the total memory to calculate the number
of jobs, and assume that each job takes at most 2.5GiB mem. in the
case above, the # of job is 96.

after this change, we use the free memory, and increse the mem per job
to 3.0GiB. in the case above, the # of job would be 85.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 21:02:33 +08:00
Kefu Chai
e9ac37d424
Merge pull request #41669 from tchaikov/wip-crimson-asok-dump-metrics
crimson/admin: s/perf dump_seastar/dump_metrics/

Reviewed-by: Amnon Hanuhov <ahanukov@redhat.com>
2021-06-03 19:45:23 +08:00
Kefu Chai
284884218a vstart.sh: use here document to display multi-line message
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 19:43:10 +08:00
Kefu Chai
edd05d764d vstart.sh: add an option named --without-restful"
so we don't need to wait for restful module to be loaded if not working
on this mgr module.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 19:43:10 +08:00
Kefu Chai
a3dd9d51a5 vstart.sh: extract create_mgr_restful_secret() out
for better readability, and so it's easier to make this step optional if
developer is not interested in using the restful mgr module.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 19:43:10 +08:00
Sridhar Seshasayee
76420f9d59 doc: Update mclock-config-ref to reflect automated OSD benchmarking
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
2021-06-03 14:45:21 +05:30
Kefu Chai
94224614e0
Merge pull request #41671 from liu-chunmei/seastore-logger
crimson/seastore: cleanup ceph_subsystem_filestore to seastore

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-03 15:39:16 +08:00
Kefu Chai
ea3a00a513 test/crimson/seastore: declare return type explicitly
for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 15:37:34 +08:00
Kefu Chai
b6e9822058 test/crimson/seastore: always handle returned future<>
this change also silences the [-Wunused-result] warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 15:37:34 +08:00
Radoslaw Zarzynski
f60d0a1106 common: fix a formatting nit in OpTracker::dump_ops_in_flight().
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-06-03 14:47:39 +08:00
Radoslaw Zarzynski
84e8a81ecd crimson: improve const-correctness of Operation::dump()s.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2021-06-03 14:47:39 +08:00
chunmei-liu
4a76d8f79d crimson/seastore: cleanup ceph_subsystem_filestore to seastore
Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
2021-06-02 23:44:35 -07:00
Kefu Chai
a7d99054f8
Merge pull request #41666 from tchaikov/wip-crimson-stop
crimson/osd: wait for SIGINT and SIGTERM before stopping

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
2021-06-03 14:33:52 +08:00
Kefu Chai
ff8d35d35e crimson/admin/osd_admin: sort forward declarations
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 13:59:21 +08:00
Kefu Chai
87949afba0 crimson/admin: fix the indent
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 13:59:21 +08:00
Kefu Chai
1a48978794 crimson/admin: s/perf dump_seastar/dump_metrics/
as a user-facing interface, no need to expose seastar in the name,
what matters to user is the content not the underlying technology or library.

so rename the command prefix to "dump_metrics"

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 13:59:21 +08:00
Kefu Chai
9052c8ccd4 crimson/admin: s/SeastarMetricsHook/DumpMetricsHook/
seastar is the name of one of the libraries used to implement crimson,
but the asok hook dumps not only builtin metrics in seastar, but also
the ones registered by crimson and seastore, so rename it to a more
general name.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 13:59:21 +08:00
Zulai Wang
158a1f4313 rgw: require bucket name in bucket chown
Checking and reporting missing the mandatory parameter avoid clueless error
message for bucket chown.

Signed-off-by: Zulai Wang <zl31wang@gmail.com>
2021-06-03 13:41:07 +08:00
Kefu Chai
4945ff12e9 crimson/osd: wait for SIGINT and SIGTERM before stopping
this change addresses an regression introduced by
37b83f4ed7. as we should not stop
services without being asked to do so.

in this change, signal handler for SIGINT and SIGTERM is registered to
handle these signals, and in the seastar thread, we wait until any of
these two signals is caught.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 13:30:01 +08:00
Kefu Chai
a9ea8c7268
Merge pull request #41627 from tchaikov/wip-mgr-repl-doc
doc/mgr/modules: add a "debugging" section

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
2021-06-03 09:36:15 +08:00
Kefu Chai
6dbb567de4
Merge pull request #41138 from kalebskeithley/python39
do_cmake: build with python3.9 on RHEL9

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-03 09:34:56 +08:00
Kefu Chai
d58a88c464 do_cmake: build with python3.9 on RHEL9
rhel9 has python3.9 as of rhel9beta

Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>
2021-06-03 09:34:01 +08:00
Kefu Chai
09357b1be7
Merge pull request #41496 from Huber-ming/correct_spell
rgw: correct the spelling of "instace"

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-03 09:16:42 +08:00
Patrick Donnelly
5871240363
Merge PR #41635 into master
* refs/pull/41635/head:
	qa: increase fragmentation to improve uniform distribution

Reviewed-by: Ramana Raja <rraja@redhat.com>
2021-06-02 08:18:22 -07:00
Kefu Chai
7025d50813
Merge pull request #41644 from rzarzynski/wip-crimson-fix-blocked-peering
crimson/monc: fix subscription stall that blocked peering.

Reviewed-by: Kefu Chai <kchai@redhat.com>
2021-06-02 22:43:40 +08:00