Commit Graph

94544 Commits

Author SHA1 Message Date
Kefu Chai
d42daf378c
Merge pull request #25616 from s0nea/wip-dashboard-osd-recv-speed-default-vals
mgr/dashboard: consider config option default values

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
2019-01-27 22:17:49 +08:00
Kefu Chai
11c9331263
Merge pull request #24813 from wjwithagen/wjw-fix-alpine-compat
common: Fix Alpine compatability for TEMP_FAILURE_RETRY and ACCESSPERMS

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-01-27 21:42:51 +08:00
Kefu Chai
77109d7582
Merge pull request #25916 from tchaikov/wip-debian-librados-dev
debian: librados-dev should replace librados2-dev

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-01-27 21:41:39 +08:00
Kefu Chai
fa6a69288c
Merge pull request #26099 from jdurgin/wip-remove-ms-async-affinity
common/options: remove unused ms async affinity options

Reviewed-by: Sage Weil <sage@redhat.com>
2019-01-27 21:41:10 +08:00
Kefu Chai
0cec1bdaad
Merge pull request #26026 from tchaikov/wip-memstore-ceph-mutex
os/memstore: use ceph::mutex and friends

Reviewed-by: Sage Weil <sage@redhat.com>
2019-01-27 21:40:30 +08:00
Sage Weil
1881d9abba msg/async: fix should_use_msgr2 behavior (including monc)
Be consistent about whether we should use v2 to connect or not, and fix
teh monclient check to use the same logic.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-27 07:33:51 -06:00
Kefu Chai
3ba9eaa890
Merge pull request #26159 from dzafman/wip-gitignore
git: Ignore tags anywhere

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-01-27 21:18:35 +08:00
Kefu Chai
94a84b6f5a test: listen on random port in tests which start ceph-mon
See-also: http://tracker.ceph.com/issues/36737
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 21:16:54 +08:00
Ilya Dryomov
11ee8f02b0
Merge pull request #26150 from dillaman/wip-rados-lock-info
rados: return legacy address in 'lock info'

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2019-01-27 08:18:45 +01:00
Kefu Chai
a8d6b2170d cmake: use $CMAKE_BINARY_DIR for default $CEPH_BUILD_VIRTUALENV
to avoid conflict with other `make check` runs in the same host

See-also: http://tracker.ceph.com/issues/36737
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 14:34:39 +08:00
Patrick Donnelly
032fda563c
Merge PR #25599 into master
* refs/pull/25599/head:
	mds: do not log internal journal error
	mds: clean up log level 2 messages
	mds: log session open stats

Reviewed-by: Zheng Yan <zyan@redhat.com>
2019-01-26 21:20:06 -08:00
David Zafman
6eec939e6f git: Ignore tags anywhere
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:48:48 -08:00
David Zafman
388e54d906 test: ceph-objectstore-tool cut down on large run
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
07e4273c6a test: ceph-objectstore-tool: Add test for EC object dump to check hinfo section
Fixes: http://tracker.ceph.com/issues/38053

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
786b39f18f test: ceph-objectstore-tool: Fix EC code handling so it doesn't skip EC objects
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
0753dc2b17 test: Remove unnecessary shell code that breaks this python test
Caused by: 8a694fc2f9

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
Patrick Donnelly
a823134c4d
Merge PR #26096 into master
* refs/pull/26096/head:
	mds: remove cache drop asok command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2019-01-26 16:51:20 -08:00
Patrick Donnelly
585624d86b
Merge PR #26087 into master
* refs/pull/26087/head:
	mds: fix MDSGather inheritance
	include: remove unnecessary inheritance

Reviewed-by: Sage Weil <sage@redhat.com>
2019-01-26 16:50:16 -08:00
Patrick Donnelly
03eb8d9e99
Merge PR #26083 into master
* refs/pull/26083/head:
	mds: trim cache after journal flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2019-01-26 16:48:58 -08:00
Patrick Donnelly
6858ba6625
Merge PR #26082 into master
* refs/pull/26082/head:
	client: flush cap release on session flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2019-01-26 16:48:07 -08:00
Sage Weil
6e933fe1b8 msg/async/AsyncMessenger: clear need_addr *after* we set our new addr
We check need_addr at the top without a lock held, so we need to be sure
we finished our work before we clear it, or else when there are two racing
threads the first will get the clock and clear the value and the second
will do nothing and see the unlearned value before the first finishes.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 17:01:14 -06:00
Igor Fedotov
086863832c os/bluestore: option to disable bluestore warns on spillover.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 22:22:09 +03:00
Igor Fedotov
4a18bdbf14 mon: expose object store health warnings
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 22:22:07 +03:00
Sage Weil
d9f3022a51 msg/async/ProtocolV2: fix handling for v2 client connection with v1 addr
Switch it to be v2.  Reject the case where the client sends and addrvec, though;
that should only happen for clients that did_bind, and they should only connect to
v2 if they have a v2 bound addr.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
4c2e28f5b7 ceph_test_msgr: do not connect_to on the client side
Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
985ec03cf9 msg/async: do not connect from server
We could have a fault on a server-side of a non-lossy connectoin where
there is a fault and we have outgoing data queued.  Since we are a server,
we cannot connect; we should just go into standby and wait for the other
end to reconnect, or for someone to mark us down.

This fixes a failure reproduced by Messenger/MessengerTest.SyntheticInjectTest/0
where it would assert(!policy.server) in the connect code.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
98a9a5e538 msg/async: do not use peer to addr detection; use getsockname()
If of relying on the peer to tell us what address we are connecting from,
look at how our local socket is bound, and use that address.

This removes the possibility for error because we will infer our address
locally and that will be the one place it is decide; the server will just
use our value.  As things were previously, we had to make the local and
remote inference match, which was fragile.

This does take away the client's ability to discover if it is traversing
NAT to reach the server and learning its public/external address.  I
don't think anybody has ever tested this, so it probably didn't even work,
and I've never heard it come up as a requirement.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Kefu Chai
8b55b47cc7 install-deps.sh: use latest setuptools
this change reverts "install-deps.sh: workaround setuptools' dependency on six"

This reverts commit 30ce5e5546.

tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.

long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 01:33:10 +08:00
Kefu Chai
16e3320789 osd/PG: remove unused functions
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 01:18:25 +08:00
Igor Fedotov
8f9e159735 osd: expose object store alerts
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:21:47 +03:00
Igor Fedotov
a7013a6754 os/bluestore: indicate BlueFS spillover and lacking compressor alerts
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:21:45 +03:00
Igor Fedotov
8c74ae6cf2 os:enable ObjectStore::statfs() api to return OS alert list.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:20:39 +03:00
Junhui Tang
061ace2eb7 mds: broadcast quota message to client when disable quota
When disable quota by setting quota.max_files or quota.max_bytes
to zero, client does not receive the quota broadcast message.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/38054
2019-01-26 15:20:31 +08:00
David Zafman
74239605a1
Merge pull request #23904 from xiexingguo/wip-27985
osd/PG: fix misused FORCE_RECOVERY[BACKFILL] flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
2019-01-25 16:44:22 -08:00
Yehuda Sadeh
c7a8c7f329 rgw: es: add support for != operator
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:47:11 -08:00
Yehuda Sadeh
d9978ffeac rgw: es: add support for searchable storage_class field
Note that this will not add field to existing index field mapping.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:47:11 -08:00
Yehuda Sadeh
ef91c36fba rgw: es: fix content_type queries
Fixes: http://tracker.ceph.com/issues/38030

ES attribute is stored as content_type, while we were searching
for contenttype.

Allow searching by contenttype, content_type, and also lastmodified,
and last_modified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Yehuda Sadeh
9f8bb33fa7 rgw: es: fix attrs trimming
Fixes: http://tracker.ceph.com/issues/38028

Since we don't necessarily keep attr headers with null terminating char,
need to be careful when trying to trim them.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Yehuda Sadeh
8c5218f0f0 rgw: http_client: propagate send_data_hint differently
Fixes: http://tracker.ceph.com/issues/36092

send_data_hint is passed so that the lower http client layer
to know in in cases where it's an http request that usually doesn't
send data (e.g., GET) that the request will actually send data,
and initialize the curl client correctly. In a case that was
used by the elasticsearch mechanism it wasn't passed correctly.
Now made it a data member in RGWHTTPClient, which also cleans
the interface a bit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
2bfe396906 rgw: ES sync module: drop ESVersion struct
using std::pair as the version struct. Another decoder class is currently
introduced as we have a nested json type for version right now.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
c6b6976735 rgw: es: support username and password for ES
For ES endpoints terminated with a username and password, either via xpack or
fronted by another webserver with http basic auth, we now support "username" and
"password" configurable which should be capable of doing HTTP basic
authentication

Fixes: https://tracker.ceph.com/issues/23655
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
21bb91e809 rgw_cr_rest: support extra headers for get resource crs
Adding support to add additional headers as a part of GET and READ requests.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
67d3cbd430 rgw: es: support for ES 5.0+
from Elasticsearch 5.0+ the type string is deprecated, while it is still allowed
in 5.0, it returns a 400 error on 6.0 clusters as this type is fully removed. We
now determine the es version while initializing the cluster from elasticsearch's
default endpoint and use that to determine what string type to use. This way we
support both 2.x and 5.x,6.x es versions as we default to string type for
clusters < 5.0

Fixes: http://tracker.ceph.com/issues/22877
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
5cc6ec21c1 rgw: es: introduce new classes for storing ES version info
Getting the basic information from elasticsearch, while we're only currently
interested in elasic search version, other data such as the ES cluster name etc
may be useful in the future as well.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Abhishek Lekshmanan
43e8ea43d5 rgw: rgw_cr_rest: allow headers to be specified in Put CRs
Adding another constructor overload for cases when we need to override the
default headers

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Abhishek Lekshmanan
2d4e442b09 rgw: es: introduce ES types
This commit introduces an enum mapping various ES types. Also encode_json now
has a new member for string type so that this can be changed easily when
upgrading ES versions for eg. wherein new ES engines do not support the string
type anymore.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Arthur Liu
8da2291e7d doc: fix typo in erasure coding example
Signed-off-by: Arthur Liu <arthurhsliu@gmail.com>
2019-01-26 08:44:42 +11:00
Sage Weil
2ca61bb460 msg/async/ProtocolV2: always send non-empty addrvec for self
If we don't know our address yet, send the peer a 0.0.0.0 or :: address with an empty
port and a populated nonce.  That way the peer can infer our final addr the same way
we do from learned_addr.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 14:17:34 -06:00
Sage Weil
5acf7ec1ff msg/async: never fill out port in myaddr if we didn't bind
If we are a client and didn't bind, then we should not fill in the port for our
address.  The one the peer sent us is just the random port our outgoing connection
happened to land on!

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 14:12:59 -06:00
Sage Weil
ea517edd2d ceph_test_msgr: use v2 addrs for simplemessenger
Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 13:48:11 -06:00