Commit Graph

94582 Commits

Author SHA1 Message Date
David Zafman
388e54d906 test: ceph-objectstore-tool cut down on large run
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
07e4273c6a test: ceph-objectstore-tool: Add test for EC object dump to check hinfo section
Fixes: http://tracker.ceph.com/issues/38053

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
786b39f18f test: ceph-objectstore-tool: Fix EC code handling so it doesn't skip EC objects
Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
David Zafman
0753dc2b17 test: Remove unnecessary shell code that breaks this python test
Caused by: 8a694fc2f9

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-01-26 19:35:11 -08:00
Patrick Donnelly
a823134c4d
Merge PR #26096 into master
* refs/pull/26096/head:
	mds: remove cache drop asok command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2019-01-26 16:51:20 -08:00
Patrick Donnelly
585624d86b
Merge PR #26087 into master
* refs/pull/26087/head:
	mds: fix MDSGather inheritance
	include: remove unnecessary inheritance

Reviewed-by: Sage Weil <sage@redhat.com>
2019-01-26 16:50:16 -08:00
Patrick Donnelly
03eb8d9e99
Merge PR #26083 into master
* refs/pull/26083/head:
	mds: trim cache after journal flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
2019-01-26 16:48:58 -08:00
Patrick Donnelly
6858ba6625
Merge PR #26082 into master
* refs/pull/26082/head:
	client: flush cap release on session flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
2019-01-26 16:48:07 -08:00
Sage Weil
6e933fe1b8 msg/async/AsyncMessenger: clear need_addr *after* we set our new addr
We check need_addr at the top without a lock held, so we need to be sure
we finished our work before we clear it, or else when there are two racing
threads the first will get the clock and clear the value and the second
will do nothing and see the unlearned value before the first finishes.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 17:01:14 -06:00
Igor Fedotov
086863832c os/bluestore: option to disable bluestore warns on spillover.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 22:22:09 +03:00
Igor Fedotov
4a18bdbf14 mon: expose object store health warnings
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 22:22:07 +03:00
Sage Weil
d9f3022a51 msg/async/ProtocolV2: fix handling for v2 client connection with v1 addr
Switch it to be v2.  Reject the case where the client sends and addrvec, though;
that should only happen for clients that did_bind, and they should only connect to
v2 if they have a v2 bound addr.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
4c2e28f5b7 ceph_test_msgr: do not connect_to on the client side
Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
985ec03cf9 msg/async: do not connect from server
We could have a fault on a server-side of a non-lossy connectoin where
there is a fault and we have outgoing data queued.  Since we are a server,
we cannot connect; we should just go into standby and wait for the other
end to reconnect, or for someone to mark us down.

This fixes a failure reproduced by Messenger/MessengerTest.SyntheticInjectTest/0
where it would assert(!policy.server) in the connect code.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Sage Weil
98a9a5e538 msg/async: do not use peer to addr detection; use getsockname()
If of relying on the peer to tell us what address we are connecting from,
look at how our local socket is bound, and use that address.

This removes the possibility for error because we will infer our address
locally and that will be the one place it is decide; the server will just
use our value.  As things were previously, we had to make the local and
remote inference match, which was fragile.

This does take away the client's ability to discover if it is traversing
NAT to reach the server and learning its public/external address.  I
don't think anybody has ever tested this, so it probably didn't even work,
and I've never heard it come up as a requirement.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-26 11:44:55 -06:00
Kefu Chai
8b55b47cc7 install-deps.sh: use latest setuptools
this change reverts "install-deps.sh: workaround setuptools' dependency on six"

This reverts commit 30ce5e5546.

tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.

long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 01:33:10 +08:00
Kefu Chai
16e3320789 osd/PG: remove unused functions
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-01-27 01:18:25 +08:00
Igor Fedotov
8f9e159735 osd: expose object store alerts
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:21:47 +03:00
Igor Fedotov
a7013a6754 os/bluestore: indicate BlueFS spillover and lacking compressor alerts
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:21:45 +03:00
Igor Fedotov
8c74ae6cf2 os:enable ObjectStore::statfs() api to return OS alert list.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
2019-01-26 17:20:39 +03:00
Junhui Tang
061ace2eb7 mds: broadcast quota message to client when disable quota
When disable quota by setting quota.max_files or quota.max_bytes
to zero, client does not receive the quota broadcast message.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/38054
2019-01-26 15:20:31 +08:00
David Zafman
74239605a1
Merge pull request #23904 from xiexingguo/wip-27985
osd/PG: fix misused FORCE_RECOVERY[BACKFILL] flags

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
2019-01-25 16:44:22 -08:00
Yehuda Sadeh
c7a8c7f329 rgw: es: add support for != operator
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:47:11 -08:00
Yehuda Sadeh
d9978ffeac rgw: es: add support for searchable storage_class field
Note that this will not add field to existing index field mapping.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:47:11 -08:00
Yehuda Sadeh
ef91c36fba rgw: es: fix content_type queries
Fixes: http://tracker.ceph.com/issues/38030

ES attribute is stored as content_type, while we were searching
for contenttype.

Allow searching by contenttype, content_type, and also lastmodified,
and last_modified.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Yehuda Sadeh
9f8bb33fa7 rgw: es: fix attrs trimming
Fixes: http://tracker.ceph.com/issues/38028

Since we don't necessarily keep attr headers with null terminating char,
need to be careful when trying to trim them.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Yehuda Sadeh
8c5218f0f0 rgw: http_client: propagate send_data_hint differently
Fixes: http://tracker.ceph.com/issues/36092

send_data_hint is passed so that the lower http client layer
to know in in cases where it's an http request that usually doesn't
send data (e.g., GET) that the request will actually send data,
and initialize the curl client correctly. In a case that was
used by the elasticsearch mechanism it wasn't passed correctly.
Now made it a data member in RGWHTTPClient, which also cleans
the interface a bit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
2bfe396906 rgw: ES sync module: drop ESVersion struct
using std::pair as the version struct. Another decoder class is currently
introduced as we have a nested json type for version right now.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
c6b6976735 rgw: es: support username and password for ES
For ES endpoints terminated with a username and password, either via xpack or
fronted by another webserver with http basic auth, we now support "username" and
"password" configurable which should be capable of doing HTTP basic
authentication

Fixes: https://tracker.ceph.com/issues/23655
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
21bb91e809 rgw_cr_rest: support extra headers for get resource crs
Adding support to add additional headers as a part of GET and READ requests.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
67d3cbd430 rgw: es: support for ES 5.0+
from Elasticsearch 5.0+ the type string is deprecated, while it is still allowed
in 5.0, it returns a 400 error on 6.0 clusters as this type is fully removed. We
now determine the es version while initializing the cluster from elasticsearch's
default endpoint and use that to determine what string type to use. This way we
support both 2.x and 5.x,6.x es versions as we default to string type for
clusters < 5.0

Fixes: http://tracker.ceph.com/issues/22877
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:57 -08:00
Abhishek Lekshmanan
5cc6ec21c1 rgw: es: introduce new classes for storing ES version info
Getting the basic information from elasticsearch, while we're only currently
interested in elasic search version, other data such as the ES cluster name etc
may be useful in the future as well.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Abhishek Lekshmanan
43e8ea43d5 rgw: rgw_cr_rest: allow headers to be specified in Put CRs
Adding another constructor overload for cases when we need to override the
default headers

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Abhishek Lekshmanan
2d4e442b09 rgw: es: introduce ES types
This commit introduces an enum mapping various ES types. Also encode_json now
has a new member for string type so that this can be changed easily when
upgrading ES versions for eg. wherein new ES engines do not support the string
type anymore.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2019-01-25 15:45:56 -08:00
Arthur Liu
8da2291e7d doc: fix typo in erasure coding example
Signed-off-by: Arthur Liu <arthurhsliu@gmail.com>
2019-01-26 08:44:42 +11:00
Sage Weil
2ca61bb460 msg/async/ProtocolV2: always send non-empty addrvec for self
If we don't know our address yet, send the peer a 0.0.0.0 or :: address with an empty
port and a populated nonce.  That way the peer can infer our final addr the same way
we do from learned_addr.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 14:17:34 -06:00
Sage Weil
5acf7ec1ff msg/async: never fill out port in myaddr if we didn't bind
If we are a client and didn't bind, then we should not fill in the port for our
address.  The one the peer sent us is just the random port our outgoing connection
happened to land on!

Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 14:12:59 -06:00
Sage Weil
ea517edd2d ceph_test_msgr: use v2 addrs for simplemessenger
Signed-off-by: Sage Weil <sage@redhat.com>
2019-01-25 13:48:11 -06:00
Ricardo Dias
1b097e8164
msg/async: msgr2: don't force write event on every message received
Signed-off-by: Ricardo Dias <rdias@suse.com>
2019-01-25 19:42:37 +00:00
Patrick Donnelly
fef88d9581
Merge PR #25254 into master
* refs/pull/25254/head:
	mds: optimize resuming stale caps
	client: avoid unnecessary wakeup when handling RENEWCAPS
	client: set cap->wanted when adding new cap
	client: don't wakeup cap waiters twice when mds recovered
	mds: optimize revoking stale caps
	mds: put notable caps at the front of session's caps list
	mds: track if client has writeable range in Capability
	mds: add session pointer to Capability
	client: skip updating 'wanted' caps if caps are already issued
	client: sync 'retain caps' logical from kernel client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-01-25 10:52:47 -08:00
Sebastian Wagner
2b9daafaf4
Merge pull request #25497 from jmolmo/wip_ansible_orchestrator
mgr/ansible: Create/Remove OSDs

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
2019-01-25 19:37:22 +01:00
Kefu Chai
5931341c2e
Merge pull request #25848 from majianpeng/debug-buffer-compiler-error
common/buffer: fix compiler bug when enable DEBUG_BUFFER.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-01-26 01:03:05 +08:00
Jan Fajerski
af9d0b1fe5 ceph-volume: strategies allow for external wal lv's
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-01-25 16:14:48 +01:00
Jason Dillaman
958b087f6b rados: return legacy address in 'lock info'
There is no need to differentiate v1 vs v2 addresses and we should
avoid breaking backwards compatibility in the output.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-01-25 09:12:15 -05:00
Kefu Chai
cc26a67890
Merge pull request #26022 from theanalyst/common-mclock-prio
common, osd: set mclock priority as 1 by default.

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
2019-01-25 22:07:17 +08:00
Kefu Chai
17d2fd5609
Merge pull request #26043 from tchaikov/wip-37975
osd: unmount store after service.shutdown()

Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-01-25 22:06:27 +08:00
Kefu Chai
eedc1f1a50
Merge pull request #26053 from Aran85/pipeline-fix
osd: ec saves a write access to the memory under most circumstances

Reviewed-by: Kefu Chai <kchai@redhat.com>
2019-01-25 22:05:58 +08:00
Jason Dillaman
a89a836acc mgr/MgrClient: reduce log level when issuing a command when not connected
The mgr client will eventually connect and issue the command so it's not
necessary to spam the logs at level 0.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-01-25 08:39:10 -05:00
Jason Dillaman
836d005fd7 pybind/rados: fixed Rados.create_ioctx2 type error
The pool name within the Ioctx was incorrectly stored as a
C-style string instead of a Python str object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-01-25 08:39:10 -05:00
Jason Dillaman
0d2f21cd66 pybind/mgr: exclude osd_perf_query module from being installed
This is a proof-of-concept example module that isn't expected to be
used in a production environment.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2019-01-25 08:39:05 -05:00