RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-01-01 08:32:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
73,136 Commits 212 Branches 394 Tags 1.5 GiB
0806a00514
Commit Graph

73136 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yehuda Sadeh
de9d2cb482 rgw-admin: zone create handles --tier-config 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:54 -07:00
Yehuda Sadeh
a680aa11a3 rgw: es: use "null" as instance in case key.instance is empty 
when indexing objects. We can get either.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:54 -07:00
Yehuda Sadeh
2c9a7bc993 rgw: es: generic keys are not case sensitive 
make map case insensitive, and explicitly define all generic keys
so that they can be forced to be lower case.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:53 -07:00
Yehuda Sadeh
4a91a542e9 rgw: fix time format that is sent to elasticsearch 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:53 -07:00
Yehuda Sadeh
3f40938125 rgw: add missing custom meta map initialization 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:53 -07:00
Yehuda Sadeh
8bfe1379a1 rgw: es: configurable number of replicas and shards 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:53 -07:00
Yehuda Sadeh
fbae80ba50 rgw: es: add a configurable to override index path 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:52 -07:00
Yehuda Sadeh
f5b158fcbd rgw: es: elasticsearch index path unique per sync instance 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:52 -07:00
Yehuda Sadeh
151a6eba60 rgw: es: configurable list of buckets and owners to index 
and a tier configuration that allows to select both list of buckets
and list of bucket owners (if both configured then will intersect
the selection).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:52 -07:00
Yehuda Sadeh
b02d6162e4 rgw: create helper for configurable item selection 
with optional prefix and suffix.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:02 -07:00
Yehuda Sadeh
85310f0e8f rgw: add "explicit_custom_meta" configurable to es sync module 
Modify all the map<string, string> that is used to pass in module config
to map<string, string, ltstr> so that it can be used with the conf get
vals util. Also, switch to using shared_ptr to hold the ElasticConfig,
so that later when we hold maps and such it doesn't need to be copied
on every call.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:26:02 -07:00
Yehuda Sadeh
6d5555accd rgw: expose conf_get(), conf_get_int(), conf_get_bool() 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:45 -07:00
Yehuda Sadeh
f1e681bf83 common/ceph_json: map type can handle 2 or 3 template arguments 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:45 -07:00
Yehuda Sadeh
874223b251 rgw: es: handle get config api in es zone 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:45 -07:00
Yehuda Sadeh
603863a46e rgw: es: api to delete bucket mdsearch config 
DELETE /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:44 -07:00
Yehuda Sadeh
acba916125 rgw: es: api to retrieve bucket mdsearch config 
GET /bucket?mdsearch

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:44 -07:00
Yehuda Sadeh
7b61c992b7 rgw: es: also return custom int and date results 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:44 -07:00
Yehuda Sadeh
e4e6e1e51c rgw: es: limit indexing to only configured user meta 
only metadata keys that are configured on user's bucket are allowed
to be indexed. Also, use the type of meta that was configured.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:44 -07:00
Yehuda Sadeh
8249f76b80 rgw: new api to configure bucket's custom keys for mdsearch 
POST /bucket?mdsearch
x-rgw-meta-search: <x-amz-meta-key>[;<str|int|date>][,...]

note that x-amz-meta-search header can also be used.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:43 -07:00
Yehuda Sadeh
3f62f17f94 rgw: guard restricted fields in es query 
don't allow users to use the 'permissions' field.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:43 -07:00
Yehuda Sadeh
de5bd34eb2 rgw: add dout defs 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:43 -07:00
Yehuda Sadeh
484fd2b8cb rgw: support field aliases in es compiler 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:43 -07:00
Yehuda Sadeh
21ee7c11b5 rgw: support more meaningful compilation error string 
in es module

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:42 -07:00
Yehuda Sadeh
42746ac9d2 rgw: support for generic entities types in es 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:42 -07:00
Yehuda Sadeh
d026a23d1f rgw: initial support for multiple types in es requests 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:42 -07:00
Yehuda Sadeh
1e886475bb rgw: metadata search pagination 
Leveraging elasticsearch "size" and "from" for doing pagination. This
is not optimal, but sadly it's currently the way to go as elasticsearch
does not have an efficient way to retrieve ordered entries (other than
using stateful api). This should be revisited in the future. A preferable
way would be to hold a unique identifier for each object doc, and have
the query sort by that unique identifier (that could be used as marker).
At the moment sorting by anything would require elasticseatch to load
all data into memory.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:42 -07:00
Yehuda Sadeh
99b49acc2e rgw: add permissions and bucket to es query 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:41 -07:00
Yehuda Sadeh
0180650d09 rgw: also dump custom meta entries 
when doing metadata search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:41 -07:00
Yehuda Sadeh
86fe1223c9 rgw: send request to elasticsearch and parse response 
and send back response to user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:41 -07:00
Yehuda Sadeh
698763fb13 rgw: extend interfaces to allow sending data 
when retrieving REST resource

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:41 -07:00
Yehuda Sadeh
b6fe05d9a4 rgw: move code around 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:40 -07:00
Yehuda Sadeh
952e72d979 rgw: can send data in RGWRESTStreamRWRequest::send_request() 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:40 -07:00
Yehuda Sadeh
9f494ff473 rgw: make key param in RGWRESTStreamRWRequest::send_request() optional 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:40 -07:00
Yehuda Sadeh
c9d6882275 rgw: rename a few methods 
just rename calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:40 -07:00
Yehuda Sadeh
e515bf3b15 rgw: meta search rest handler can access es module 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:39 -07:00
Yehuda Sadeh
bb46f19e44 rgw: work on REST handler for es module 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:39 -07:00
Yehuda Sadeh
42130df362 rgw: rename rgw_rest_es.cc to rgw_es_query.cc 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:38 -07:00
Yehuda Sadeh
0fdd24ff80 rgw: simplify es compile interface 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:38 -07:00
Yehuda Sadeh
86ea424313 rgw: move code into class 
just cleaning up

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:38 -07:00
Yehuda Sadeh
6de586ca74 rgw: handle nested fields in es queries 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:38 -07:00
Yehuda Sadeh
67bfdb4541 rgw: implement init_sync() callback in es module 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:24:37 -07:00
Yehuda Sadeh
49b27f4f90 rgw: define es index mapping 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:18:58 -07:00
Yehuda Sadeh
6d4733c58b rgw: add init callback to sync modules 
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:18:58 -07:00
Yehuda Sadeh
8fd55cc2e0 rgw: initial implementation of mdsearch query compiler 
convert infix queries that look as follows:

[(]<name> <operator> <value> [)] [<and|or> ...]

into a prefix structure that is understood by elasticsearch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:17:18 -07:00
Yehuda Sadeh
760c5e4f86 Merge pull request #15184 from cbodley/wip-qa-rgw-cleanup 
qa/rgw: remove apache/fastcgi and radosgw-agent tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
 2017-05-30 13:09:31 -07:00
Casey Bodley
6049d97740 Merge pull request #13851 from zhangsw/optimize-datasync 
rgw: optimize data sync. Add zones_trace in log to avoid needless sync.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
 2017-05-30 14:16:07 -04:00
David Zafman
1dbb15d340 Merge pull request #15368 from dzafman/wip-20089 
osd: When scrub finds an attr error mark shard inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>
 2017-05-30 10:46:23 -07:00
Sage Weil
e88cccf969 Merge pull request #15350 from mrbojangles3/master 
libradosstriper: Add example code
 2017-05-30 12:32:06 -05:00
Sage Weil
3f99839b41 Merge pull request #15356 from joscollin/wip-test-msgr-warning 
test/msgr: silence warnings from -Wsign-compare
 2017-05-30 12:29:51 -05:00
Sage Weil
b38495c84e Merge pull request #15362 from andihit/doc-rados-xattr-commands 
doc: add rados xattr commands to manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>
 2017-05-30 12:28:34 -05:00