Commit Graph

107519 Commits

Author SHA1 Message Date
Guillaume Abrioux
f67610c73c cephadm: fix error handling in command_check_host()
`find_program()` raises `ValueError` when the executable hasn't been
found. It means we need to catch `ValueError` exception in
`command_check_host()` and raise `Error` instead of `RuntimeError` since
only `Error` is caught at the end.

Typical failure:

```
INFO:cephadm:/usr/bin/ceph:stderr Error ENOENT: New host mon1 failed check: ['INFO:cephadm:podman|docker (/bin/podman) is present', 'INFO:cephadm:systemctl is present', 'Traceback (most recent call last):', '  File "<stdin>", line 2820, in <module>', '  File "<stdin>", line 2434, in command_check_host', '  File "<stdin>", line 796, in find_program', 'ValueError: lvcreate not found']
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-02-03 18:43:39 +01:00
Abhishek Lekshmanan
949aa83ae5 rgw: move public access conf to perm_state_base
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
f88a48c6c0 test: rgw_iam_policy update tests with new Get Actions
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
bc8b8abda3 rgw: iam_policy: add all the actions to actpairs map
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
2831d4876b rgw: public access: drop unused function
Also cleanup the comment to mention why we've deviated from the spec here

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
97c57f05ee rgw: rename ACL & policy IsPublic to is_public
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
b7ddec14ac rgw: PublicAccessConfiguration -> PublicAccessBlockConfiguration
also drop iam namespace

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
2c98fa754d rgw_op: get_public_access_from_attr indent fixes
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
6fd6897bca rgw_rest_s3: use formatter->dump_bool
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
451b18e08f public_access: reuse formatter->dump_bool
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
e97b7d6426 rgw: Fix IgnorePublicACLs for bucket ACLs
Currently Bucket ACLs with IgnorePublicACLs were broken this should fix that

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:34 +01:00
Abhishek Lekshmanan
ff5cadbe8f drop redundant bucket policy status in rgw_common
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
065ecd3f43 rgw: implement IgnorePublicACLs
This allows for ignoring bucket/object acls that are configured to be public

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
ab745eae21 rgw: move PublicAccessConfiguration to req_state
This allows for evaluation of more complex use cases where IgnorePublicACLs and
the like are set which need to be evaluated for GET/HEAD requests as well

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
0c594c8b86 rgw: op: Get Policy Status checks if there is a policy first
before evaluating its public nature

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
c4bc1e8e75 rgw_op: reuse function to get public access conf
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
9fc16df7ed rgw: honor PublicAccessBlockConfiguration for put object
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
11f92eab4c rgw: block public access for Policies
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
f692d042d1 rgw: enforce BlockPublicPolicy on put bucket policy
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:33 +01:00
Abhishek Lekshmanan
065b5358f7 rgw: public_access: rename the getters to reflect what the conf actually does
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:32 +01:00
Abhishek Lekshmanan
f2e1840aab rgw_op: use ldpp_dout version wherever logs are required
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:32 +01:00
Abhishek Lekshmanan
c654e709a0 rgw: move IsPublic to RGWAccessControlPolicy class
This helps reusing when evaluating for PutACLs and Put Policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:32 +01:00
Abhishek Lekshmanan
2ba4a0f6f5 rgw: implement get/put/delete public access block for buckets
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:32 +01:00
Abhishek Lekshmanan
824b26c7ed add RGWPublicAccess for configuring public access settings for a resource
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:31 +01:00
Abhishek Lekshmanan
36bb77d8b7 rgw: both princ and nonprinc will not coexist in a single statement
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:31 +01:00
Abhishek Lekshmanan
ff972d6956 rgw: initial implementation of a public policy tester
doesn't cover all the cases involving a nonprinc user yet

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:31 +01:00
Abhishek Lekshmanan
e0b4562c61 rgw: s3: implement GetBucketPolicyStatus API
This API returns whether the Bucket Policies/ACLs are public. There are a couple
of caveats:
- AWS currently returns PolicyNotFound error in case a bucket policy doesn't
exist, though a non existant bucket policy would mean the default ACLs apply
where the bucket is private, so error return here seems like an error
- the API spec mentions TRUE and FALSE as the response IsPublic element value,
however in practice both boto/aws clients and AWS S3 return/expect a lowercase
response.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Conflicts:
	src/rgw/rgw_rest_s3.h
merge conflict after zipper rework, dropped a spurious newline in rgw_rest_s3.h
after get_obj_op decl.
src/rgw/rgw_common.h
src/rgw/rgw_rest_s3.cc
src/rgw/rgw_rest_s3.h:
merge conflict after bucket replication merge, trivial conflicts
2020-02-03 17:53:30 +01:00
Abhishek Lekshmanan
57baa84044 rgw: acl: drop unused function & make get_group_perm const
Drop the unused RGWAccessControlPolicy::get_group_perm, make the ACL
get_group_perm as a const member function

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 17:53:30 +01:00
Sage Weil
ebca44ccaa qa/suites/rados: improve valgrind leak check
Verify we can detect leak in the osd, mon, and mgr independently.  Also
include a negative test (no leaks).

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-03 10:25:39 -06:00
Sage Weil
e807669b77 common/ceph_context: add an asok command to deliberately leak memory
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-03 10:25:39 -06:00
Guillaume Abrioux
fed96416dc cephadm: increase default retry_max value
When playing with cephadm, at multiple times, I've reached the max
number of attempt in `is_available()`

Increasing the `retry_max` helps to avoid failure like following:

```
INFO:cephadm:mgr not available, waiting (1/5)...
INFO:cephadm:mgr not available, waiting (2/5)...
INFO:cephadm:mgr not available, waiting (3/5)...
INFO:cephadm:mgr not available, waiting (4/5)...
INFO:cephadm:mgr not available, waiting (5/5)...
ERROR: mgr not available after 5 tries

```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-02-03 17:25:06 +01:00
Casey Bodley
d486b5bc45 qa/rgw: test with non-default rgw-obj-stripe-size
each job will select one of the striping strategies at random

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-02-03 11:24:52 -05:00
Yuval Lifshitz
c5af4b3505
Merge pull request #32155 from yuvalif/wip-yuval-multidelete-notifications
rgw/pubsub: send notifications from multi-delete op
2020-02-03 18:02:06 +02:00
Sage Weil
1ad536ba40 mgr/cephadm: prefix daemon ids with hostname
This is friendlier to a human operator since they can immediately see
where an instance is located, as with the legacy scheme, while still
keeping the unique random suffix.  Use a . to separate so that we can
set per-host options.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-03 08:23:37 -06:00
xie xingguo
d08c1ff199 osd/OSD: prevent down osds from immediately rejoining the culster
In 114c65fc I posted a work-around to fix a heartbeat brain-split case
but it really looks to me now like I am missing some other cases where
an immediate attempt to rejoin is bad, like when the network actually
isn't working properly rather than being predictably manipulated by an
admin.
This patch instead slows the unconditionally rejoining attempt down,
especially make sure that we don't try to immediately rejoin the culster
when an osd has just been marked down by mon.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2020-02-03 21:35:05 +08:00
Sebastian Wagner
97fffb13fe
Merge pull request #32859 from liewegas/wip-cephadm-services
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-02-03 14:27:38 +01:00
Abhishek L
b6aa44cad5
Merge pull request #33030 from smithfarm/wip-luminous-12.2.13-release-notes
doc: release notes for Luminous v12.2.13

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 13:32:50 +01:00
Abhishek L
f2f868c107
Merge pull request #33031 from smithfarm/wip-nautilus-14.2.7-release-notes
doc: release notes for Nautilus 14.2.7

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
2020-02-03 13:32:05 +01:00
xie xingguo
59db3379ae osd/OSD: trim osd_markdown_log in tick() thread
so we don't have to do it in multiple places. Note that
we can't do it in the tick_without_osd_lock thread instead
because we we can not access it safely without the protection
of osd_lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2020-02-03 20:09:37 +08:00
Kefu Chai
80487f4a60
Merge pull request #33038 from tchaikov/wip-librados-reinterpret-cast
librados: prefer reinterpret_cast over c-style cast

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
2020-02-03 19:58:13 +08:00
Kefu Chai
0ac42dc7fd
Merge pull request #32933 from DimStar77/obs-mini
ceph.spec.in: Use pkgconfig() style BuildRequires for udev/libudev-devel

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-02-03 19:45:07 +08:00
Lenz Grimmer
4c9c4229ed
Merge pull request #32553 from tspmelo/wip-ts-strict
mgr/dashboard: Enable compiler options used by Angular --strict flag

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
2020-02-03 11:33:33 +00:00
Kefu Chai
5fcf0641cd librados: prefer reinterpret_cast over c-style cast
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-02-03 18:16:09 +08:00
Kefu Chai
7c1f7807d7
Merge pull request #33016 from tchaikov/wip-do-cmake
do_cmake.sh: bail out if something goes wrong

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2020-02-03 17:15:16 +08:00
xie xingguo
991354f17c osd/OSD: choose more heartbeat peers from different subtrees
We want to avoid a situation like:
- host.A consists of OSDs from 0 to 10
- cut off network of host.A from the rest of the cluster
- osd.1 is marked down when enough votes have been
  collected by mon
- osd.1 re-selects osd.0,2,3,..., and two extra
  osds from two different hosts as heartbeat peers
- osd.1 has more than 1/3 heartbeat peers becoming pingable,
  e.g., because they belongs to the same host.A, and will
  try to mark itself as up again

which as a result may cause a longer client op latency now.
Fix by (always) trying to select as many heartbeat peers
from different subtrees as possible instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2020-02-03 16:52:05 +08:00
Michael Fritch
4535216267
qa/workunits/cephadm/test_cephadm.sh: add missing monitoring tests
add tests for node-exporter, prometheus, and grafana

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-02-02 21:08:02 -07:00
Michael Fritch
c37504e81c
cephadm: simplify Monitoring.components structure
by removing the nested image/metadata dict

Signed-off-by: Michael Fritch <mfritch@suse.com>
2020-02-02 21:08:02 -07:00
Zac Dover
97df0b5274 More commits to the landing page project.
Signed-off-by: Zac Dover <zac.dover@gmail.com>
2020-02-03 11:21:34 +10:00
Yuval Lifshitz
ab26135f51 rgw/pubsub: fix tests to sync from master
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
2020-02-02 21:03:25 +02:00
Sage Weil
128095c219 cephadm: bootstrap: name mgr with $hostname.$random
Signed-off-by: Sage Weil <sage@redhat.com>
2020-02-02 09:43:13 -06:00