Commit Graph

98796 Commits

Author SHA1 Message Date
Casey Bodley
658e5932fb qa/rgw: rgw task can override --rgw-dns-name on the command line
the value for rgw_dns_name isn't known until a machine is assigned, so
it can't be set via 'override: conf:'. add a per-client config option
to the rgw task so it can add the endpoint's hostname and/or s3website
hostname on the radosgw command line

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 13:07:41 -04:00
Casey Bodley
0fc2c8ecee qa/rgw: remove ceph-ansible from s3a-hadoop suite
Fixes: http://tracker.ceph.com/issues/39706

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 13:07:41 -04:00
Casey Bodley
3ff5f0c240 qa/rgw: use default ports (80 or 443) unless overridden
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 13:07:41 -04:00
Casey Bodley
921faebb72 qa/rgw: allow rgw client config to override port
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 13:07:41 -04:00
Casey Bodley
dead71f5f9
Merge pull request #28146 from smanjara/wip-39551
rgw/multisite:RGWListBucketIndexesCR for data full sync pagination

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 12:53:09 -04:00
Casey Bodley
5a2661e48e
Merge pull request #26915 from dvanders/dvanders_doc_heap
doc: describe metadata_heap cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 12:51:13 -04:00
Kefu Chai
cf8c952a9b
Merge pull request #28359 from tchaikov/wip-split-ceph-dencoder
tools/ceph-dencoder: split types.h into smaller pieces

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2019-06-03 23:51:20 +08:00
Sage Weil
fd00136eb3 qa/suites/rados/multimon: skew clocks 2s (< paxos lease)
If the leader is the one with the accurate clock, it can still
form quorum, but if the leader has the skewed clock, all other mons appear
skewed from its perspective and no quorum is formed.  This leads to
intermittent failures, depending on the non-deterministic teuthology
deployment order and how the mon IPs sort.

Fix by reducing the skew.  This is enough skew to trigger a warning, but
not enough that it will break quorum.  This ensures that the parts of the
teuthology test that issue random mon commands won't fail (e.g., 'ceph osd
dump').

Fixes: http://tracker.ceph.com/issues/40112
Signed-off-by: Sage Weil <sage@redhat.com>
2019-06-03 10:49:02 -05:00
Dan van der Ster
e76604224c doc: describe metadata_heap cleanup
Fixes: http://tracker.ceph.com/issues/18174
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
2019-06-03 17:14:39 +02:00
Kefu Chai
8ceb11416d global/global_context: always add '\0' after strncpy()
see also f35fa1c58c

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 21:38:32 +08:00
Kefu Chai
11689c0e88 osd/ClassHandler: use std::variant for storing func
for better readablity, and less error-prone

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 21:21:52 +08:00
Casey Bodley
bdfc3bb92d
Merge pull request #28305 from rzarzynski/wip-bug-38827
qa/valgrind.supp: generalize the whiterule for aes-128-gcm to help rgw suite.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-06-03 09:09:24 -04:00
Casey Bodley
088d3ea26c
Merge pull request #25498 from cbodley/wip-27217
rgw: parse_copy_location defers url-decode

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-03 09:00:13 -04:00
Casey Bodley
62472c0ca3
Merge pull request #16659 from rzarzynski/wip-rgw-20797
rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-03 08:59:12 -04:00
Casey Bodley
8a0b2c7a67
Merge pull request #12709 from rzarzynski/wip-rgw-18364-dbgs
rgw: improve debugs on the path of RGWRados::cls_bucket_head.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-03 08:57:37 -04:00
Casey Bodley
eb51579ba4
Merge pull request #12470 from rzarzynski/wip-rgw-18241
rgw: dump s3_code as the Code response element in RGWDeleteMultiObj_ObjStore_S3

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
2019-06-03 08:57:13 -04:00
xie xingguo
a37106f0fa osd: maybe_remove_pg_upmaps -> clean_pg_upmaps
It should always be the preferred option to kill the unnecessary
or duplicated code, which is good for maintenance.
Also I've noticed there is already a clean_temps helper, so re-naming
maybe_remove_pg_upmaps to clean_pg_upmaps to at least keep pace with
that sounds to be a natural choice for me..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-03 17:13:50 +08:00
Kefu Chai
c63b33b61a cmake: link libkv against libheap_profiler
PriorityCache.cc uses `ceph_heap_*` functions, and libkv contains
`PriorityCache.cc`, so it should link against libheap_profiler.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 16:41:23 +08:00
Kefu Chai
7a9830323c osd/ClassHandler: no need to add struct before struct type
unlike C language, C++ is put struct names and other names in a single
namespace. so we don't need to add `strcut` before struct identifiers.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 16:03:02 +08:00
Kefu Chai
69611dcc54 osd/ClassHandler: use ceph::mutex instead of Mutex
for better performance

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 16:03:02 +08:00
Kefu Chai
9eef9a8cdd osd/ClassHandler: remove unused function
rook does not use "embedded" ceph anymore. and we've dropped the
embedded build support, so remove this function.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 16:03:01 +08:00
Kefu Chai
053906af9f osd/ClassHandler: s/string/std::string/ in header
and such-like, we cannot assume `using namespace std` or `using
std::string`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 16:03:01 +08:00
Kefu Chai
588f515a5c cmake: link libkv against common_prioritycache_obj
as both `kv/rocksdb_cache/BinnedLRUCache.cc` and
`kv/rocksdb_cache/ShardedCache.cc` uses `PriorityCache` which is in turn
offered by common_prioritycache_obj, we need to link libkv against the
latter.

libos uses `PriorityCache` as well, but since libos links against
libkv, so it's safe.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 15:12:44 +08:00
Kefu Chai
e7cf91ba16 cmake: remove kv_objs target
libkv is better than kv_objs, as it is able to bring more
properties with with it, for instance, the public include
directories, the public/private libraries it is linked against.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-03 14:52:52 +08:00
Kefu Chai
ed2ce0efad
Merge pull request #28339 from tchaikov/wip-cmake-asan
cmake/modules: replace ";" with " " in compile flags

Reviewed-by: Yingxin Cheng <yingxincheng@gmail.com>
2019-06-03 12:28:32 +08:00
xie xingguo
02e5499b35 osd/OSDMap: maybe_remove_pg_upmaps - avoid do_crush twice
which is extremely time-consuming.
Half of the amount of time of calling maybe_remove_pg_upmaps
has been saved by applying this patch as a result..

Was: maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s
Now: maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:56s

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-03 11:17:24 +08:00
xie xingguo
d9ed406a61 osd/OSDMap: maybe_remove_pg_upmaps - s/pg_to_raw_up/pg_to_raw_upmap/
The upmap results are directly applied after calling
_pg_to_raw_osds, which means it basically has nothing to do
with the up/down status.
In other words, if a pg_upmap/pg_upmap_items remapped a pg
into some down osds and is now causing collided result,
we should still be able to detect and cancel that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-03 11:06:29 +08:00
xie xingguo
c0ce22b8c8 test/osd: add performance test case for maybe_remove_pg_upmap
Tom Byrne reported that maybe_remove_pg_upmap might become
super inefficient for large clusters with balancer on.
To identify and resolve the problem, we need to add some good
measurements first.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2019-06-03 10:50:45 +08:00
Jan Fajerski
76204d32fb ceph.spec.in: install grafana dashboards world readable
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
2019-06-02 13:31:27 +02:00
Kefu Chai
eaaec3d319 tools/ceph-dencoder: split types.h into smaller pieces
instead of putting all types in a single compilation unit, split them
into smaller groups, in hope to reduce the memory footprint of compiler
when compiling ceph_dencoder.cc.

sometimes, GCC just fails to compile this source file.

Fixes: http://tracker.ceph.com/issues/39595
Signed-off-by: Kefu Chai <kchai@redhat.com>
2019-06-01 23:12:16 +08:00
Patrick Donnelly
880468b4bf
Merge PR #28354 into master
* refs/pull/28354/head:
	doc: Add warning that the root directory cannot be fragmented

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
2019-05-31 19:13:06 -07:00
Samuel Just
6d66072cc4 osd/: clarify rebuilt_missing_with_deletes and set in init
Updates PeeringState::init to set may_include_deletes_in_missing when
possible to avoid needlessly rebuilding the missing set on the next map.
The previous behavior was also subtly wrong as until the the next map
the osd would be possibly adding deletes to the missing set without the
flag having been set in the missing set on disk

This patch also renames the flag to reflect the fact that it simply
denotes wheter the may_include_deletes_in_missing omap entry needs to be
written out.  This may happen during rebuild_missing_set_with_deletes,
but it'll also happen more commonly during init now.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:08:48 -07:00
Samuel Just
8c86678ba7 crimson: hook PeeringState into pg
This patch:
1) Adds partially implemented stubs for PeeringState::PeeringListener
2) Revamps the message handling to batch the peering events from
   a single message.
3) Replaces the message handling with hand-offs to PeeringState
4) Adds support for MOSDPGCreate2 enabling creation of pgs.
5) Introduces ShardServices for exposing osd machinery to the pgs.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
909f4e260e crimson/osd/main: add --debug option
Really, what we need to do is examine the ceph conf and set
the right logger levels based on configured log levels, but
this'll suffice for now.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
1b22a8313a crimson/osd/pg_backend: clear exists in remove
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
8864de8164 cyan_store: map -ENOENT to 0 for _remove
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
c5fec5f7bd crimson/net/ProtocolV1: close connection when replaced
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
c489c53ce7 Revert "cmake: define BOOST_SP_DISABLE_THREADS for crimson"
For whatever reason, this patch appears to cause the reactor/peering
to hang much of the time.  Reverting until we have time to debug it.

This reverts commit a09a4f342c.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:45 -07:00
Samuel Just
395a2f6828 crimson/common/assert.cc: flush cout before abort
Seastar logger() instances don't flush the channel, so recent messages
won't be present otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
0c254023cf dout: let <= 20 correspond to debug for crimson
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
5c76f63078 vstart: add --redirect-output option to redirect output for --nodaemon
Useful for crimson which doesn't currently support writing directly
to a log file.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
e799b38aed crimson/osd: use configured ceph-osd uuid in mkfs
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
d0dbb36e2e crimson/net/Connection: add peer_is_* helpers
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
212593dac5 crimson/net/SocketMessenger: wait for accepter thread to die on shutdown
Otherwise, it ends up trying to operator<< *this after it's been
deleted.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
d07592eab7 crimson/mon/MonClient: handle mon client connection race
The first to establish a connection and close the others needs
to ensure that the others don't mess with broken state.  To that
end, leave a cleared pending_conns map while the conns are being
closed.  Also, take care to clean up outstanding promises and
fallout from conns disappearing.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
c839c2d42b crimson/os/cyan_store: run t.get_on_* contexts
PeeringState makes liberal use of those callbacks, run them
upon completion of do_transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
382a649fa0 crimson/os/cyan_store: add some omap machinery
Adds transaction methods for PeeringState.

Mainly omap_set, but also a stub for collection hint and the other omap
update methods.  Also adds omap_get and debugging to dump the
transaction upon encountering a problem.

Also fixes up std and ceph::bufferlist namespace qualifications.

Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
201dd35656 os/cyan_collection: add pragma once
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:05:44 -07:00
Samuel Just
b6e40a32cb osd/: move pg/osd state machinery into osd_types for use in crimson
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:04:32 -07:00
Samuel Just
fdd38eabd6 PGLog: add cyan_store machinery for reading the log
Signed-off-by: Samuel Just <sjust@redhat.com>
2019-05-31 14:04:28 -07:00