Commit Graph

55931 Commits

Author SHA1 Message Date
xie xingguo
af82dbcbeb os/bluestore: add support for allocating 0-sized extent
For testing, we may choose randomized length of extent to allocate,
which means we could trim down the length to 0.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-10 21:25:52 +08:00
xie xingguo
39458dd308 os/bluestore: fix allocation of discontiguous blocks
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-10 21:25:50 +08:00
xie xingguo
6147ea0f29 os/bluestore: add sanity check for get_free() method
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-10 21:25:49 +08:00
xie xingguo
0c86d5ec0f os/bluestore: simplify is_exhausted() method
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-10 21:25:47 +08:00
xie xingguo
b7d09ed137 os/bluestore: make zone/span size of bitmap-allocator configurable
Perviously as zone_size_block and span_size are identical, so we could use
span_size only to simplify the calcuation.

Now that the zone_size_block and span_size are both configurable, so
it is necessary to treat them respectively.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-10 21:25:45 +08:00
Kefu Chai
89ba840a6c Merge pull request #10190 from ktdreyer/doc-delete-pool
doc: more details for pool deletion

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-10 19:57:07 +08:00
Brad Hubbard
3ad56cbf61 makefile: change librgw_file_* as check_PROGRAMS
These targets depend on libgmock_main.la which is not built during
a default build.

Fixes: http://tracker.ceph.com/issues/16646
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
2016-07-10 16:42:19 +10:00
Michal Jarzabek
72f75d9f63 msg/Event.h: add const to member function
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>
2016-07-09 08:38:20 +01:00
Josh Durgin
becdbe2917 ReplicatedPG: clear log update waiters during shutdown
This prevents leaking repops that are referenced by LogUpdateCtx for
updates that were in flight.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:14 -07:00
Josh Durgin
fbf3b79cae test/librados: add test that requires correct dup error detection
This only works reliably with the
objecter_retry_writes_after_first_reply setting, so make it part of
the test setup.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:14 -07:00
Josh Durgin
56f7115632 test/librados: add a way to pass ceph config options
This way individual tests or testcases can change settings

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:14 -07:00
Josh Durgin
6faa449fc6 Objecter: add option for testing osd dup handling
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:14 -07:00
Josh Durgin
fa8aff9a25 ReplicatedPG: require kraken feature bit on osdmap to record write errors in the pg log
Older OSDs can't handle the error entries.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:14 -07:00
Josh Durgin
112eab06fb OSDMonitor: add kraken feature bit
This is needed to turn on persisting write errors, since older OSDs
won't be able to handle them.

Other features for kraken could potentially use this as well.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
4c709e9b6d ReplicatedPG: return error codes for dups from the pg log
This prevents reordering guarded writes or deletes.

Without this, the following sequence:

delete foo -> -ENOENT
write foo -> success
(client connection fails)
resend delete foo -> success, object deleted
resend write foo -> success - dup op, so no write performed

results in the object not existing, instead of containing data. After
this change, both delete and write are detected as dups and the
original ordering is preserved.

Fixes: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
ed33ffcbd3 ReplicatedPG: skip stat invalidation when recording write errors
This is only needed for the lost/unfound use of submit_log_entries() etc.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
cf8e466f34 ReplicatedPG: persist write errors in the pg log
This is required to prevent re-ordering of guarded writes or deletes
in the presence of network failures and resends.

Use the existing submit_log_entries() method to initiate a repop that
only updates the pg log.

Keep the write error semantics close to the existing implementation -
if we have a buffer, return it, but do not persist the buffer for now.

Refs: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
41861fae08 PGLog: skip indexing errors by object
Dup detection only needs them indexed by version, and keeping them out
of the object index prevents error entries from contributing to the
missing set during recovery.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
ea66e5ca4d PGLog: ignore error entries when constructing the missing set
Errors should only be used for dup detection.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
2b8062ebc1 osd_types: add an error pg log entry type
This will store write error codes for use in dup op detection.
A few places use checks assuming is_update() or is_delete() are
opposites - fix those to ignore or consider errors, as appropriate.

Refs: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
b675726c0e ReplicatedPG: removed unused obc check
The one place this was set was removed by e7edf20fb7

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:13 -07:00
Josh Durgin
62864beebd ReplicatedPG: removed unused OnComplete struct
Added but never used in e7edf20fb7

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:12 -07:00
Josh Durgin
2609dd922f ReplicatedPG: remove unused mark_object_lost()
The only caller was removed in e7edf20fb7

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 18:33:12 -07:00
Sage Weil
181f6c67da Merge pull request #10201 from xiexingguo/xxg-wip-bluestore-2016-07-08
os/bluestore: misc cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2016-07-08 18:29:50 -04:00
Casey Bodley
0273d5595c Merge pull request #10157 from cbodley/wip-rgw-sync-error-repo
rgw: fix error_repo segfault in data sync
2016-07-08 16:12:26 -04:00
Casey Bodley
28609029cf rgw: fix error_repo segfault in data sync
RGWDataSyncShardCR will only allocate an error_repo if it's doing
incremental sync, so RGWDataSyncSingleEntryCR needs to guard against a
null error_repo

also, RGWDataSyncShardCR::stop_spawned_services() was dropping the last
reference to the error_repo before calling drain_all(), which meant that
RGWDataSyncSingleEntryCR could still be holding a pointer. now uses a
boost::intrusive_ptr in RGWDataSyncSingleEntryCR to account for its
reference

Fixes: http://tracker.ceph.com/issues/16603

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 15:53:57 -04:00
vasukulkarni
299a5210e3 Merge pull request #10184 from ktdreyer/doc-rados-operations-systemd
doc: rm SysV instructions, add systemd
2016-07-08 11:56:50 -07:00
Yehuda Sadeh
ed1be58a86 Merge pull request #9570 from zaitcev/wip-5073-multen-doc
rgw: Add documentation for the Multi-tenancy feature

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2016-07-08 11:46:48 -07:00
Yehuda Sadeh
6489be299c Merge pull request #10151 from oritwas/wip-rgw-fix-cls-sync-stats
rgw: register the correct handler for cls_user_complete_stats

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2016-07-08 11:40:01 -07:00
Daniel Gryniewicz
395f2c5ffe CMake - stop pip checking for updates
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
2016-07-08 13:41:27 -04:00
Casey Bodley
16976eedb7 rgw: remove datalog keys from error repo on ENOENT
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
6f65d192cd test/rgw: add optional --tenant flag to test_multi.py
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
3cf715c682 rgw: add tenant id to GetBucketInstanceInfo
use the rgw_bucket overload of get_bucket_instance_info() so it can
supply the tenant id

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
d945e20995 rgw: carry tenant id with data sync
use rgw_bucket_shard to track buckets, which includes tenant id

Fixes: http://tracker.ceph.com/issues/16469

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
2a1e1a776c rgw: add tenant to url for RGWRESTStreamWriteRequest
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
fc0df4802b rgw: add tenant to rgw_bucket json format
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:26 -04:00
Casey Bodley
d5ac140040 rgw: add rgw_bucket_parse_bucket_key()
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:25 -04:00
Casey Bodley
c8aa668fc3 rgw: add tenant name to datalog entries
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:25 -04:00
Casey Bodley
3eae201c8a rgw: convert bucket instance listings back to metadata key format
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:25 -04:00
Casey Bodley
a0befa1e7b rgw: use tenant/ for bucket instance metadata keys
to work around the ambiguity of parsing tenant: and :shard in the same
bucket instance metadata key, use tenant/ instead

to preserve backward compatibility with existing objects, new helper
function rgw_bucket_instance_key_to_oid() converts this / back to a :
before being used as an object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:25 -04:00
Casey Bodley
51ff8ef7c6 rgw: add get_key() methods to format rgw_buckets
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2016-07-08 12:20:25 -04:00
Kefu Chai
21402181f5 Merge pull request #9934 from tchaikov/wip-build-on-armf
common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>

Reviewed-by: Samuel Just <sjust@redhat.com>
2016-07-08 22:58:28 +08:00
Kefu Chai
5ba22025a8 Merge pull request #10204 from ceph/wip-cmake
cmake: fixes for pypi changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2016-07-08 22:51:48 +08:00
Orit Wasserman
e1bd0118ad rgw: remove unused realm from radosgw-admin zone modify
Fixes:http://tracker.ceph.com/issues/16632

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
2016-07-08 16:20:32 +02:00
Ken Dreyer
3c43185002 doc: more details for pool deletion
Prior to this change, the documention instructed administrators to
delete rulesets and users, but did not go into details regarding how to
do that.

Add example commands that admins may use to search for rulesets and
users that might reference the to-be-deleted pool.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2016-07-08 07:42:29 -06:00
Ilya Dryomov
ccc834d3d8 AsyncConnection: don't limit recv_max_prefetch with TCP_PREFETCH_MIN_SIZE
simple messenger's prefetch buffer is ms_tcp_prefetch_max_size.
Don't cap it with TCP_PREFETCH_MIN_SIZE in the async messenger.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2016-07-08 15:36:32 +02:00
Ilya Dryomov
f6e7e40394 AsyncConnection: don't use state_buffer for connect authorizer
state_buffer is 4096 bytes long, while connect authorizer can be
arbitrarily big - an OSD service ticket with lots of per-pool or
per-namespace caps is one example.  We end up scribbling on invalid
memory past the state_buffer and eventually crash.

Move authorizer_bl up into connection and read directly into it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
2016-07-08 15:34:30 +02:00
Kefu Chai
31db4c5f9f common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>
this fixes the build on armf.

on 32bit platforms, cstdint is very likely to

 typedef long long int int64_t;

this results in compilation error like

 `common/strtol.cc:190:75: error: duplicate explicit instantiation of 'T
 strict_si_cast(const char, std::string) [with T = long long int;
 std::string = std::basic_string]'

 [-fpermissive]
 template int64_t strict_si_cast(const char *str, std::string *err);
 ^`

we can address this by instantiate the primitive type of `long long`
instead of `in64_t`.

Fixes: http://tracker.ceph.com/issues/16398
Signed-off-by: Kefu Chai <kchai@redhat.com>
2016-07-08 21:20:53 +08:00
Kefu Chai
37dbe98a31 Merge pull request #8189 from chardan/wip-chardan-15100
librados examples: link and include from current source tree by default.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2016-07-08 21:10:35 +08:00
xie xingguo
ff0329634a os/bluestore: release compressor if comp_mode turned out to be none
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
2016-07-08 20:44:42 +08:00