Commit Graph

38508 Commits

Author SHA1 Message Date
Yehuda Sadeh
44bc63b166 rgw: decode the req_state bucket instance id if needed
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:25 -08:00
Yehuda Sadeh
d31e84ea94 rgw: improve bucket sharding hashing
Amplify small source changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:25 -08:00
Yehuda Sadeh
a33ca59e26 rgw: data changes log, log info by bucket shard id
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:25 -08:00
Yehuda Sadeh
87934706fc rgw: use new BucketShard structure for index manipulation calls
Instead of recalculating the hash every call, do it once, and pass this
structure around. Also, will be used for logging changes into the data
log.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:25 -08:00
Yehuda Sadeh
381f68acde rgw: bi log list/trim can get specific bucket shard
bucket shard can be specified on the bucket instance param. It can be
added like this: <bucket-instance>[:shard-id]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:25 -08:00
Guang Yang
8a04c0a61b Fix the multipart uploads functional test failures due to bucket index sharding.
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>
2015-01-13 19:21:24 -08:00
Guang Yang
231fa0e670 Fix get_bucket_instance_info, only build the oid if it is empty.
Signed-off-by: Guang Yang <yguang@yahoo-inc.com>
2015-01-13 19:21:24 -08:00
Guang Yang
9e45a7cd36 Adjust bi log trim implementation to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:24 -08:00
Guang Yang
f9b280ea89 Adjust bi log listing to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:24 -08:00
Yehuda Sadeh
47665b23df cls_rgw, rgw: switch different ops to new concurrent infrastructure
Make all the relevant ops use the CLSRGWConcurrentIO infrastructure,
which simplifies things.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:24 -08:00
Yehuda Sadeh
30d0a49c84 rgw: generalize container type for concurrent IO base class
Turned the ConcurrentIO class a template, so that we could use different
kind of containers that are needed for the different operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:23 -08:00
Yehuda Sadeh
04441f2878 cls_rgw, rgw: create base class for common bucket shard operations
Instead of copy pasting the same code all over again, create a base
class for the needed concurrent IO operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-01-13 19:21:23 -08:00
Guang Yang
9c5acd67c4 Adjust bucket stats/index checking/index rebuild/tag timeout implementation to work with multiple shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:23 -08:00
Guang Yang
56feee792e Adjust bucket listing to work with multiple shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:23 -08:00
Guang Yang
751fd07bec Adjust rgw bucket prepare/complete OP to work with multiple bucket index shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:22 -08:00
Guang Yang
5d004d3eac Implement sharding for bucket creation.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:22 -08:00
Guang Yang
90a3920c44 Add a new field to bucket info indicating the number of shards of this bucket and make it configurable.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
2015-01-13 19:21:22 -08:00
Sage Weil
364b86813f mon/Paxos: consolidate finish_round()
Signed-off-by: Sage Weil <sage@redhat.com>
2015-01-13 14:51:22 -08:00
Sage Weil
67a90dd75c mon: accumulate a single pending transaction and propose it all at once
Previous we would queue lots of distinct encoded Transactions from various
callers, usually one per PaxosService.  These would be sent through paxos
one at a time.

If there is a completed transaction there is no reason to delay; it is
more efficient to push it through immediately.  Since we will propose
anything pending right when we finish, there is minimal opportunity for
other work to get done.

Instead, accumulate everything in a single MonitorDBStore::Transaction and
propose all pending changes all at once.  Encode at propose time and
expose the Transaction to the callers so they can add their changes.

Signed-off-by: Sage Weil <sage@redhat.com>
2015-01-13 14:51:04 -08:00
Sage Weil
d15958631b PendingReleaseNotes: make a note about librados flag changes
Signed-off-by: Sage Weil <sage@redhat.com>
2015-01-13 12:23:37 -08:00
Sage Weil
5a1fd855df Merge pull request #3360 from mattrichards/bump_rados_version
librados: bump rados version number

Reviewed-by: Sage Weil <sage@redhat.com>
2015-01-13 12:18:04 -08:00
Jenkins
725d66098c 0.91 2015-01-13 12:10:22 -08:00
Josh Durgin
6f8b54ca29 Merge pull request #2697 from ceph/wip-8900
RBD image watcher and new exclusive lock handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2015-01-13 11:17:29 -08:00
Samuel Just
b8ce73f253 Merge pull request #3254 from trociny/feature-10036
osd: osd tree to show primary-affinity value

Reviewed-by: Loic Dachary <ldachary@redhat.com>
2015-01-13 10:56:29 -08:00
Samuel Just
6c4a523c36 Merge pull request #3281 from ceph/wip-10441-b
osd: fix watch ordering bug 10441 option b

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2015-01-13 10:55:29 -08:00
Samuel Just
00c30dd0d4 Merge pull request #3290 from ceph/wip-da-SCA-20150102
Coverity and SCA fixes

Reviewed-by: Sage Weil <sage@redhat.com>
2015-01-13 10:54:45 -08:00
Samuel Just
733ce25ccb Merge pull request #3302 from ceph/wip-9956
os/FileStore: verify kernel is new enough before using extsize ioctl

Reviewed-by: Samuel Just <sjust@redhat.com>
2015-01-13 10:54:21 -08:00
Samuel Just
c82f8177f7 Merge pull request #3305 from majianpeng/fix5
fix bugs about sync_filesystem

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2015-01-13 10:53:34 -08:00
Sage Weil
9264d2508a common/Formatter: new_formatter -> Formatter::create
Also make the default formatter selection more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>
2015-01-13 08:32:45 -08:00
Sage Weil
617ad5ddc1 common/Formatter: improve json-pretty whitespace
The whitespace for the current pretty mode is awkward and weird.  Move
to a more standard approach.  It's more newlines, but *much* more
readable.

Signed-off-by: Sage Weil <sage@redhat.com>
2015-01-13 08:20:08 -08:00
Sage Weil
83c3b1305e common/Formatter: add newline to flushed output if m_pretty
This applies to json-pretty and xml-pretty modes.

Signed-off-by: Sage Weil <sage@inktank.com>
2015-01-13 08:06:17 -08:00
Gregory Farnum
a6f5682d55 Merge pull request #3364 from ceph/wip-quota-test
qa: set -e explicitly in quota test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2015-01-13 07:08:30 -08:00
John Spray
b578a53b03 qa: set -e explicitly in quota test
Previously was set in hashbang, which meant
that "./quota.sh" was OK, but "sh ./quota.sh" would
just run through ignoring errors.

Signed-off-by: John Spray <john.spray@redhat.com>
2015-01-13 14:58:57 +00:00
Gregory Farnum
d8e1f675f6 Merge pull request #3336 from ceph/wip-fs-reset
mon: implement `fs reset`

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
2015-01-13 06:47:04 -08:00
Frank Yu
7c664fadff Doc: Fix the indentation in doc/rbd/rbd-snapshot.rst 2015-01-13 18:27:08 +08:00
Loic Dachary
fc7fa3087d Merge pull request #3343 from dachary/wip-10505-centos-parted
tests: install parted in centos Dockerfile

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>
2015-01-13 11:07:55 +01:00
Jianpeng Ma
9ef6ce5647 FileJournal: check fsync/fdatasync result.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2015-01-13 15:51:51 +08:00
Jianpeng Ma
3cbe5da1c2 mon: check size must larger than zero.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2015-01-13 14:27:36 +08:00
Jianpeng Ma
3f0300358b test/mon: Add test case for ceph osd pool set size/min_size.
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2015-01-13 14:27:36 +08:00
Jianpeng Ma
e3678f41e3 mon: check min_size range.
When set min_size value:
A:for replicated pool, the value must be between in 1 and size
B:for erasure pool, the value must be between in data_chunk_count)
 and size(data_chunk_count + code_chunk_count).

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
2015-01-13 14:27:36 +08:00
Jason Dillaman
7945f8d688 librbd: flush pending AIO requests under all existing flush scenarios
AIO requests that are waiting on the image lock should be flushed
during all existing RBD flush scenarios.  A few flush cases were
missed in the original implementation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 23:17:50 -05:00
Jason Dillaman
2dd0f038d6 librbd: AIO requests should retry lock requests
Added a timer to support retrying AIO lock requests until
they are successful.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 23:14:11 -05:00
Jason Dillaman
6176ec5fde librbd: differentiate between R/O vs R/W RBD features
The new RBD exclusive lock feature should be treated as a
feature that is only applied when the image is opened in
R/W mode.

Older clients will need to handle the updated
cls_rbd::get_features method in order to properly determine
the incompatible features for an image depending on the
current mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:02:25 -05:00
Jason Dillaman
544ed961c6 librbd: Add internal unit test cases
The new unit tests cover the modifications made to integrate
the internal librbd functionality with the new ImageWatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:02:24 -05:00
Jason Dillaman
17f22d98df librbd: Add ImageWatcher unit test cases
Directly unit test the new ImageWatcher class to complement
the existing librbd integration tests of exclusive lock
handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:02:06 -05:00
Jason Dillaman
5160281b1a librbd: Add convenience library to support unit tests
Unit tests need access to the private symbols of librbd no
longer exported from librbd.so.  A new librbd_internal
convenience library was created to allow access.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:01:21 -05:00
Jason Dillaman
4297e3bf8e rbd: Allow CLI to optionally create shared images
Images that are flagged as shared cannot use the RBD
object map nor RBD mirroring features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:01:21 -05:00
Jason Dillaman
ccadff1445 librbd: Integrate librbd with new exclusive lock feature
Operations that update the image now require the exclusive lock
if the feature is enabled.  AIO write and discard operations will
automatically request the exclusive lock from the current leader
to support live-migration.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-01-12 20:01:19 -05:00
Matt Richards
9ee80b3322 librados: bump rados version number
As a follow-on to 49d114f1ff,
increment the "extra" version field so clients can easily
determine if they have a version of librados that properly
translates C API operation flags.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>
2015-01-12 17:00:52 -08:00
Josh Durgin
d784bc47c4 Merge pull request #3316 from ceph/wip-10471
rgw: index swift keys appropriately

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2015-01-12 16:20:28 -08:00