Commit Graph

81634 Commits

Author SHA1 Message Date
Kefu Chai
4e621762ed qa/standalone/ceph-helpers.sh: silence ceph-disk DEPRECATION_WARNING
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-13 19:42:50 +08:00
John Spray
ab3faa446e
Merge pull request #19229 from wido/mgr-influx
mgr/influx: PEP-8 and other fixes to Influx module

Reviewed-by: John Spray <john.spray@redhat.com>
2017-12-13 11:35:17 +00:00
Wido den Hollander
0333f74953
ceph-volume: Try to cast OSD metadata to int while scanning directory
By doing so values like 'whoami' and 'bluefs' will be stored as a
integer in the resulting JSON rather then a String.

Signed-off-by: Wido den Hollander <wido@42on.com>
2017-12-13 11:20:53 +01:00
Jos Collin
9081d862d2
Merge pull request #19472 from qrGitHub/wip-rgw-typo-woild
rgw: fix typo woild to would

Reviewed-by: Jos Collin <jcollin@redhat.com>
2017-12-13 09:39:36 +00:00
Bingyin Zhang
daf6a1330b rgw: correct typo woild to would
Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>
2017-12-13 16:38:27 +08:00
Bingyin Zhang
428f21c13a rgw: remove redundant S3AnonymousEngine
* When receives unauthorized requests, RGW(rados auth only) runs four
* engines as follows:
*     1. anonymous engine in s3_main_strategy_plain
*     2. local engine in s3_main_strategy_plain
*     3. anonymous engine in s3_main_strategy_boto2
*     4. local engine in s3_main_strategy_boto2
* The third engine which is exactly the same with the first one is
* redundant.

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>
2017-12-13 16:30:23 +08:00
Kefu Chai
b6c3e9968d msg/async: update to work with dpdk shipped with spdk v17.10
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-13 14:55:15 +08:00
Kefu Chai
4231cfbc99 cmake: DPDK works w/o CRYPTOPP
Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-13 14:54:17 +08:00
myoungwon oh
d70b71aa2f osd: fix unordered read bug (for chunked object)
The current implementation for chunked object only supports
proxy_read(if offset is within range) and write(local write)
In this case, a read request can be handled before a write request.
This commit prevents unordered read processing because
proxy_read() will be executed if the chunk is missing state.
If chunked object has been overwritten, its state will not be missing.

Fixes: http://tracker.ceph.com/issues/22369
Signed-off-by: Myoungwon Oh <omwmw@sk.com>
2017-12-13 15:48:33 +09:00
Kefu Chai
b69e3efe64 install-deps.sh: avoid re-installing g++-7
* add ubuntu-toolchain-r mirrors in case the ppa.launchpad.net is not
  accessible
* add ppa repo manually, it's faster than installing
  `software-properties-common` and then launch `add-apt-repository`
* hardwire $old to the gcc version shipped by the distro, simpler this
  way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2017-12-13 14:22:41 +08:00
Shinobu Kinjo
8cef8447a9 osdc: Remove a bit too redundant public label...
Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>
2017-12-13 14:59:22 +09:00
Patrick Donnelly
e8c6882cc9
memstore: write fsid to fsid not fs_fsid
Fixes: http://tracker.ceph.com/issues/20736

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2017-12-12 21:07:29 -08:00
Haomai Wang
5216309c25 msg/async/AsyncConnection: unregister connection when racing happened
Signed-off-by: Haomai Wang <haomai@xsky.com>
2017-12-13 09:28:47 +08:00
vasukulkarni
36aa1b8c94
Merge pull request #17651 from ceph/wip-ceph-deploy-upgrade
qa/tests: Wip ceph deploy upgrade
2017-12-12 16:10:47 -08:00
Wido den Hollander
392603a881
ceph-volume: Add linesep/newline at end of JSON file when writing
Instead of:

    "systemd": "",
    "type": "bluestore",
    "whoami": "0"
  }root@alpha:~#

A newline is added to the JSON file when writing:

    "type": "bluestore",
    "whoami": "0"
  }
  root@alpha:~#

Makes it a bit easier to read the JSON files on a terminal

Signed-off-by: Wido den Hollander <wido@42on.com>
2017-12-12 21:27:07 +01:00
Sage Weil
86dc162686 qa/standalone/ceph-helpers: pass --verbose to ceph-disk
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-12 12:56:45 -06:00
Mark Kogan
a6a1b664d3 rgw: fix chained cache invalidation to prevent cache size growth
above the rgw_cache_lru_size limit

Fixes: http://tracker.ceph.com/issues/22410

Signed-off-by: Mark Kogan <mkogan@redhat.com>
2017-12-12 13:39:43 -05:00
Sage Weil
1908c06149 osd/ECBackend: fix on_write ordering w/ sync onreadable callbacks
When we call handle_sub_write after a write completion, we may
do a sync read completion and then call back into check_ops().  Attaching
the on_write events to the op we're applying means that we don't ensure
that the on_write event(s) happen before the next write in the queue
is submitted (when we call back into check_ops()).

For example, if we have op A, on_write event W, then op B, a sync
applied completion would mean that we would queue the write for A, call
back into SubWriteApplied -> handle_sub_write_reply -> check_ops and then
process B... before getting to W.

Resolve this by attaching the on_write callback to a separate Op that is
placed into the queue, just like any other Op.  This keeps the ordering
logic clean, although it is a bit ugly with the polymorphism around Op
being either an Op or an on_write callback.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-12 11:21:49 -06:00
Yuri Weinstein
f760313f94
Merge pull request #19144 from zhangsw/cleanup-rgwputobj-md5
rgw: no need to deal with md5 header in get_data.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 08:28:43 -08:00
Yuri Weinstein
a2dc12bb0c
Merge pull request #19167 from joke-lee/Bucket_Policy_SourceIp_Condition
rgw: implement ipv4 aws:SourceIp condition for bucket policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 08:28:25 -08:00
Yuri Weinstein
597884065f
Merge pull request #19210 from Liuchang0812/rgw-camelcase
rgw: use camelcase format in request headers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2017-12-12 08:27:23 -08:00
Yuri Weinstein
0764e7b019
Merge pull request #19236 from dplyakin/get-website-error-code
rgw: fix GET website response error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 08:26:32 -08:00
Yuri Weinstein
51ee04f7fb
Merge pull request #19250 from qrGitHub/wip-rgw-unused-param2
rgw: remove unused param in AWSGeneralAbstractor::get_auth_data_v4

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
2017-12-12 08:25:54 -08:00
Yuri Weinstein
a7abf49395
Merge pull request #19390 from yaozongyou/remove-duplicated-include
rgw,common: remove already included header files

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
2017-12-12 08:25:17 -08:00
Yuri Weinstein
c101f2fd00
Merge pull request #19391 from qrGitHub/wip-rgw-memory-optimize
rgw: optimize memory usage in function rgw_bucket::get_key

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 08:23:05 -08:00
Yuri Weinstein
66bc71d646
Merge pull request #19430 from tianshan/fix_recursive_lock
rgw: fix recursive lock

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 08:22:47 -08:00
Sage Weil
0d98f262c0 osd/PG: include primary in PG operator<< for ec pools
Otherwise it is confusing!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-12 09:22:37 -06:00
Kefu Chai
e38ef3709e
Merge pull request #19418 from tchaikov/wip-gcc-7-on-xenial
install-deps.sh, cmake: use GCC-7 on xenial also

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2017-12-12 23:15:35 +08:00
Jason Dillaman
722795de5a
Merge pull request #19402 from b-ranto/wip-fix-arm-build
test/librbd: Do not instantiate TrimRequest template class

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-12-12 10:11:09 -05:00
Sage Weil
49b327e865 mon/LogMonitor: add mon_cluster_log_to_stderr
Optionally send cluster log messages to stderr (prefixed by the
channel).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-12 09:09:28 -06:00
Jason Dillaman
548bc8bad2
Merge pull request #19437 from trociny/wip-mock-valgrind
test/rbd_mirror: "use of uninitialised value" valgrind warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
2017-12-12 10:09:15 -05:00
Sage Weil
c142ae3f15 log: add log_stderr_prefix option
Allows you to set a prefix for debug log messages send to stderr (e.g.,
"debug ").

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-12 08:38:41 -06:00
Yehuda Sadeh
a01c014772
Merge pull request #19253 from oritwas/wip-rgw-22124
rgw: reshard should not update stats when linking new bucket instance

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2017-12-12 14:14:41 +02:00
Mykola Golub
bad51ffe7f rbd-nbd: support optionally setting device timeout
Fixes: http://tracker.ceph.com/issues/22333
Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>
2017-12-12 11:10:54 +02:00
Sage Weil
4389b55435 Merge remote-tracking branch 'gh/mimic-dev1' 2017-12-11 22:27:35 -06:00
Sage Weil
6fdd8ba284
Merge pull request #19435 from smithfarm/wip-20559
doc: globally change CRUSH ruleset to CRUSH rule

Reviewed-by: Sage Weil <sage@redhat.com>
2017-12-11 18:55:13 -06:00
Boris Ranto
ce2e26d39a librbd: Do not instantiate TrimRequest template class
We include TrimRequest.cc in librbd tests at two places:
 - operation/test_mock_TrimRequest.cc
 - operation/test_mock_ResizeRequest.cc

That causes linking errors when doing the builds because some of the
instantiated classes are defined twice.

We can fix this by not instantiating the template class in the
TrimReqeust.cc file when including it in the tests.

Signed-off-by: Boris Ranto <branto@redhat.com>
2017-12-12 01:08:04 +01:00
Casey Bodley
f8909bb6a6 qa/rgw: disable log trim in multisite suite
the multisite tests run manual trim operations with radosgw-admin, which
can race with internal log trimming to produce tests failures

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2017-12-11 16:48:02 -05:00
Sage Weil
fef5fe8ba0 osd/PrimarLogPG: remove jewel compat for error code logging
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
3ccc436f9f os/bluestore: simplify Onode tracking for removed object
Make the only caller of removed() not need to call note_modified_object
separately, dropping the unneeded erase() call.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
0ededde59f os/bluestore: store shard, not shard_hint,in OpSequencer
Good suggestion from Igor!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
75b71abc76 osd/PrimaryLogPG: sync complete from C_OSD_OpApplied
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
0347518d02 os/bluestore: keep ref to deleted onode in _do_remove
We need to make sure we carry this ref through until the object is
deleted or else another request right try to read it before the kv
txn is applied.  (This is easy to trigger now that onreadable is completed
at queue time instead of commit time.)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
8dae6d198e ceph_test_objectstore: flush() before collection_list()
The one exception to the "immediately readable" it collection_list, which
is not readable until the kv transaction is applied.  Our choices are

1. Wait until kv to apply to trigger onreadable (for any create/remove
ops).  This wipes away much of the benefit of fully sync onreadable.

2. Add tracking for created/removed objects in BlueStore so that we can
incorporate those into collection_list.  This is complex.

3. flush() from collection_list.  Unfortunately we don't have osr linked
to Collection, so this doesn't quite work with the current ObjectStore
interface.

4. Require the caller flush() before list and put a big * next to the
"immediately onreadable" claim.  It turns out that because of FileStore,
the OSD already does flush() before collection_list anyway, so this does
not require any actual change... except to store_test tests.  (This didn't
affect filestore because store_test is using apply_transaction, which
waits for readable, and on filestore that also implies visible by
collection_list.)

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
354a6bab56 os/bluestore/BitmapFreelistManager: disable bluestore_debug_freelist
This doesn't work as implemented.  We are doing _txc_finalize_kv() from
queue_transactions, which calls into the freelist and does this verify
code.  However, we have no assurance that a previous txc in the sequencer
has applied its changes to the kv store, which means that a simple sequence
like

 - write object
 - delete object

can trigger if the write is waiting for aio.  This currently happens
with ObjectStore/StoreTest.SimpleRemount/2.

Comment out the verify, but leave _verify_range() helper in place in case
we can use it in the future in some other context.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
ec9f1fe4ca os/bluestore: fix _set_alloc_sizes dout
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
dbb365de83 os/bluestore: avoid OpSequencer::parent
The parent may go away, so we need to keep our own copy of shard_hint in
OpSequencer to avoid a user-after-free (e.g., when the user drops their
osr and calls OpSequencer::discard()).

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
c77cb1b98d os/ObjectStore: expose is_sync_onreadable()
Only FileStore is false.

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
6d7d8ba94b osd/PrimaryLogPG: sync_complete for C_OSD_AppliedRecoveryObject[Replica]
Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00
Sage Weil
779e01085b osd/PG: sync_complete for C_UpdateLastRollbackInfoTrimmedToApplied
One of these is attached to every write!

Signed-off-by: Sage Weil <sage@redhat.com>
2017-12-11 15:05:38 -06:00