Commit Graph

43021 Commits

Author SHA1 Message Date
Yehuda Sadeh
f899310273 rgw: cannot re-init an orphan scan job
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
d7d1171218 rgw: stat_async() sets the object locator appropriately
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
20bd490beb rgw: list_objects() sets namespace appropriately
list_objects() now uses parse_raw_oid(), so that it can set the correct
namespace. It only affects users of the function that want to get all
objects in bucket, regardless to the namespace associated with it. This
makes it so that the orphan code actually works now with namespaced
objects, and with special named objects (namely, start with underscore).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
fa61ea2ccc rgw: modify orphan search fingerprints
Now works with multipart uploads.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
1bc63d98ff rgw: compare oids and dump leaked objects
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
f19b2f087c rgw: keep accurate state for linked objects orphan scan
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
75902fdb12 rgw: iterate over linked objects, store them
only keep part of the oid name if it is in a namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
7c1aa8373a rgw: add rgw_obj::parse_raw_oid()
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
05a953d965 rgw: iterate asynchronously over linked objects
Read objects manifest. So that we could keep the relevant info later.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:02 -07:00
Yehuda Sadeh
13adf3cf4f rgw: async object stat functionality
An async functionality that stats object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:01 -07:00
Yehuda Sadeh
16a2dbd838 rgw-admin: build index of bucket indexes
for the orphan search tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:01 -07:00
Yehuda Sadeh
767fc29aa6 rgw: initial work of orphan detection tool implementation
So far doesn't do much, iterate through all objects in a specific pool
data, store it in a sharded index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-29 15:09:01 -07:00
Ken Dreyer
69a22b2920 Merge pull request #5094 from SUSE/wip-sharutils-dupe
ceph.spec.in: remove duplicate BuildRequires: sharutils

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2015-06-29 13:27:28 -06:00
branto1
0537cf241c Merge pull request #5082 from ceph/wip-fedora-babeltrace
ceph.spec.in: fix babeltrace handling on Fedora

Reviewed-by: Ira Cooper ira@samba.org
Reviewed-by: Boris Ranto branto@redhat.com
2015-06-29 16:01:33 +02:00
Loic Dachary
90b2ccabc0 Merge pull request #5098 from PierreChaumont/master
doc: homogenize Librados (Python) documentation's code samples

Reviewed-by: Loic Dachary <ldachary@redhat.com>
2015-06-28 13:36:37 +02:00
PierreChaumont
8a221c3509 doc: homogenize Librados (Python) documentation's code samples
I put in doc/rados/api/python.rst line numbers on every code samples in
order to make them look all the same.

Fixes: #12174

Signed off: Pierre Chaumont pierre.chaumont31@gmail.com
2015-06-27 16:00:35 +02:00
Yehuda Sadeh
8103908548 rgw: only scan for objects not in namespace
Fixes: #11984
The tool should only work on the head objects, and these are not inside
any namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-26 17:07:02 -07:00
Nathan Cutler
c418bd9615 ceph.spec.in: remove duplicate BuildRequires: sharutils
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-26 20:21:59 +02:00
Ken Dreyer
6dfba7678e Merge pull request #4957 from SUSE/wip-python-flask
ceph.spec.in: fix python-flask dependency for SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2015-06-26 12:20:33 -06:00
Nathan Cutler
39c28b9faf ceph.spec.in: fix python-flask dependency for SUSE
In SLE and openSUSE, the package is called python-Flask with an upper-case F.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-26 20:12:45 +02:00
Ken Dreyer
909301917e Merge pull request #5086 from SUSE/wip-12173
logrotate.conf: fixes for systemd

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2015-06-26 11:49:59 -06:00
Nathan Cutler
05424a803b logrotate.conf: fixes for systemd
Before this patch, the command 'logrotate -f /etc/logrotate.d/ceph'
was generating an error "Failed to reload ceph.target: Job type reload is not
applicable for unit ceph.target".

Before we issue systemctl reload, check that there is at least
one active ceph-* service. (The hyphen is significant.)

Since we use grep, make the grep package a dependency.

http://tracker.ceph.com/issues/12173 Fixes: #12173

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-26 19:43:44 +02:00
Yehuda Sadeh
04a69faff2 Merge pull request #5091 from yehudasa/wip-multipart-parts-limit
rgw: Multipart Upload: Support to configure and enforce no of parts a…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-26 09:40:28 -07:00
Abhishek Dixit
876fbc64ba rgw: Multipart Upload: Support to configure and enforce no of parts allowed
Fixes: #12146
Config parameter added for no of parts limit in multipart upload and checked while
completing multipart upload

Signed-off-by: Abhishek Dixit <dixitabhi@gmail.com>
2015-06-26 09:38:31 -07:00
Yehuda Sadeh
fb6f2717fb Merge pull request #5089 from rzarzynski/wip-enforce_content-type_for_swift-next
rgw [next]: enforce Content-Type in Swift responses.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-26 09:18:54 -07:00
Ken Dreyer
e579e12c5b Merge pull request #4970 from SUSE/wip-12034-master
ceph.spec.in: python-argparse only in Python 2.6

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2015-06-26 10:06:15 -06:00
Josh Durgin
cb7b26bfbc Merge pull request #5090 from ceph/wip-12176-next
librbd: assertion failure race condition if watch disconnected

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2015-06-26 11:52:26 -04:00
Josh Durgin
398467c79d Merge pull request #5080 from ceph/wip-12165-next
librbd: prevent object map updates from being interrupted

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2015-06-26 11:51:48 -04:00
Jason Dillaman
af276de4f2 librbd: assertion failure race condition if watch disconnected
It's possible for librbd's watch of the header object to be reset by
connection issues just prior to the image being removed.  This will
causes an assertion failure which assumes at least one watcher on the
image.

Fixes: #12176
Backport: hammer, firefly
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-06-26 09:59:36 -04:00
Radoslaw Zarzynski
106aeba206 rgw: enforce Content-Type in Swift responses.
Swift sends Content-Type HTTP header even if the response
doesn't contain body. We have this behaviour implemented
until applying some changes in end_header() function.
Unfortunately, lack of Content-Type causes early exits in
many Tempest's tests for Swift API verification.

Fixes: #12157
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
2015-06-26 15:26:35 +02:00
Nathan Cutler
23171c9529 ceph.spec.in: python-argparse only in Python 2.6
argparse is a widely-used Python module for parsing command-line arguments.
Ceph makes heavy use of Python scripts, both in the build environment and on
cluster nodes and clients.

Until Python 2.6, argparse was distributed separately from Python proper.
As of 2.7 it is part of the Python standard library.

Although the python package in a given distro may or may not Provide:
python-argparse, this cannot be relied upon.

Therefore, this commit puts appropriate conditionals around Requires:
python-argparse and BuildRequires: python-argparse. It does so for Red
Hat/CentOS and SUSE only, because the last Fedora version with Python 2.6
was Fedora 13, which is EOL.

argparse is required by both the ceph and ceph-common packages, but since ceph
requires ceph-common, the argparse Requires and BuildRequires need only appear
once, under ceph-common.

http://tracker.ceph.com/issues/12034 Fixes: #12034

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-26 12:46:46 +02:00
Haomai Wang
891f3e0e08 Merge pull request #5035 from yuyuyu101/wip-async-fix-13
AsyncConnection: Fix out-of-sequence problem introduced by send/write separated logic
2015-06-26 18:01:18 +08:00
Orit Wasserman
32a7d561e1 Merge pull request #5081 from ceph/wip-11851
rgw: error out if frontend did not send all data
2015-06-26 11:45:33 +02:00
Kefu Chai
2f2a5f6d2d Merge pull request #5022 from liewegas/wip-temp-ctor
osd: make ghobject_t(hobject_t) ctor explicit

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2015-06-26 11:02:54 +08:00
Yan, Zheng
d91b9d1949 Merge pull request #5001 from ceph/wip-11985
#11985 Fixes to MDLog (and MDS) shutdown
2015-06-26 09:18:14 +08:00
Yehuda Sadeh
daa679c3dd rgw: error out if frontend did not send all data
Fixes: #11851
The civetweb mg_write() doesn't return error when it can't flush all data
to the user, it just sends the total number of bytes written. Modified the
client io to return total number of bytes and return an error if didn't
send anything.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-25 14:35:14 -07:00
Jason Dillaman
590cdc90ed librbd: prevent object map updates from being interrupted
Object map updates were being canceled in-flight when the exclusive lock
is released.  This resulted in an ERESTART error code bubbling up to
AioRequest.

Fixes: 12165
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
2015-06-25 16:51:31 -04:00
Ken Dreyer
39b47f9f6f Merge pull request #5077 from SUSE/wip-spec-comment-clarify
ceph.spec.in: clarify two important comments
2015-06-25 13:28:48 -06:00
Nathan Cutler
dd212fd1d3 ceph.spec.in: clarify two important comments
First, the terms "common" and "specific" are vague. Second,
"common" can easily be confused with the ceph-common subpackage.

Fix this by rephrasing to "distro-unconditional dependencies" and
"distro-conditional dependencies", respectively.

Third, move the "distro-unconditional dependencies" header so it
is above the part where these dependencies are actually defined.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-25 19:55:12 +02:00
Yehuda Sadeh
83539c07fe Merge pull request #5033 from wuxingyi/fixuserquota
rgw: fix empty json response when getting user quota

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
2015-06-25 10:30:43 -07:00
Radoslaw Zarzynski
5f47b11791 rgw: send Content-Length in response for GET on Swift account.
Fixes: #12158
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
2015-06-25 18:53:28 +02:00
Ken Dreyer
c0ace46cc2 Merge pull request #4942 from SUSE/wip-11991
rpm: add missing Java conditionals

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
2015-06-25 10:47:21 -06:00
John Spray
39cf071185 mds: fix MDLog shutdown process
We must join threads before completing ::shutdown,
because otherwise these threads might try to use
torn-down resources like the objecter.

The replay/recovery threads may be blocking on
journaler calls like wait_for_readable, so we
must signal them using Journaler::shutdown.  In
order for that to be safe, we must also protect
the assignment of ::journaler from the threads
using the mds_lock.

Fixes: #11985
Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 16:19:25 +01:00
John Spray
f2daa19c8d mds: drop MDSIOContext on mds->stopping
`stopping` is true once the MDS has entered suicide().  During
this phase, any MDSIOContexts that are called (e.g. from objecter,
journaler) become no-ops.

Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 16:19:24 +01:00
John Spray
ae387b6577 mds: refine shutdown, add ::stopping
Add a ::stopping flag, set at start of suicide(),
that other contexts must inspect after taking
mds_lock.

This guards against the possibility of multiple
threads entering suicide, and more generally
against the possibility of other procedures
starting while we're in the middle of shutting down.

Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 16:19:24 +01:00
John Spray
8072b19d37 osdc/Journaler: add a shutdown() method
Because consumers may be blocked on the on_readable
condition, it is necessary for Journaler to have
an explicit shutdown method that fires the completion

Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 16:19:24 +01:00
John Spray
fe23c6c5e6 common/Thread: print msg on join errors
Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 16:19:24 +01:00
Nathan Cutler
0734cd1ec3 packaging: make logrotate a hard dependency across all distros
Given ceph's reputation for voluminous logs, it makes sense for
logrotate to be a hard dependency for all distros.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
2015-06-25 16:55:10 +02:00
John Spray
95796a3db9 mds: add a ceph_file_layout wrapper for dencoder
So that we can decode it from tests.  This should go away
at some point when encoding of these raw-encoded structures
is modernized.

Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 14:47:13 +01:00
John Spray
c097881856 test/ceph-dencoder: add boilerplate
... mainly just for the modelines.

Signed-off-by: John Spray <john.spray@redhat.com>
2015-06-25 14:47:13 +01:00