Commit Graph

31330 Commits

Author SHA1 Message Date
Yehuda Sadeh
1b0567b0af radosgw-admin: add usage for user stats [--sync-stats]
add missing usage

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:25:51 -08:00
John Wilkins
97edd2fcad doc: Switched logging to true. Now required for sync agent.
Signed-off-by: John Wilkins <john.wilkins@inktank.com>
2014-01-24 15:11:32 -08:00
Yehuda Sadeh
9a55fa1cac cls_user: add generate_test_instances() implementation
to all the encoded/decoded stuff

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:07:36 -08:00
Yehuda Sadeh
e5dc8d6583 rgw, cls_user: handle error cases related to response decoding
Certain operations weren't handling errors in decode, add some missing
logic.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:07:36 -08:00
Yehuda Sadeh
8a69ac89e2 rgw: implement restful set user quota request
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:07:36 -08:00
Yehuda Sadeh
92cabd411c rgw: new restful api for retrieving user quota info
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:07:36 -08:00
Yehuda Sadeh
0f7b3c722f rgw: quota thread for full user stats sync
Get user stats up to date periodically. Add configurables for different
periods, whether we update idle users.
Make sure radosgw-admin does not start the quota threads.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 15:07:35 -08:00
Greg Farnum
51ffede3e3 mon: do not use CEPH_FEATURES_ALL for things that touch the disk
We want to encode with our quorum_features instead. Remaining uses of
CEPH_FEATURES_ALL are:
1) when the Elector is sharing its supported features
2) in a MonMap function which is used by monmaptool
3) In the Monitor for winning a standalone election, for ephemeral data,
  and for doing mkfs (when we necessarily don't have quorum_features).
4) When doing ceph-mon --inject-monmap (we don't persist the quorum_features
  to disk, so we can't use them here).
5) in MMonElection, for doing the default monmap encoding (which is
  re-encoded later if the final features don't match CEPH_FEATURES_ALL).
6) As the default encode features for the OSDMap (the monitor always
  supplies quorum_features instead).

Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:56 -08:00
Greg Farnum
3d4a6739f2 Elector: send an OP_NAK MMonElection to old peers who support it
Only new monitors support receiving OP_NAK from a peer without crashing, but
when we add new required features in the future, our monitors can accept
an OP_NAK message which tells them what features they're missing. Then they
will print out an error message and shut down.
(Unfortunately, doing a clean shutdown from here would require a lot of
infrastructure, so we just call exit(0).)

Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:56 -08:00
Greg Farnum
687b570b71 Elector: ignore messages from mons without required feature capabilities
We maintain a list of required_features which the other monitor's features
must supply. This starts out at 0 and is initialized from the monitor's
list of features whenever we start electing.
Despite the scary sound of "just ignore it", this is safe: the monitor
will only record features as required once a quorum has formed in which
every monitor supports them. After that happens, monitors which do not
support those features will be unable to read the whole mon store/understand
the pg reports/whatever else, so letting them into the quorum would be buggy
behavior.
So if we ignore a monitor, it will not be able to start nor join
an election round with anybody who was in our quorum -- that is, the
ignored monitor cannot form a separate quorum. By ignoring it here, we
also prevent it from endlessly calling elections against the real
quorum.

Unfortunately there is no way to communicate to old monitors that they
cannot join the quorum -- there are no existing messages for that purpose,
and eg adding a new op to the MMonElection message will just cause it
to crash, which we don't want to do either.

Fixes: #7215

Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:56 -08:00
Greg Farnum
41796c07ac Monitor: add a function to get the required quorum features from the local compatset
We will use this to avoid deferring to people who can't read our store.

Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:56 -08:00
Greg Farnum
f8e4af2ddb OSDMonitor: do not allow creation of EC pools if the quorum can't handle it
Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:56 -08:00
Greg Farnum
e79e1d28a6 Monitor: introduce a function that translates quorum features into disk features
After an election, we call apply_quorum_to_compatset_features(). It translates
from the quorum features into monitor disk state features we care about and
adds them to the disk store. This prevents an older daemon from starting up
on a store it doesn't understand.

While strictly speaking we don't need to add the EC feature until we create
an EC pool, all monitors which speak the new OSDMap encoding also support EC
pools, and having more than one feature makes the pattern clearer.

Signed-off-by: Greg Farnum <greg@inktank.com>
2014-01-24 12:33:55 -08:00
Yehuda Sadeh
2f9a93d385 rgw: round bytes to the next 4k for quota use
Be more consistent as we round it like this in the objclass, so we need
the accounting to match.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:53 -08:00
Yehuda Sadeh
6f34df45ed rgw: sync bucket stats thread
a new thread that periodically sync stats of recently modified buckets.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:53 -08:00
Yehuda Sadeh
7545177ab7 rgw, cls_user: fix bucket creation
There's a single op to create and update the user bucket info, however,
the cases differ a bit, as we only need to guard against ENOENT if we're
updating the info.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:53 -08:00
Yehuda Sadeh
d9920439e2 radosgw-admin: dump header when requested for user stats
header also contains sync timestamps, so we should dump it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:52 -08:00
Yehuda Sadeh
5976bfcc58 radosgw-admin: update stats sync completion
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:52 -08:00
Yehuda Sadeh
ba39512892 cls_user: op to keep track of sync completion time
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:52 -08:00
Yehuda Sadeh
a68b52afa4 cls_user: keep time info about stats update
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:52 -08:00
Yehuda Sadeh
0b7968cda9 cls_user: header cleanup
put stats under a new struct

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:52 -08:00
Yehuda Sadeh
45b222930d rgw: user quota should rely on cached info
user quota should rely on cached info if it says that we're at the
limit, since backend info might not be up to date.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:51 -08:00
Yehuda Sadeh
c59a98e0b1 rgw: more quota fixes
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:51 -08:00
Yehuda Sadeh
12f5801bbd rgw: configure user quota also through region map
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:51 -08:00
Yehuda Sadeh
a6fae68900 rgw: more rgw_quota fixes
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:51 -08:00
Yehuda Sadeh
04b5a71116 rgw: some quota fixes
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:50 -08:00
Yehuda Sadeh
cfb9553c84 rgw: enable user quota
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:50 -08:00
Yehuda Sadeh
15c01895fb radosgw-admin: user quota interface
also add needed functionality in RGWUser

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:50 -08:00
Yehuda Sadeh
a854d8937d rgw: some fixes
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:50 -08:00
Yehuda Sadeh
c1c40df12b rgw: implement quota handlers for user stats
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:50 -08:00
Yehuda Sadeh
c13634e888 rgw: extend infrastructure to handle user stats read
Some helper functions that return data in required format for quota, add
async functions.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:49 -08:00
Yehuda Sadeh
364f7c54fc rgw: move some code around, no actual changes
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:49 -08:00
Yehuda Sadeh
92a87b3dfd rgw: refactor quota cache
bucket quota cache is going to be reused, add a new abstract layer.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:49 -08:00
Yehuda Sadeh
4aee3fa228 rgw: rename RGWBucketStats to RGWStorageStats
make it more generic, will be reused for user's stats.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:49 -08:00
Yehuda Sadeh
5612ded3b1 rgw: pass bucket owner all around
User quota operations require that we know who the actual user we do the
operation on is. Pass that info when creating new object and when
removing objects.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:48 -08:00
Yehuda Sadeh
cbe1da85c8 radosgw-admin: bucket stats --sync-stats [--bucket=...]
Sync user stats, either for a single buckets, or for all user's buckets.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:48 -08:00
Yehuda Sadeh
c8793fbad0 cls/user: some fixes + handle old buckets case
If a bucket was created prior to the new accounting, we can't substract
the old values, as the header stats didn't account for them.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:48 -08:00
Yehuda Sadeh
eef7ba48df rgw, radosgw-admin, cls/user: retrieve user stats
add a new op to retrieve user header, new radosgw-admin command to stat
user.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:48 -08:00
Yehuda Sadeh
edcf9fed02 cls/user: a new op to retrieve user header
user header holds user total stats

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:47 -08:00
Yehuda Sadeh
438aea7f7f cls/user: accounting functionality
Keep user total accounting in omap header, handle add/dec functionality.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:47 -08:00
Yehuda Sadeh
c7b4d00815 rgw: move bucket add / remove to new user objclass
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:47 -08:00
Yehuda Sadeh
23aa65f62a rgw: replace user bucket listing with objclass call
instead of directly accessing the omap operation

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:47 -08:00
Yehuda Sadeh
248c4aefff rgw: new user objclass
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
2014-01-24 10:28:41 -08:00
Sage Weil
5756f7c492 Merge pull request #1102 from ceph/wip-rgw-civetweb-2
Wip rgw civetweb 2

Reviewed-by: Sage Weil <sage@inktank.com>
2014-01-24 08:31:37 -08:00
David Moreau Simard
b90570f3dc Fix 404 broken links to logging and debug configuration
Signed-off-by: David Moreau Simard <dmsimard@iweb.com>
2014-01-24 10:41:46 -05:00
Josh Durgin
28af392bc3 Merge pull request #1116 from jcsp/doc_toofull
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-01-23 22:43:43 -08:00
Josh Durgin
e23513fbe2 Merge pull request #1135 from ceph/use-ceph-sharedptr
libc++: convert tr1 uses

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2014-01-23 22:36:18 -08:00
Noah Watkins
faae5b94b4 libc++: convert tr1 uses
Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
2014-01-23 20:35:53 -08:00
John Wilkins
ea026c6cc9 doc: Added domain pool, and changed zone configs to use domain tool so they don't clash.
Signed-off-by: John Wilkins <john.wilkins@inktank.com>
2014-01-23 17:11:59 -08:00
John Wilkins
0303f0f17a doc: Change zone domain to include "domain" so it is clear.
Signed-off-by: John Wilkins <john.wilkins@inktank.com>
2014-01-23 17:11:23 -08:00