Commit Graph

74 Commits

Author SHA1 Message Date
Zack Cerza
12af2abb4c Rename some functions and variables
This is to make the refactoring a little smoother and easier to read.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
2014-04-15 13:12:00 -05:00
Zack Cerza
f82f663782 Fix all but one of the PEP-8 issues
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
2014-04-15 10:58:36 -05:00
Alfredo Deza
15a3acb713 fix redefinition of region
Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>
2014-04-09 16:26:47 -04:00
Zack Cerza
158f9ba1ff Revert "Lines formerly of the form '(remote,) = ctx.cluster.only(role).remotes.keys()'"
This reverts commit d693b3f895.
2014-03-27 11:35:28 -05:00
Warren Usui
d693b3f895 Lines formerly of the form '(remote,) = ctx.cluster.only(role).remotes.keys()'
and '(remote,) = ctx.cluster.only(role).remotes.iterkeys()' would fail with
ValueError and no message if there were less than 0 or more than 1 key.
Now a new function, get_single_remote_value() is called which prints out
more understandable messages.

Fixes: 7510
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Signed-off-by: Warren Usui <warren.usui@inktank.com>
2014-03-26 18:43:48 -07:00
Josh Durgin
b2cf052a08 rgw: allow overriding options
This is mainly useful for toggling ec/replicated pools right now.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-03-25 18:04:35 -07:00
Josh Durgin
f89c0d59fb rgw: fix ec-data-pool config
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-03-25 14:57:50 -07:00
Josh Durgin
47ba1711d7 rgw: add option to use erasure coding instead of replication
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-03-24 18:19:41 -07:00
Sage Weil
4e2f36fcd8 rgw: handle empty conf case
2014-03-20T23:30:28.964 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 83, in ship_config
    idle_timeout = conf.get('idle_timeout', 30)
AttributeError: 'NoneType' object has no attribute 'get'

This happens with job yamls like:

- rgw:
  - client.0

(where it is a list and not a dict of clients)

Signed-off-by: Sage Weil <sage@inktank.com>
2014-03-21 08:36:24 -07:00
Sage Weil
3a2b77c142 rgw: allow fastcgi idle timeout to be adjusted
Signed-off-by: Sage Weil <sage@inktank.com>
2014-03-20 16:09:52 -07:00
Josh Durgin
55fad94ab1 rgw: create clients in all zones when regions are used
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-03-10 00:29:50 -07:00
Josh Durgin
871fe02027 rgw: set placement targets for zones
This lets zones use different pools for actual object data.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2014-03-10 00:29:50 -07:00
Warren Usui
c631190a76 Add docstrings. Fix formatting issues for rgw.py
Fix s3tests calls for rgw-logsocket.py.

Fixes: 6540
Signed-off-by: Warren Usui <warren.usui@inktank.com>
2014-02-14 17:08:28 -08:00
Sandon Van Ness
c9774893a9 Use worker httpd instead of prefork (like ubuntu) on rpm distros.
Ubuntu's default apache uses worker instead of prefork like rpm
based distro's. If rpm use httpd.worker instead of httpd so that
the -X behavior will not be blocked by a single request.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
2013-10-25 13:14:21 -07:00
Sage Weil
6a2dd66d6b rgw: revert to -X for apache2
-DNO_DETACH doesn't seem to work.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-10-22 14:36:35 -07:00
tamil
01ce92c497 replacing -X option with -DNO_DETACH to ensure the
apache process is detached at the end of run.

Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
2013-10-16 17:14:49 -07:00
tamil
7b889797c3 ignoring -X option in apache as it was causing s3test failures on the rpms.
Signed-off-by: tamil <tamil.muthamizhan@inktank.com>
2013-10-09 17:12:16 -07:00
Sage Weil
6b4014452f rgw: fix valgrind when no valgrind
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-09-06 17:38:29 -07:00
Sage Weil
9e03c73775 misc: valgrind: fix cd behavior
The cd needs to happen at the beginning of the command, not at the end, or
else the funky wrapper scripts don't work right.

This also cleans up the command line construction a bit so that it is more
flexible, explicit, and hopefully less fragile.

Signed-off-by: Sage Weil <sage@inktank.com>
2013-09-06 16:09:27 -07:00
Zack Cerza
53b8e27da9 Helper scripts live in /usr/local/bin now! 2013-09-06 15:10:11 -05:00
Joe Buck
db518885c9 rgw: persist the region info
The rgw task deletes the region info
from the config structure. The s3tests
task needs this info, so we persist
it by sticking it in the ctx object.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewd-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-28 09:00:52 -07:00
Joe Buck
549bac3ede radosgw-admin: use dynamic ports for testing
The rgw.py task was extended to  dynamically
assign port numbers to radosgateways.
This patch extends the radosgw-admin task
to use those ports rather than making out-dated
assumptions of port numbering.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-20 21:21:17 -07:00
Joe Buck
7061e3c6dc rgw: fix domain root pool name assignment
Do not make the domain root pool the same
as the zone root pool. That causes sync issues.
Also, clarify a logging message.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-16 19:26:21 -07:00
Joe Buck
1d90d7bc83 rgw.py: fix example config
In the example config, the region root and
zone root where pointing to the same pool,
which is not a best practice. Updated the
example to show them pointing to different pools.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-12 23:25:43 -07:00
Joe Buck
9ee6452404 rgw.py: add a better error message
When pulling region info from the config
structure, if the region info isn't there,
log a more helpful message.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-12 23:25:33 -07:00
Joe Buck
e3b9add3fb rgw.py: refactor configure
Extend the rados pool configuration options to
specify all pools (if desired).
Also, reordered zone and region configuration
so that they're configure (per client) in
this order: zone, region, set default region

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-12 23:25:16 -07:00
Joe Buck
18e80147f3 rgy.py: make log_(data|metadata) configurable
The log_data and log_metadata are made configurable
via the YAML file and default to false
(meaning neither data nor metadata operations are
logged).

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-08-12 23:25:06 -07:00
Joe Buck
4bd5b591d4 rgw.py: add None object check when parsing info
The rgw task was failing to check for a None object
when parsing user info in the case where there were
config options set for the client that did not include
user info (e.g. valgrind: ).
2013-08-03 17:16:52 -07:00
Joe Buck
3accf1c68a rgw.py: adding a safety check
Correcting a bug where specifying
a rgw server for a client but not specifying
a system user would throw an exception.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 21:05:15 -07:00
Josh Durgin
9110c7d8ca rgw: add log settings to example config
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 19:12:05 -07:00
Josh Durgin
fdde365858 rgw: don't make domain_root pool the same as zone root
These are supposed to be separate pools.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 19:12:01 -07:00
Josh Durgin
d79a2db578 rgw: create system users with the --system flag
without this they have no special permissions

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 19:11:59 -07:00
Joe Buck
3eb62d2a36 rgy.py: make log_(data|metadata) configurable
The log_data and log_metadata are made configurable
via the YAML file and default to false
(meaning neither data nor metadata operations are
logged).

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 19:11:58 -07:00
Josh Durgin
ad6a9e537f rgw: ignore exit status when removing base apache dir
subdirs have already been removed, and this only needs to succeed once
per host.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-08-01 17:33:35 -07:00
Joe Buck
1552a4b97a rgw.py: check for empty client config
Fixes a bug where an rgw client without
a system user specified would cause teuthology
to error out.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Yehuda Sadeh <yehuda@inktank.com>
2013-08-01 10:28:26 -07:00
Joe Buck
7f76c1ced1 rgw.py: refactoring to separate user creation
By separating out the user creation from
generating the region/zone info, we can generate
users for RGW tests that run against the default
pools.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-31 16:02:42 -07:00
Joe Buck
8c8e9a1403 rgw.py: change --secret-key to --secret
A 'user create' call was being passed to radosgw-admin
with '--secret-key' instead of the valid '--secret'
which was causing a random secret to be generated,
which was causing subsequent tests to fail.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-07-31 14:18:20 -07:00
Josh Durgin
2f2108b3c0 rgw: fix dir creation and keyring
fastcgi_sock dir needs to exist before radosgw starts, and apache-execed radosgw needs an explicit keyring argument.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-31 12:47:48 -07:00
Josh Durgin
f2cddda154 rgw: correct socket option name
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-26 14:18:24 -07:00
Josh Durgin
699d0a3b27 rgw: add rgw log socket to daemonized radosgw too
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-26 14:15:28 -07:00
Josh Durgin
65172a0938 rgw: pass socket path directly to radosgw
Only radosgw needs this option, and each one will be different, so
remove it from the ceph.conf template.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-26 14:13:19 -07:00
Joe Buck
3c9382a7fa teuth: rgw task example config update
Due to bug #5716, pools need to start with a '.' at present.
Updating the examples to follow this convention.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-24 13:00:18 -07:00
Joe Buck
4fb5781afb teuth: fix issue in cleanup code
The post-yield code in create_dirs needed to
be tweaked to correctly delete the {tdir}/apache
directory (if it exists) on each client.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-24 13:00:10 -07:00
Josh Durgin
8ad065dc9c rgw: add multi-region and zone support
Take client<->zone/region and the associated pools from ceph.conf, so
we don't have to invent a new format to specify it.

General region info is added to a new configuration section in the rgw
task. Each client is assumed to be a different zone, and a system user
is created with the key specified in the yaml, so it can be passed to
later task configuration as well. This isn't strictly necessary, but
avoids having to lookup this info in later tasks through something
like radosgw-admin.

Ports are allocated automatically because there's no obvious mapping
from host to client in the task configuration. Later tests can get the
endpoints desired by reading the region map.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-24 09:59:50 -07:00
Josh Durgin
721280b7f3 task_util: move rados command here
Six copies are replaced with one, with an added option to check status
automatically. This should probably be used in a few places where the
return code is ignored.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-24 09:59:50 -07:00
Josh Durgin
6d2434b48f rgw: move common rgwadmin function to a new utility file
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-24 09:59:50 -07:00
Joe Buck
88cab4712f teuth: reworked rgw to support regions / zones
Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-07-24 09:59:50 -07:00
Joe Buck
afd0d87eda teuthology: updating RGW task to support regions
Extending the rgw.py task to allow for regions and zones
to be specified in the YAML file.

Signed-off-by: Joe Buck <jbbuck@gmail.com>
2013-07-24 09:59:50 -07:00
Josh Durgin
f0123db7d9 rgw: turn off continue print on centos
Also note that a ceph.conf setting is needed for that too

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-19 14:42:38 -07:00
Josh Durgin
c0a7808164 rgw: use different daemon name and apache module paths
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
2013-07-10 18:00:25 -07:00