Commit Graph

150 Commits

Author SHA1 Message Date
Sage Weil
86a27e5a9a Merge PR #33047 into master
* refs/pull/33047/head:
	qa/rgw: exercise DeleteRange in test_bucket_index_log_trim

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>
2020-03-04 09:58:45 -06:00
Casey Bodley
08e0a4289d qa/rgw: verify suite selects a random bucket sharding configuration
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-02-28 14:38:13 -05:00
Casey Bodley
2aa526b96a qa/rgw: exercise DeleteRange in test_bucket_index_log_trim
test_bucket_index_log_trim doesn't create enough keys to exceed the
default threshold, so set it to 0

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-02-03 11:46:59 -05:00
Casey Bodley
d486b5bc45 qa/rgw: test with non-default rgw-obj-stripe-size
each job will select one of the striping strategies at random

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-02-03 11:24:52 -05:00
Sage Weil
acd0e31177 qa/suites/rgw/website: run test on ubuntu
The dnsmasq package on centos 8.0 is broken, see
https://tracker.ceph.com/issues/43744

For now, run this test on ubuntu.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-01-22 15:17:57 -06:00
Ali Maredia
2552974fe7 qa: pin rgw/verify to 8.0
Signed-off-by: Ali Maredia <amaredia@redhat.com>
2020-01-21 13:02:42 -05:00
Sage Weil
5c1e0e5f1f qa/suite/rgw/verify: valgrind on centos again!
Signed-off-by: Sage Weil <sage@redhat.com>
2020-01-19 18:47:13 -06:00
Kefu Chai
7b0973e33b
Merge pull request #32624 from alimaredia/wip-s3-tests-python-3
qa/suites: use s3-tests with python3 support

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
2020-01-17 23:47:51 +08:00
Casey Bodley
6a6d4a4ffa
Merge pull request #32620 from cbodley/wip-qa-rgw-hadoop-versions
qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2020-01-14 13:36:29 -05:00
Casey Bodley
b10873142e qa/rgw: remove test against hadoop v2.8.5
the hadoop branch rel/release-2.8.5 fails to build with:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:37 min
[INFO] Finished at: 2020-01-14T13:09:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) on project hadoop-aws: An Ant BuildException has occured: Unable to create javax script engine for javascript

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-01-14 09:42:54 -05:00
Casey Bodley
5bb03f143e qa/rgw: remove test against hadoop trunk
hadoop's trunk fails to build with many 'package does not exist' errors:

[INFO] 100 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  48.792 s
[INFO] Finished at: 2020-01-14T13:07:36Z
[INFO] ------------------------------------------------------------------------

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-01-14 09:39:33 -05:00
Ali Maredia
0ed77d1c1a qa: remove s3readwrite & s3roundtrip tasks
As a result of the s3-tests python3 port, the
s3readwrite & s3roundtrip testing files were
deleted

Signed-off-by: Ali Maredia <amaredia@redhat.com>
2020-01-13 15:46:31 -05:00
Ali Maredia
07a8ac5778 qa: s3tests/s3tests-website tasks no longer only use ubunutu
Signed-off-by: Ali Maredia <amaredia@redhat.com>
2020-01-13 15:46:28 -05:00
Casey Bodley
abe5277269
Merge pull request #32188 from cbodley/wip-qa-rgw-obj-manifest
qa/rgw: add rgw_obj and throttle tests to rgw verify suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
2020-01-13 14:45:19 -05:00
Casey Bodley
cfa5922fd8 qa/rgw: add rgw_obj and throttle tests to rgw verify suite
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-01-09 16:51:37 -05:00
Casey Bodley
f51a7cae7d qa/rgw: remove failing radosgw_admin_rest from multisite suite
this was added to test that admin apis forward relevent requests to the
master zone, but radosgw_admin_rest.py tries to create an admin user
with 'radosgw-admin user create'. this fails with:

Please run the command on master zone. Performing this operation on
non-master zone leads to inconsistent metadata between zones
Are you sure you want to go ahead? (requires --yes-i-really-mean-it)

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2020-01-08 10:27:33 -05:00
Patrick Donnelly
4562823a19
qa: add .qa link
Continuation of 716db6e2fd.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
2019-12-19 14:31:09 -08:00
Sage Weil
aa9272d872 qa/tasks/swift: remove swift tests
- This is an ancient swift version
- The tempest tests are newer and show provide similar coverage
- It somehow broke with the py3 transition

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-19 12:21:21 -06:00
Sage Weil
9dd7ed8860 Revert "qa/suites/rgw: ragweed on ubuntu"
This reverts commit 73c2779524.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-17 14:40:34 -06:00
Sage Weil
ce308a8d0d Revert "qa/suites/rgw: disable remaining ragweed test"
This reverts commit 5a80fbe505.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-17 14:40:21 -06:00
Sage Weil
ea21829ae7 qa/suites/rgw: disable valgrind for tests that require py2/ubuntu
Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-17 12:26:15 -06:00
Sage Weil
5a80fbe505 qa/suites/rgw: disable remaining ragweed test
Needs to be py3 since it relies on the python bindings.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-17 10:47:49 -06:00
Sage Weil
2c1dde3b2a qa/suites/rgw: pin swift tests to py2/ubuntu
Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-16 14:48:36 -06:00
Sage Weil
73c2779524 qa/suites/rgw: ragweed on ubuntu
ragweed uses nosetests; stick to py2 for now.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-16 11:47:48 -06:00
Sage Weil
5b4e9de5fd qa/suites: run s3tests on ubuntu
Note that this does not capture the rgw/website test on valgrind, since
valgrind is centos only.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-12-16 11:37:10 -06:00
Ali Maredia
73d9131839 qa: add force-branch to suites running s3readwrite & s3roundtrip tasks
Signed-off-by: Ali Maredia <amaredia@redhat.com>
2019-12-12 16:09:07 -05:00
Sergio de Carvalho
f9d7b685c1 rgw: improvements to SSE-KMS with Vault
* Minor improvements to Vault documentation
* Add teuthology tests for Transit secrets engine
* Add unit tests for KV secrets engine, minor improvements to Transit
  secrets engine
* use string_view::npos instead of string::npos

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
2019-11-28 09:19:39 +00:00
Sage Weil
61ba2d7b66 Merge PR #31677 into master
* refs/pull/31677/head:
	qa/standalone/ceph-helpers.sh: remove osd down check
	qa/standalone/ceph-helpers.sh: destroy_osd: mark osd down
	osd: add osd_fast_shutdown option (default true)

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
2019-11-25 08:54:45 -06:00
Sage Weil
cf352c3ac0 osd: add osd_fast_shutdown option (default true)
If we get a SIGINT or SIGTERM or are deleted from the OSDMap, do a fast
shutdown by exiting immediately.  This has a few important benefits:

 - We immediately stop responding (binding) to any sockets, which means
   other OSDs will immediately decide we are down (and dead!).  This
   minimizes IO interruption.
 - We avoid the complex "clean" shutdown process, which is historically a
   source of bugs.

In reality, the only purpose of the "clean" shutdown is to try to tear down
everything in memory so we can do memory leak checking with valgrind.  Set
this option to false for valgrind QA runs so we can still do that.

Not that with the new read leases in octopus, we rely on the default
behavior that a ECONNREFUSED is taken to mean that the OSD is fully dead,
so that we don't have to wait for any leases to time out.  This works in
sane environments with normal IP networks, but that behavior could
conceivably be a bad idea if there are some weird network shenanigans
going on.  If osd_fast_fail_on_connection_refused were disabled, then this
fast shutdown procedure might be *worse* than the clean shutdown because
we would have to wait for the heartbeat timeout.

Signed-off-by: Sage Weil <sage@redhat.com>
2019-11-15 09:31:50 -06:00
Sergio de Carvalho
2650ebe8af rgw: improvements to SSE-KMS with Vault
* add 'rgw crypt vault prefix' config setting to allow restricting
  secret space in Vault where RGW can retrieve keys from
* refuse Vault token file if permissions are too open
* improve concatenation of URL paths to avoid constructing an invalid
  URL (missing or double '/')
* doc: clarify SSE-KMS keys must be 256-bit long and base64 encoded,
  document Vault policies and tokens, plus other minor doc improvements
* qa: check SHA256 signature of Vault zip download
* qa: fix teuthology tests broken by previous PR which made SSE-KMS
  backend default to Barbican

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
2019-11-12 13:51:25 +00:00
Casey Bodley
d5863f5c2b qa/rgw: use 'testing' kms backend for other rgw subsuites
resolves test failures under rgw/{multifs,thrash,website} similar to
https://github.com/ceph/ceph/pull/30940

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-05 09:02:50 -05:00
Casey Bodley
ad4ff5f948 qa/rgw: use 'testing' kms backend for multisite tests
a missing piece from https://github.com/ceph/ceph/pull/30940

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-11-04 12:49:07 -05:00
Casey Bodley
250a65e045
Merge pull request #30997 from cbodley/wip-qa-rgw-objectstores
qa/rgw: drop some objectstore types

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
2019-10-23 11:37:32 -04:00
Casey Bodley
604db96bbb
Merge pull request #28421 from pritha-srivastava/wip-rgw-omap-offload
rgw: add cls_queue and cls_rgw_gc for omap offload

Reviewed-by: Casey Bodley <cbodley@redhat.com>
2019-10-23 09:59:28 -04:00
Casey Bodley
0e76d40aa1 test/rgw: run ceph_test_rgw_gc_log in rgw verify suite
since it requires a running ceph cluster, it can't run in 'make check'
as a unittest. add it to the rgw/verify suite instead

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-10-19 13:28:18 +05:30
Casey Bodley
85a37896b8 qa/rgw: drop some objectstore types
use the subset of objectstore configurations from .qa/objectstore_cephfs
instead of .qa/objectstore

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-10-18 13:20:20 -04:00
Casey Bodley
f3f68d8b6d qa/rgw: use config overrides for vault kms backend
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-10-15 11:30:22 -04:00
Casey Bodley
21526396f6 qa/rgw: use config overrides for barbican kms backend
except for the barbican endpoint, which isn't known until the barbican
task runs

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-10-15 11:30:18 -04:00
Casey Bodley
aa283c5464 qa/rgw: use config overrides for 'testing' kms backend
and configure the 'testing' backend for the rgw/verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-10-15 11:30:15 -04:00
Sage Weil
71d74aa8c6 qa: more tries for mon tell when injecting msgr failures
With failure injection the default 2 tries isn't quite enough

Signed-off-by: Sage Weil <sage@redhat.com>
2019-10-11 14:16:42 -05:00
Ali Maredia
d4872ce97a
Merge pull request #29783 from scarvalhojr/ssevault
rgw: add SSE-KMS with Vault using token auth
2019-10-06 22:33:36 -04:00
David Zafman
ded58ef91d test: Ignore OSD_SLOW_PING_TIME* if injecting socket failures
Fixes: https://tracker.ceph.com/issues/41743

Signed-off-by: David Zafman <dzafman@redhat.com>
2019-10-03 09:09:10 -07:00
Andrea Baglioni
955d3b517a rgw: add SSE-KMS with Vault using token auth
Restructure SSE-KMS tests which now has 3 scenarios for each KMS
backend: Barbican, Vault, and testing (keys stored in ceph.conf).

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>
2019-10-02 16:38:01 +01:00
Casey Bodley
97f1be5d3a qa/rgw: rgw/website suite uses subdomain calling format
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-17 08:24:52 -04:00
Casey Bodley
bb56ac01c2 qa/rgw: add new rgw/website suite for static website tests
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-17 08:24:51 -04:00
Casey Bodley
8c801f073d
Merge pull request #30218 from cbodley/wip-qa-rgw-barbican
qa/rgw: add integration test for sse-kms with barbican

Reviewed-by: Ali Maredia <amaredia@redhat.com>
2019-09-17 08:17:30 -04:00
Casey Bodley
71a87afb5d qa/barbican: pin barbican to 5.0.1 release
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Casey Bodley
53cd9b8dc6 qa/rgw: crypt suite uses clusters/fixed-1.yaml
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Casey Bodley
1c5a12fd9e qa/rgw: abstract out barbican under 2-kms/
Signed-off-by: Casey Bodley <cbodley@redhat.com>
2019-09-13 15:04:43 -04:00
Adam Kupczyk
a5ec971402 tasks/barbican: Added task to deploy and configure barbican
Fixed keystone, rgw and s3tests tasks to work with it.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
2019-09-13 15:04:43 -04:00