Commit Graph

91 Commits

Author SHA1 Message Date
Sage Weil
05b3ce2585 qa/tasks/cephadm: add apply() method/task
Signed-off-by: Sage Weil <sage@newdream.net>
2021-04-05 11:44:22 -05:00
Sage Weil
a17593a177 qa/tasks/cephadm: use 'orch apply mon' to deploy mons
The 'orch daemon add ...' command is not idempotent and can cause
duplicate (and failing) attempts to add the same mon.

Maintain the ability to add mons the old way for the benefit of testing
upgrades from early octopus versions of cephadm.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-22 16:28:40 -05:00
Sage Weil
0a139c1ffc qa/tasks/cephadm.py: don't set mgr count to +1
Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-11 14:16:15 -05:00
Sage Weil
1bfb0cd84b qa/tasks/cephadm: drop realm.zone convention for rgw
Note that cephadm.py will no longer do anything with rgw realms and
zones.  That means that the setup of rgw roles here is only useful
for the default zone and a non-multisite config.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-09 14:29:43 -05:00
Sage Weil
208407c5c0 qa/tasks/cephadm: remove mirror code
This is being done from ansible now.  Also, it breaks when
the conf file has unqualified-search-registries but not 'registry'
entries.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-03-04 07:39:15 -06:00
Neha Ojha
3ced450cdd Revert "qa/tasks/cephadm: use cluster.sh and remote.sh for better readibility"
This reverts commit 6d7ba1180e.

Fixes: https://tracker.ceph.com/issues/49228
Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-02-09 22:15:37 +00:00
Sage Weil
5842058d12 qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git.  However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.

Signed-off-by: Sage Weil <sage@newdream.net>
2021-01-29 11:45:02 -06:00
Sage Weil
9eef1bcced qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm
Older cephadm is not smart enough to not combine --cap-add=SYS_PTRACE
and --privileged, which some version of podman cannot handle.

For upgrades, leave off the allow_ptrace behavior since we may be starting
on one of those old versions.

See also https://tracker.ceph.com/issues/46429

Fixes: https://tracker.ceph.com/issues/48142
Signed-off-by: Sage Weil <sage@newdream.net>
2021-01-20 16:40:17 -06:00
Neha Ojha
f3af610721 qa/tasks/cephadm.py: do not create rbd pool by default
rados/cephadm/smoke* does not use the install task and the adjust-ulimits
dependency is met as a part of it. create_rbd_pool needs adjust-ulimits,
so for now we will disable create_rbd_pool by default and only set it
to true for the upgrade suite.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-01-06 17:35:15 +00:00
Neha Ojha
276a4fb8b0 qa/tasks/cephadm.py: create RBD pool
The ceph task already does this and a bunch of tests rely on this pool
being already present. Can be disabled by setting create_rbd_pool to False.

Signed-off-by: Neha Ojha <nojha@redhat.com>
2021-01-05 21:32:41 +00:00
Yuri Weinstein
5ab1e53221
Merge pull request #38707 from ceph/wip-yuriw-octopus-x-4-master
qa/tests: initial check in for octopus-x/parallel tests (try 2)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
2020-12-23 13:09:51 -08:00
Josh Durgin
c79fa6d780 qa/tasks/cephadm: include first mgr host when applying mgrs
This prevents the first mgr from being shut down due to lack of
appropriate placements.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
2020-12-23 11:30:54 -08:00
Dan Mick
f30c55a242 qa/distros: configure podman to pull from docker-mirror
This also requires a change to cephadm.py to search for
registry entries for docker.io by 'prefix' or 'location',
and to only add the mirror if a mirror isn't already present

Signed-off-by: Dan Mick <dmick@redhat.com>
2020-12-18 09:53:50 +00:00
Deepika Upadhyay
4da3b23b89 qa/tasks/ceph: rename s/update_archive_setting/update_info_yaml
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-12-03 22:08:52 +05:30
Deepika Upadhyay
6d7ba1180e qa/tasks/cephadm: use cluster.sh and remote.sh for better readibility
in extention for: 9536625558

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-12-03 21:23:12 +05:30
Deepika Upadhyay
22b832e94e qa/tasks/cephadm: refactor
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-12-03 21:23:12 +05:30
Deepika Upadhyay
2f0209ae43 qa/tasks/cephadm: update_info_yaml() only if ctx.archive is valid
related to: 6cd8e4b55883b2d282f101f11521a2759b3db564
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-12-03 21:23:11 +05:30
Kyr Shatskyy
f75b00984e qa/task/cephadm: run cephadm only on bootstrap_remote
Because not all the node in teuthology config might
have any cephadm related role we should not execute
cephadm on random node.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-11-11 23:17:59 +01:00
Sebastian Wagner
130c7a438e qa/cephadm: Also fetch iscsi logs
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-09-11 14:17:09 +02:00
Sebastian Wagner
430f2c5132 qa/cephadm: Add iSCSI
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-09-11 14:17:09 +02:00
Kyr Shatskyy
5e9cd8624e qa/tasks/cephadm: use remote.read_file instead of misc.get_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:16 +02:00
Kyr Shatskyy
bdb3c8a23d qa/tasks/cephadm: deprecate misc.sudo_write_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-09-04 00:02:15 +02:00
Neha Ojha
fb44c2f500
Merge pull request #36527 from neha-ojha/wip-whitelist-blacklist
Replace 'whitelist' and 'blacklist' language

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
2020-08-25 12:24:36 -07:00
Sage Weil
2ee9365d0b qa: log-whitelist -> log-ignorelist
Signed-off-by: Sage Weil <sage@newdream.net>
2020-08-24 19:53:08 +00:00
Shraddha Agrawal
d2ff5c5eb6 qa/tasks/cephadm.py: add ceph logs directory in job's info.yaml
This commit adds the file path of ceph log directories to the job's
info.yaml log file. The motivation behind this is, in case of job
timeout, the logs would still be tranferred to teuthology host
before nuking test machines using these ceph log directory paths in
job's info.yaml log file.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
2020-08-19 17:52:06 +05:30
Kefu Chai
2a377b01fd qa/tasks/cephadm: drop python2 support
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-07-21 19:52:11 +08:00
Georgios Kyratsas
7e50ee3496
Merge pull request #35927 from gekios/wip-cephadm-image-fix
qa/tasks: fix cephadm config image check
2020-07-09 14:47:37 +02:00
Georgios Kyratsas
390ca4d92f qa/tasks: fix cephadm config image check
Check container_image_name only if ceph cluster image is not pre-defined in config.
We shouldn't care about container_image_name if there cephadm or ceph already have image defined.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>
2020-07-09 10:28:49 +02:00
Deepika Upadhyay
b0a14d6e1e docker/podman: tag crimson flavor container as 'sha-1-flavor'
see also: ceph/ceph-container#1704

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
2020-07-06 16:19:28 +05:30
Kefu Chai
2fa726b88c qa/tasks: flake8 fixes
Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-06-23 23:00:56 +08:00
Kyr Shatskyy
d8edde0c53 qa/tasks/cephadm: setup site based container registry
Add containers registry mirror for 'docker.io' to the config
file /etc/containers/registries.conf

Since we need site based config, so each teuthology instance
could have own containers' mirror, store corresponding data
in /etc/teuthology.yaml under 'overrides/cephadm' task specific
section.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
2020-06-10 17:03:58 +02:00
Sebastian Wagner
115492f28f
Merge pull request #35069 from sebastian-philipp/qa-cephadm-add-rgw-and-iscsi
qa/cephadm: Add RGW

Reviewed-by: Michael Fritch <mfritch@suse.com>
2020-06-03 09:50:04 +02:00
Sebastian Wagner
eddb6354e1 qa/cephadm: FileNotFoundError doesn't exist in py2
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-05-29 16:04:20 +02:00
Sebastian Wagner
ffef64d046 qa/cephadm: Add RGW
Fixes: https://tracker.ceph.com/issues/43911

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-05-29 13:33:05 +02:00
Georgios Kyratsas
fabcbeab2c qa/tasks/cephadm: Add check fox already set image
Adding a check for already bootstrapped clusters where the image is
already set to avoid overriding it.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>
2020-05-28 17:15:59 +02:00
Sebastian Wagner
9310b6e278 qa/cephadm: Docker doesn't ship a registries.conf
I guess not using the mirror for docker based tests is ok for now.

For adding docker support, we need to:

1. change the docker config
2. restart the docker daemon

Note that Docker's config only supports to mirror the docker.io registry.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-05-27 13:17:58 +02:00
Sebastian Wagner
d6edd8199f qa/cephadm: Add local registry mirror
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
2020-05-26 17:38:39 +02:00
Kefu Chai
fdb2d1e62c
Merge pull request #35104 from varshar16/wip-fix-roleless-mon-issue
qa/cephadm: Add roleless mon names to ctx.cluster.remotes role dict

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Georgios Kyratsas <gkyratsas@suse.com>
2020-05-22 12:44:09 +08:00
Kefu Chai
d67b0f1e3a qa/tasks/cephadm: ignore tarfile.ReadError
there is chance that /var/log/ceph (and other directories) is not
created when cephadm fails, but we always try to collect /var/log/ceph
by tar the directory on the test node and transfer it back for archiving
it. when we fail to do so, an exception is thrown, and this exception is
in turn logged as the cause of the failure instead of the one which
fails the test.

in this change, `tarfile.TarError`s is caught and ignored if it is
thrown by `misc.pull_directory()`, as normally, we don't consider it a fatal
failure when `misc.pull_directory()` fails because of `tar`. the
failures for the test are.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-05-19 19:10:56 +08:00
Varsha Rao
58eaeb53e4 qa/cephadm: Add roleless mon names to ctx.cluster.remotes role dict
Fixes: https://tracker.ceph.com/issues/45584
Signed-off-by: Varsha Rao <varao@redhat.com>
2020-05-19 09:57:27 +00:00
Georgios Kyratsas
35e158605e qa/tasks/cephadm: Add support for raw disks
Currently tests on raw disks are failing due to stripping '/dev/' from
device name. (ie https://tracker.ceph.com/issues/45383#change-165721)

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>
2020-05-14 16:54:57 +02:00
Georgios Kyratsas
a002b28bc1 qa/tasks/cephadm.py: Support bootstrapped clusters
Add the possibility of being able to run cephadm.py on clusters that
were already bootstrapped by some other task/way so you can deploy the
rest of the daemons/services.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>
2020-05-14 16:54:30 +02:00
Kefu Chai
6eb57a2a4f qa/tasks/cephadm.py: decode data returned by teuthology.get_file()
it's used as part of the command line sent to shell, so we need to
decode it first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-07 23:06:20 +08:00
Kefu Chai
8bfe977854 qa/tasks: use StringIO for capturing string output
see d8d44ed156

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-04-07 21:51:22 +08:00
Sage Weil
9e609c9ed7 qa/tasks/cephadm: add 'roleless' mode
Allow cephadm to start up with roles like:

roles:
- - host.a
  - client.0
  - osd.0
  - osd.1
- - host.b
  - osd.2
  - osd.3

Cephadm will pick the mon names (based on host) and provision all
services by default.

The cephadm task can still provision other daemons, but it may
fight with mgr/cephadm.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-04-02 23:16:42 +00:00
Sage Weil
65ad142d59 qa/tasks/cephadm: no need to explicitly reconfig
cephadm does this magically now.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-03-27 11:49:29 -05:00
Sage Weil
29d2d7d64b qa/tasks/cephadm: fetch final ceph.conf
Otherwise we'll distribute the initial ceph.conf, which only has the
bootstrap mon in it.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-03-27 11:49:29 -05:00
Sage Weil
4c78dfe26f qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes
Revert part of 96220c0c05 so that we still
distribute a *final* ceph.conf and admin keyring to all nodes, right after
all of the mons are up.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-03-27 10:43:18 -05:00
Kefu Chai
947a74349d qa: import with full path
to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>
2020-03-24 18:27:55 +08:00
Sage Weil
d7de2abc2f cephadm: add allow_ptrace option to enable SYS_PTRACE
In any environments it is helpful to have SYS_PTRACE so that you can
gdb attach or strace a daemon.

Leave this off by default so that the container is more secure.

Enable this in teuthology and vstart.

Signed-off-by: Sage Weil <sage@redhat.com>
2020-03-16 21:04:22 -05:00