mirror of
https://github.com/ceph/ceph
synced 2025-02-03 17:03:51 +00:00
0f2489406a
for 2 reasons: * sphinx renders codeblock using python syntax highlighting by default, so it's not surprising that it highlight keywords like "export" in command line samples. so to render command line code blocks, we'd better specify the syntax explicitly for better rendering result. * with the help of "prompt" directive, user is able to copy and paste the command without the prompt. for instance, with the default "::" directive, user will copy "$ ceph df", which is not very convenient, but with "prompt" directive, user only copies "ceph df". Signed-off-by: Kefu Chai <kchai@redhat.com>
290 lines
10 KiB
ReStructuredText
290 lines
10 KiB
ReStructuredText
Running Tests in the Cloud
|
|
==========================
|
|
|
|
In this chapter, we will explain in detail how use an OpenStack
|
|
tenant as an environment for Ceph `integration testing`_.
|
|
|
|
Assumptions and caveat
|
|
----------------------
|
|
|
|
We assume that:
|
|
|
|
1. you are the only person using the tenant
|
|
2. you have the credentials
|
|
3. the tenant supports the ``nova`` and ``cinder`` APIs
|
|
|
|
Caveat: be aware that, as of this writing (July 2016), testing in
|
|
OpenStack clouds is a new feature. Things may not work as advertised.
|
|
If you run into trouble, ask for help on `IRC`_ or the `Mailing list`_, or
|
|
open a bug report at the `ceph-workbench bug tracker`_.
|
|
|
|
.. _`ceph-workbench bug tracker`: http://ceph-workbench.dachary.org/root/ceph-workbench/issues
|
|
|
|
Prepare tenant
|
|
--------------
|
|
|
|
If you have not tried to use ``ceph-workbench`` with this tenant before,
|
|
proceed to the next step.
|
|
|
|
To start with a clean slate, login to your tenant via the Horizon dashboard
|
|
and:
|
|
|
|
* terminate the ``teuthology`` and ``packages-repository`` instances, if any
|
|
* delete the ``teuthology`` and ``teuthology-worker`` security groups, if any
|
|
* delete the ``teuthology`` and ``teuthology-myself`` key pairs, if any
|
|
|
|
Also do the above if you ever get key-related errors ("invalid key", etc.)
|
|
when trying to schedule suites.
|
|
|
|
Getting ceph-workbench
|
|
----------------------
|
|
|
|
Since testing in the cloud is done using the ``ceph-workbench ceph-qa-suite``
|
|
tool, you will need to install that first. It is designed
|
|
to be installed via Docker, so if you don't have Docker running on your
|
|
development machine, take care of that first. You can follow `the official
|
|
tutorial <https://docs.docker.com/engine/installation/>`_ to install if
|
|
you have not installed yet.
|
|
|
|
Once Docker is up and running, install ``ceph-workbench`` by following the
|
|
`Installation instructions in the ceph-workbench documentation
|
|
<http://ceph-workbench.readthedocs.io/en/latest/#installation>`_.
|
|
|
|
Linking ceph-workbench with your OpenStack tenant
|
|
-------------------------------------------------
|
|
|
|
Before you can trigger your first teuthology suite, you will need to link
|
|
``ceph-workbench`` with your OpenStack account.
|
|
|
|
First, download a ``openrc.sh`` file by clicking on the "Download OpenStack
|
|
RC File" button, which can be found in the "API Access" tab of the "Access
|
|
& Security" dialog of the OpenStack Horizon dashboard.
|
|
|
|
Second, create a ``~/.ceph-workbench`` directory, set its permissions to
|
|
700, and move the ``openrc.sh`` file into it. Make sure that the filename
|
|
is exactly ``~/.ceph-workbench/openrc.sh``.
|
|
|
|
Third, edit the file so it does not ask for your OpenStack password
|
|
interactively. Comment out the relevant lines and replace them with
|
|
something like::
|
|
|
|
.. prompt:: bash $
|
|
|
|
export OS_PASSWORD="aiVeth0aejee3eep8rogho3eep7Pha6ek"
|
|
|
|
When ``ceph-workbench ceph-qa-suite`` connects to your OpenStack tenant for
|
|
the first time, it will generate two keypairs: ``teuthology-myself`` and
|
|
``teuthology``.
|
|
|
|
.. If this is not the first time you have tried to use
|
|
.. ``ceph-workbench ceph-qa-suite`` with this tenant, make sure to delete any
|
|
.. stale keypairs with these names!
|
|
|
|
Run the dummy suite
|
|
-------------------
|
|
|
|
You are now ready to take your OpenStack teuthology setup for a test
|
|
drive
|
|
|
|
.. prompt:: bash $
|
|
|
|
ceph-workbench ceph-qa-suite --suite dummy
|
|
|
|
Be forewarned that the first run of ``ceph-workbench ceph-qa-suite`` on a
|
|
pristine tenant will take a long time to complete because it downloads a VM
|
|
image and during this time the command may not produce any output.
|
|
|
|
The images are cached in OpenStack, so they are only downloaded once.
|
|
Subsequent runs of the same command will complete faster.
|
|
|
|
Although ``dummy`` suite does not run any tests, in all other respects it
|
|
behaves just like a teuthology suite and produces some of the same
|
|
artifacts.
|
|
|
|
The last bit of output should look something like this::
|
|
|
|
pulpito web interface: http://149.202.168.201:8081/
|
|
ssh access : ssh -i /home/smithfarm/.ceph-workbench/teuthology-myself.pem ubuntu@149.202.168.201 # logs in /usr/share/nginx/html
|
|
|
|
What this means is that ``ceph-workbench ceph-qa-suite`` triggered the test
|
|
suite run. It does not mean that the suite run has completed. To monitor
|
|
progress of the run, check the Pulpito web interface URL periodically, or
|
|
if you are impatient, ssh to the teuthology machine using the ssh command
|
|
shown and do
|
|
|
|
.. prompt:: bash $
|
|
|
|
tail -f /var/log/teuthology.*
|
|
|
|
The `/usr/share/nginx/html` directory contains the complete logs of the
|
|
test suite. If we had provided the ``--upload`` option to the
|
|
``ceph-workbench ceph-qa-suite`` command, these logs would have been
|
|
uploaded to http://teuthology-logs.public.ceph.com.
|
|
|
|
Run a standalone test
|
|
---------------------
|
|
|
|
The standalone test explained in `Reading a standalone test`_ can be run
|
|
with the following command
|
|
|
|
.. prompt:: bash $
|
|
|
|
ceph-workbench ceph-qa-suite --suite rados/singleton/all/admin-socket.yaml
|
|
|
|
This will run the suite shown on the current ``master`` branch of
|
|
``ceph/ceph.git``. You can specify a different branch with the ``--ceph``
|
|
option, and even a different git repo with the ``--ceph-git-url`` option. (Run
|
|
``ceph-workbench ceph-qa-suite --help`` for an up-to-date list of available
|
|
options.)
|
|
|
|
The first run of a suite will also take a long time, because ceph packages
|
|
have to be built, first. Again, the packages so built are cached and
|
|
``ceph-workbench ceph-qa-suite`` will not build identical packages a second
|
|
time.
|
|
|
|
Interrupt a running suite
|
|
-------------------------
|
|
|
|
Teuthology suites take time to run. From time to time one may wish to
|
|
interrupt a running suite. One obvious way to do this is::
|
|
|
|
.. prompt:: bash $
|
|
|
|
ceph-workbench ceph-qa-suite --teardown
|
|
|
|
This destroys all VMs created by ``ceph-workbench ceph-qa-suite`` and
|
|
returns the OpenStack tenant to a "clean slate".
|
|
|
|
Sometimes you may wish to interrupt the running suite, but keep the logs,
|
|
the teuthology VM, the packages-repository VM, etc. To do this, you can
|
|
``ssh`` to the teuthology VM (using the ``ssh access`` command reported
|
|
when you triggered the suite -- see `Run the dummy suite`_) and, once
|
|
there
|
|
|
|
.. prompt:: bash $
|
|
|
|
sudo /etc/init.d/teuthology restart
|
|
|
|
This will keep the teuthology machine, the logs and the packages-repository
|
|
instance but nuke everything else.
|
|
|
|
Upload logs to archive server
|
|
-----------------------------
|
|
|
|
Since the teuthology instance in OpenStack is only semi-permanent, with
|
|
limited space for storing logs, ``teuthology-openstack`` provides an
|
|
``--upload`` option which, if included in the ``ceph-workbench ceph-qa-suite``
|
|
command, will cause logs from all failed jobs to be uploaded to the log
|
|
archive server maintained by the Ceph project. The logs will appear at the
|
|
URL::
|
|
|
|
http://teuthology-logs.public.ceph.com/$RUN
|
|
|
|
where ``$RUN`` is the name of the run. It will be a string like this::
|
|
|
|
ubuntu-2016-07-23_16:08:12-rados-hammer-backports---basic-openstack
|
|
|
|
Even if you don't providing the ``--upload`` option, however, all the logs can
|
|
still be found on the teuthology machine in the directory
|
|
``/usr/share/nginx/html``.
|
|
|
|
Provision VMs ad hoc
|
|
--------------------
|
|
|
|
From the teuthology VM, it is possible to provision machines on an "ad hoc"
|
|
basis, to use however you like. The magic incantation is::
|
|
|
|
.. prompt:: bash $
|
|
|
|
teuthology-lock --lock-many $NUMBER_OF_MACHINES \
|
|
--os-type $OPERATING_SYSTEM \
|
|
--os-version $OS_VERSION \
|
|
--machine-type openstack \
|
|
--owner $EMAIL_ADDRESS
|
|
|
|
The command must be issued from the ``~/teuthology`` directory. The possible
|
|
values for ``OPERATING_SYSTEM`` AND ``OS_VERSION`` can be found by examining
|
|
the contents of the directory ``teuthology/openstack/``. For example
|
|
|
|
.. prompt:: bash $
|
|
|
|
teuthology-lock --lock-many 1 --os-type ubuntu --os-version 16.04 \
|
|
--machine-type openstack --owner foo@example.com
|
|
|
|
When you are finished with the machine, find it in the list of machines
|
|
|
|
.. prompt:: bash $
|
|
|
|
openstack server list
|
|
|
|
to determine the name or ID, and then terminate it with
|
|
|
|
.. prompt:: bash $
|
|
|
|
openstack server delete $NAME_OR_ID
|
|
|
|
Deploy a cluster for manual testing
|
|
-----------------------------------
|
|
|
|
The `teuthology framework`_ and ``ceph-workbench ceph-qa-suite`` are
|
|
versatile tools that automatically provision Ceph clusters in the cloud and
|
|
run various tests on them in an automated fashion. This enables a single
|
|
engineer, in a matter of hours, to perform thousands of tests that would
|
|
keep dozens of human testers occupied for days or weeks if conducted
|
|
manually.
|
|
|
|
However, there are times when the automated tests do not cover a particular
|
|
scenario and manual testing is desired. It turns out that it is simple to
|
|
adapt a test to stop and wait after the Ceph installation phase, and the
|
|
engineer can then ssh into the running cluster. Simply add the following
|
|
snippet in the desired place within the test YAML and schedule a run with the
|
|
test::
|
|
|
|
tasks:
|
|
- exec:
|
|
client.0:
|
|
- sleep 1000000000 # forever
|
|
|
|
(Make sure you have a ``client.0`` defined in your ``roles`` stanza or adapt
|
|
accordingly.)
|
|
|
|
The same effect can be achieved using the ``interactive`` task::
|
|
|
|
tasks:
|
|
- interactive
|
|
|
|
By following the test log, you can determine when the test cluster has entered
|
|
the "sleep forever" condition. At that point, you can ssh to the teuthology
|
|
machine and from there to one of the target VMs (OpenStack) or teuthology
|
|
worker machines machine (Sepia) where the test cluster is running.
|
|
|
|
The VMs (or "instances" in OpenStack terminology) created by
|
|
``ceph-workbench ceph-qa-suite`` are named as follows:
|
|
|
|
``teuthology`` - the teuthology machine
|
|
|
|
``packages-repository`` - VM where packages are stored
|
|
|
|
``ceph-*`` - VM where packages are built
|
|
|
|
``target*`` - machines where tests are run
|
|
|
|
The VMs named ``target*`` are used by tests. If you are monitoring the
|
|
teuthology log for a given test, the hostnames of these target machines can
|
|
be found out by searching for the string ``Locked targets``::
|
|
|
|
2016-03-20T11:39:06.166 INFO:teuthology.task.internal:Locked targets:
|
|
target149202171058.teuthology: null
|
|
target149202171059.teuthology: null
|
|
|
|
The IP addresses of the target machines can be found by running ``openstack
|
|
server list`` on the teuthology machine, but the target VM hostnames (e.g.
|
|
``target149202171058.teuthology``) are resolvable within the teuthology
|
|
cluster.
|
|
|
|
.. _Integration testing: ../tests-integration-tests
|
|
.. _IRC: ../essentials/#irc
|
|
.. _Mailing List: ../essentials/#mailing-list
|
|
.. _Reading A Standalone Test: ../testing-integration-tests/#reading-a-standalone-test
|
|
.. _teuthology framework: https://github.com/ceph/teuthology
|