2020-01-06 07:40:50 +00:00
|
|
|
Running Integration Tests using Teuthology
|
|
|
|
==========================================
|
|
|
|
|
|
|
|
Getting binaries
|
|
|
|
----------------
|
|
|
|
To run integration tests using teuthology, you need to have Ceph binaries
|
|
|
|
built for your branch. Follow these steps to initiate the build process -
|
|
|
|
|
|
|
|
#. Push the branch to `ceph-ci`_ repository. This triggers the process of
|
|
|
|
building the binaries.
|
|
|
|
|
|
|
|
#. To confirm that the build process has been initiated, spot the branch name
|
|
|
|
at `Shaman`_. Little after the build process has been initiated, the single
|
|
|
|
entry with your branch name would multiply, each new entry for a different
|
|
|
|
combination of distro and flavour.
|
|
|
|
|
|
|
|
#. Wait until the packages are built and uploaded, and the repository offering
|
|
|
|
them are created. This is marked by colouring the entries for the branch
|
|
|
|
name green. Preferably, wait until each entry is coloured green. Usually,
|
|
|
|
it takes around 2-3 hours depending on the availability of the machines.
|
|
|
|
|
|
|
|
.. note:: Branch to be pushed on ceph-ci can be any branch, it shouldn't
|
|
|
|
necessarily be a PR branch.
|
|
|
|
|
|
|
|
.. note:: In case you are pushing master or any other standard branch, check
|
|
|
|
`Shaman`_ beforehand since it already might have builds ready for it.
|
|
|
|
|
|
|
|
Triggering Tests
|
|
|
|
----------------
|
|
|
|
After building is complete, proceed to trigger tests -
|
|
|
|
|
|
|
|
#. Log in to the teuthology machine::
|
|
|
|
|
|
|
|
ssh <username>@teuthology.front.sepia.ceph.com
|
|
|
|
|
|
|
|
This would require Sepia lab access. To know how to request it, see: https://ceph.github.io/sepia/adding_users/
|
|
|
|
|
|
|
|
#. Next, get teuthology installed. Run the first set of commands in
|
|
|
|
`Running Your First Test`_ for that. After that, activate the virtual
|
|
|
|
environment in which teuthology is installed.
|
|
|
|
|
|
|
|
#. Run the ``teuthology-suite`` command::
|
|
|
|
|
|
|
|
teuthology-suite -v -m smithi -c wip-devname-feature-x -s fs -p 110 --filter "cephfs-shell"
|
|
|
|
|
|
|
|
Following are the options used in above command with their meanings -
|
|
|
|
-v verbose
|
|
|
|
-m machine name
|
|
|
|
-c branch name, the branch that was pushed on ceph-ci
|
|
|
|
-s test-suite name
|
|
|
|
-p higher the number, lower the priority of the job
|
|
|
|
--filter filter tests in given suite that needs to run, the arg to
|
|
|
|
filter should be the test you want to run
|
|
|
|
|
|
|
|
.. note:: The priority number present in the command above is just a
|
|
|
|
placeholder. It might be highly inappropriate for the jobs you may want to
|
|
|
|
trigger. See `Testing Priority`_ section to pick a priority number.
|
|
|
|
|
|
|
|
.. note:: Don't skip passing a priority number, the default value is 1000
|
|
|
|
which way too high; the job probably might never run.
|
|
|
|
|
|
|
|
#. Wait for the tests to run. ``teuthology-suite`` prints a link to the
|
|
|
|
`Pulpito`_ page created for the tests triggered.
|
|
|
|
|
|
|
|
Other frequently used/useful options are ``-d`` (or ``--distro``),
|
|
|
|
``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``.
|
|
|
|
Run ``teuthology-suite --help`` to read description of these and every other
|
|
|
|
options available.
|
|
|
|
|
2020-06-11 12:46:35 +00:00
|
|
|
Testing QA changes (without re-building binaires)
|
|
|
|
-------------------------------------------------
|
|
|
|
While writing a PR you might need to test your PR repeatedly using teuthology.
|
|
|
|
If you are making non-QA changes, you need to follow the standard process of
|
|
|
|
triggering builds, waiting for it to finish and then triggering tests and
|
|
|
|
wait for the result. But if changes you made are purely changes in qa/,
|
|
|
|
you don't need rebuild the binaries. Instead you can test binaries built for
|
|
|
|
the ceph-ci branch and instruct ``teuthology-suite`` command to use a separate
|
|
|
|
branch for running tests. The separate branch can be passed to the command
|
|
|
|
by using ``--suite-repo`` and ``--suite-branch``. Pass the link to the GitHub
|
|
|
|
fork where your PR branch exists to the first option and pass the PR branch
|
|
|
|
name to the second option.
|
|
|
|
|
|
|
|
For example, if you want to make changes in ``qa/`` after testing ``branch-x``
|
|
|
|
(of which has ceph-ci branch is ``wip-username-branch-x``) by running
|
|
|
|
following command::
|
|
|
|
|
|
|
|
teuthology-suite -v -m smithi -c wip-username-branch-x -s fs -p 50 --filter cephfs-shell
|
|
|
|
|
|
|
|
You can make the modifications locally, update the PR branch and then
|
|
|
|
trigger tests from your PR branch as follows::
|
|
|
|
|
|
|
|
teuthology-suite -v -m smithi -c wip-username-branch-x -s fs -p 50 --filter cephfs-shell --suite-repo https://github.com/username/ceph --suite-branch branch-x
|
|
|
|
|
|
|
|
You can verify if the tests were run using this branch by looking at values
|
|
|
|
for the keys ``suite_branch``, ``suite_repo`` and ``suite_sha1`` in the job
|
|
|
|
config printed at the very beginning of the teuthology job.
|
|
|
|
|
2020-01-06 07:40:50 +00:00
|
|
|
About Suites and Filters
|
|
|
|
------------------------
|
|
|
|
See `Suites Inventory`_ for a list of suites of integration tests present
|
|
|
|
right now. Alternatively, each directory under ``qa/suites`` in Ceph
|
|
|
|
repository is an integration test suite, so looking within that directory
|
|
|
|
to decide an appropriate argument for ``-s`` also works.
|
|
|
|
|
|
|
|
For picking an argument for ``--filter``, look within
|
|
|
|
``qa/suites/<suite-name>/<subsuite-name>/tasks`` to get keywords for filtering
|
|
|
|
tests. Each YAML file in there can trigger a bunch of tests; using the name of
|
|
|
|
the file, without the extension part of the file name, as an argument to the
|
|
|
|
``--filter`` will trigger those tests. For example, the sample command above
|
|
|
|
uses ``cephfs-shell`` since there's a file named ``cephfs-shell.yaml`` in
|
|
|
|
``qa/suites/fs/basic_functional/tasks/``. In case, the file name doesn't hint
|
|
|
|
what bunch of tests it would trigger, look at the contents of the file for
|
|
|
|
``modules`` attribute. For ``cephfs-shell.yaml`` the ``modules`` attribute
|
|
|
|
is ``tasks.cephfs.test_cephfs_shell`` which means it'll trigger all tests in
|
|
|
|
``qa/tasks/cephfs/test_cephfs_shell.py``.
|
|
|
|
|
|
|
|
Killing Tests
|
|
|
|
-------------
|
|
|
|
Sometimes a teuthology job might not complete running for several minutes or
|
|
|
|
even hours after tests that were trigged have completed running and other
|
|
|
|
times wrong set of tests can be triggered is filter wasn't chosen carefully.
|
|
|
|
To save resource it's better to termniate such a job. Following is the command
|
|
|
|
to terminate a job::
|
|
|
|
|
|
|
|
teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
|
|
|
|
|
|
|
|
Let's call the the argument passed to ``-r`` as test ID. It can be found
|
|
|
|
easily in the link to the Pulpito page for the tests you triggered. For
|
|
|
|
example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
|
|
|
|
|
|
|
|
Re-running Tests
|
|
|
|
----------------
|
|
|
|
Pass ``--rerun`` option, with test ID as an argument to it, to
|
|
|
|
``teuthology-suite`` command::
|
|
|
|
|
|
|
|
teuthology-suite -v -m smithi -c wip-rishabh-fs-test_cephfs_shell-fix -p 50 --rerun teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
|
|
|
|
|
|
|
|
The meaning of rest of the options is already covered in `Triggering Tests`
|
|
|
|
section.
|
|
|
|
|
|
|
|
Teuthology Archives
|
|
|
|
-------------------
|
|
|
|
Once the tests have finished running, the log for the job can be obtained by
|
|
|
|
clicking on job ID at the Pulpito page for your tests. It's more convenient to
|
|
|
|
download the log and then view it rather than viewing it in an internet
|
|
|
|
browser since these logs can easily be upto size of 1 GB. What's much more
|
|
|
|
easier is to log in to the teuthology machine again
|
|
|
|
(``teuthology.front.sepia.ceph.com``), and access the following path::
|
|
|
|
|
|
|
|
/ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
|
|
|
|
|
|
|
|
For example, for above test ID path is::
|
|
|
|
|
|
|
|
/ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
|
|
|
|
|
|
|
|
This way the log remotely can be viewed remotely without having to wait too
|
|
|
|
much.
|
|
|
|
|
|
|
|
Naming the ceph-ci branch
|
|
|
|
-------------------------
|
|
|
|
There are no hard conventions (except for the case of stable branch; see
|
|
|
|
next paragraph) for how the branch pushed on ceph-ci is named. But, to make
|
|
|
|
builds and tests easily identitifiable on Shaman and Pulpito respectively,
|
|
|
|
prepend it with your name. For example branch ``feature-x`` can be named
|
|
|
|
``wip-yourname-feature-x`` while pushing on ceph-ci.
|
|
|
|
|
|
|
|
In case you are using one of the stable branches (e.g. nautilis, mimic,
|
|
|
|
etc.), include the name of that stable branch in your ceph-ci branch name.
|
|
|
|
For example, ``feature-x`` PR branch should be named as
|
|
|
|
``wip-feature-x-nautilus``. *This is not just a matter of convention but this,
|
|
|
|
more essentially, builds your branch in the correct environment.*
|
|
|
|
|
|
|
|
Delete the branch from ceph-ci, once it's not required anymore. If you are
|
|
|
|
logged in at GitHub, all your branches on ceph-ci can be easily found here -
|
|
|
|
https://github.com/ceph/ceph-ci/branches.
|
|
|
|
|
|
|
|
.. _ceph-ci: https://github.com/ceph/ceph-ci
|
|
|
|
.. _Pulpito: http://pulpito.front.sepia.ceph.com/
|
2020-01-20 06:03:48 +00:00
|
|
|
.. _Running Your First Test: ../running-tests-locally/#running-your-first-test
|
2020-01-06 07:40:50 +00:00
|
|
|
.. _Shaman: https://shaman.ceph.com/builds/ceph/
|
2020-01-20 06:03:48 +00:00
|
|
|
.. _Suites Inventory: ../tests-integration-tests/#suites-inventory
|
|
|
|
.. _Testing Priority: ../tests-integration-tests/#testing-priority
|