RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-02-21 18:17:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

doc: update tests-integration-testing-teuthology-workflow.rst

Browse Source 
* add "Infrastructure" section.

* move "Naming the ceph-ci branch" section under
   "Getting binaries - Build Ceph". Also mention
   about centos9-only trick.

* in "Teuthology Archives", mention about developer
   playground machines and ceph log files.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
This commit is contained in:
Vallari Agrawal 2024-08-31 19:57:25 +05:30
parent 02fe44ac60
commit 9bfcb8e17d
No known key found for this signature in database
GPG Key ID: B8139A21B418CCAE
1 changed files with 140 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

184
doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst
View File

@ -3,11 +3,68 @@
Integration Tests using Teuthology Workflow
===========================================
Scheduling Test Run
-------------------
Infrastructure
--------------
Getting binaries
****************
Components:
1. `ceph-ci`_: Clone of the main Ceph repository, used for triggering Jenkins
Ceph builds for development.
2. `Ceph Jenkins`_: Responsible for triggering builds, uploading packages
to Chacra, and pushing updates about the build to Shaman.
3. `Shaman`_: UI Interface used to check build status. In its backend,
it is a REST API to query and store build information.
4. `Chacra`_: Service where packages are uploaded. The binaries uploaded
here can be downloaded and used by anyone.
5. `Teuthology CLI`_: Developers can use various Teuthology commands to schedule
and manage test runs.
6. Teuthology: This component is responsible for pushing test jobs to
the Beanstalk queue and Paddles. It also picks jobs from
the queue and runs tests.
7. Beanstalk queue: A priority queue containing all the queued jobs.
Developers typically do not need to interact with it.
8. Paddles: A backend service that stores all test run information.
Developers typically do not need to interact with it.
9. `Pulpito`_: A UI interface (for information stored in Paddles) that allows
developers to see detailed information about their scheduled tests,
including status and results.
10. Testnodes: A cluster of various machines that are used for running tests.
Developers usually schedule tests to run on `smithi`_ machines, which are
dedicated test nodes for Teuthology integration testing.
Each Teuthology test *run* contains multiple test *jobs*. Each job runs in an
environment isolated from other jobs, on a different collection of test nodes.
To test a change in Ceph, follow these steps:
1. Getting binaries - Build Ceph.
2. Scheduling Test Run:
a. About Test Suites.
b. Triggering Teuthology Tests.
c. Testing QA changes (without re-building binaries).
d. Filtering Tests.
3. Viewing Test Results:
a. Pulpito Dashboard.
b. Teuthology Archives (Reviewing Logs).
4. Killing tests.
5. Re-running tests.
Getting binaries - Build Ceph
-----------------------------
Ceph binaries must be built for your branch before you can use teuthology to run integration tests on them. Follow these steps to build the Ceph binaries:
@ -41,8 +98,44 @@ Ceph binaries must be built for your branch before you can use teuthology to run
.. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph
Triggering Tests
****************
Naming the ceph-ci branch
*************************
Prepend your branch with your name before you push it to ceph-ci. For example,
a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
``$yourname`` is replaced with your name. Identifying your branch with your
name makes your branch easily findable on Shaman and Pulpito.
If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
the name of that stable branch in your ceph-ci branch name.
For example, the ``feature-x`` PR branch should be named
``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
You can choose to only trigger a CentOS 9.Stream build (excluding other distro like ubuntu)
by adding "centos9-only" at the end of the ceph-ci branch name. For example,
``wip-$yourname-feature-centos9-only``. This helps to get quicker builds and save resources
when you don't require binaries for other distros.
Delete the branch from ceph-ci when you no longer need it. If you are
logged in to GitHub, all your branches on ceph-ci can be found here:
https://github.com/ceph/ceph-ci/branches.
Scheduling Test Run
-------------------
About Test Suites
*****************
Integration tests are organized into “suites”, which are defined in ``qa/suites``
sub-directory of the Ceph repository. These test suites can be run with the teuthology-suite
command.
See `Suites Inventory`_ for a list of available suites of integration tests.
More details understanding of how these test suites are defined can be found on `Integration Test Introduction Page`_.
Triggering Teuthology Tests
***************************
After you have built Ceph binaries for your branch, you can run tests using
teuthology. This procedure explains how to run tests using teuthology.
@ -54,7 +147,10 @@ teuthology. This procedure explains how to run tests using teuthology.
ssh <username>@teuthology.front.sepia.ceph.com
This requires Sepia lab access. To request access to the Sepia lab, see:
https://ceph.github.io/sepia/adding_users/
https://ceph.github.io/sepia/adding_users/.
#. For initial setup, follow `teuthology installation guide`_ to setup teuthology for
your user on teuthology machine. This will enable you to run teuthology commands.
#. Run the ``teuthology-suite`` command:
@ -66,7 +162,7 @@ teuthology. This procedure explains how to run tests using teuthology.
-s fs \
-p 110 \
--filter "cephfs-shell" \
-e foo@gmail.com \
-e foo@gmail.com
The options in the above command are defined here:
@ -101,10 +197,13 @@ teuthology. This procedure explains how to run tests using teuthology.
`Pulpito`_ where the test results can be viewed.
The ``--dry-run`` option allows you to demo-run ``teuthology-suite`` command without
actually scheduling teuthology tests. This is helpful to check how many jobs and which jobs
a command will schedule.
Other frequently used/useful options are ``-d`` (or ``--distro``),
``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``,
``-l`` (for limiting number of jobs) , ``-N`` (for how many times the job will
``--distro-version``, ``--filter-out``, ``--timeout``, ``flavor``, ``--rerun``,
``--limit`` (for limiting number of jobs) , ``-N`` (for how many times the job will
run), and ``--subset`` (used to reduce the number of tests that are triggered). Run
``teuthology-suite --help`` to read descriptions of these and other options.
@ -159,15 +258,15 @@ job config printed at the beginning of the teuthology job.
for the builds to finish, then triggering tests and waiting for
the test results.
About Suites and Filters
************************
See `Suites Inventory`_ for a list of available suites of integration tests.
Each directory under ``qa/suites`` in the Ceph repository is an integration
test suite, and arguments appropriate to follow ``-s`` can be found there.
Filtering Tests
***************
Test suites includes combinations of many yaml files which can results in massive
amount of jobs being scheduled for a suite. So filters can help to reduce the amount
of jobs or schedule particular jobs within a suite.
Keywords for filtering tests can be found in
``qa/suites/<suite-name>/<subsuite-name>/tasks`` and can be used as arguments
``qa/suites/<suite-name>/<subsuite-name>/tasks`` in Ceph repository and can be used as arguments
for ``--filter``. Each YAML file in that directory can trigger tests; using the
name of the file without its filename extension as an argument to the
``--filter`` triggers those tests.
@ -182,6 +281,8 @@ contents of the file for the ``modules`` attribute. For ``cephfs-shell.yaml``
the ``modules`` attribute is ``tasks.cephfs.test_cephfs_shell``. This means
that it triggers all tests in ``qa/tasks/cephfs/test_cephfs_shell.py``.
Read more about how to `Filter Tests by their Description`_.
Viewing Test Results
---------------------
@ -195,22 +296,26 @@ Teuthology Archives
*******************
After the tests have finished running, the log for the job can be obtained by
clicking on the job ID at the Pulpito page associated with your tests. It's
clicking on the job ID at the Pulpito run page associated with your tests. It's
more convenient to download the log and then view it rather than viewing it in
an internet browser since these logs can easily be up to 1 GB in size. It is
easier to ssh into the teuthology machine (``teuthology.front.sepia.ceph.com``)
and access the following path::
an internet browser since these logs can easily be up to 1 GB in size.
It is also possible to ssh into a `developer playground machine`_ and access the following path::
/ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
/teuthology/<run-name>/<job-id>/teuthology.log
For example: for the above test ID, the path is::
/ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
/teuthology/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
This method can be used to view the log more quickly than would be possible through a browser.
In addition to ``teuthology.log``, some other files are included for debugging
purposes:
To view ceph logs (cephadm, ceph monitors, ceph-mgr, etc) or system logs,
remove ``teuthology.log`` from the job's teuthology log url on browser and then navigate
to ``remote/<machine>/log/``. System logs can be found at ``remote/<machine>/syslog/``.
Similarly, these logs can be found on developer playground machines at
``/teuthology/<test-id>/<job-id>/remote/<machine>/``.
Some other files that are included for debugging purposes:
* ``unit_test_summary.yaml``: Provides a summary of all unit test failures.
Generated (optionally) when the ``unit_test_scan`` configuration option is
@ -219,7 +324,7 @@ purposes:
* ``valgrind.yaml``: Summarizes any Valgrind errors that may occur.
.. note:: To access archives more conveniently, ``/a/`` has been symbolically
linked to ``/ceph/teuthology-archive/``. For instance, to access the previous
linked to ``/teuthology/``. For instance, to access the previous
example, we can use something like::
/a/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
@ -234,9 +339,9 @@ Here is the command that terminates jobs:
.. prompt:: bash $
teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
teuthology-kill -p -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi -m smithi -o scheduled_teuthology@teuthology
Let's call the argument passed to ``-r`` as test ID. It can be found
The argument passed to ``-r`` is run name. It can be found
easily in the link to the Pulpito page for the tests you triggered. For
example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
@ -275,23 +380,9 @@ Following's the definition of new options introduced in this section:
'waiting'. Default value: 'fail,dead'
======================= ===============================================
Naming the ceph-ci branch
-------------------------
Prepend your branch with your name before you push it to ceph-ci. For example,
a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
``$yourname`` is replaced with your name. Identifying your branch with your
name makes your branch easily findable on Shaman and Pulpito.
If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
the name of that stable branch in your ceph-ci branch name.
For example, the ``feature-x`` PR branch should be named
``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
Delete the branch from ceph-ci when you no longer need it. If you are
logged in to GitHub, all your branches on ceph-ci can be found here:
https://github.com/ceph/ceph-ci/branches.
.. _ceph-ci: https://github.com/ceph/ceph-ci
.. _Ceph Jenkins: https://jenkins.ceph.com/
.. _Teuthology CLI: https://docs.ceph.com/projects/teuthology/en/latest/commands/list.html
.. _Chacra: https://github.com/ceph/chacra/blob/master/README.rst
.. _Pulpito: http://pulpito.front.sepia.ceph.com/
.. _Running Your First Test: ../../running-tests-locally/#running-your-first-test
@ -299,4 +390,9 @@ https://github.com/ceph/ceph-ci/branches.
.. _Suites Inventory: ../tests-integration-testing-teuthology-intro/#suites-inventory
.. _Testing Priority: ../tests-integration-testing-teuthology-intro/#testing-priority
.. _Triggering Tests: ../tests-integration-testing-teuthology-workflow/#triggering-tests
.. _Integration Test Introduction Page: ../tests-integration-testing-teuthology-intro/#how-integration-tests-are-defined
.. _tests-sentry-developers-guide: ../tests-sentry-developers-guide/
.. _smithi: https://wiki.sepia.ceph.com/doku.php?id=hardware:smithi
.. _teuthology installation guide: https://docs.ceph.com/projects/teuthology/en/latest/INSTALL.html#installation-and-setup
.. _Filter Tests by their Description: ../tests-integration-testing-teuthology-intro/#filtering-tests-by-their-description
.. _developer playground machine: https://wiki.sepia.ceph.com/doku.php?id=devplayground