First draft of documentation.

README.rst

@@ -0,0 +1,145 @@
+==================================================
+ `Teuthology` -- The Ceph integration test runner
+==================================================
+
+The Ceph project needs automated tests. Because Ceph is a highly
+distributed system, and has active kernel development, its testing
+requirements are quite different from e.g. typical LAMP web
+applications. Nothing out there seemed to handle our requirements,
+so we wrote our own framework, called `Teuthology`.
+
+
+Overview
+========
+
+Teuthology runs a given set of Python functions (`tasks`), with an SSH
+connection to every host participating in the test. The SSH connection
+uses `Paramiko <http://www.lag.net/paramiko/>`__, a native Python
+client for the SSH2 protocol, and this allows us to e.g. run multiple
+commands inside a single SSH connection, to speed up test
+execution. Tests can use `gevent <http://www.gevent.org/>`__ to
+perform actions concurrently or in the background.
+
+
+Build
+=====
+
+Teuthology uses several Python packages that are not in the standard
+library. To make the dependencies easier to get right, we use a
+`virtualenv` to manage them. To get started, ensure you have the
+``virtualenv`` and ``pip`` programs installed; e.g. on Debian/Ubuntu::
+
+	sudo apt-get install python-virtualenv python-pip
+
+and then run::
+
+	./bootstrap
+
+You can run Teuthology's (and Orchestra's) internal unit tests with::
+
+	./virtualenv/bin/nosetests orchestra teuthology
+
+
+Test configuration
+==================
+
+An integration test run takes three items of configuration:
+
+- ``targets``: what hosts to run on; this is a list of entries like
+  "username@hostname.example.com"
+- ``roles``: how to use the hosts; this is a list of lists, where each
+  entry lists all the roles to be run on a single host; for example, a
+  single entry might say ``[mon.1, osd.1]``
+- ``tasks``: how to set up the cluster and what tests to run on it;
+  see below for examples
+
+The format for this configuration is `YAML <http://yaml.org/>`__, a
+structured data format that is still human-readable and editable.
+
+For example, a full config for a test run that sets up a three-machine
+cluster, mounts Ceph via ``cfuse``, and leaves you at an interactive
+Python prompt for manual exploration (and enabling you to SSH in to
+the nodes & use the live cluster ad hoc), might look like this::
+
+	roles:
+	- [mon.0, mds.0, osd.0]
+	- [mon.1, osd.1]
+	- [mon.2, client.0]
+	targets:
+	- ubuntu@host07.example.com
+	- ubuntu@host08.example.com
+	- ubuntu@host09.example.com
+	tasks:
+	- ceph:
+	- cfuse: [client.0]
+	- interactive:
+
+The number of entries under ``roles`` and ``targets`` must match.
+
+Note the colon after every task name in the ``tasks`` section.
+
+You need to be able to SSH in to the listed targets without
+passphrases, and the remote user needs to have passphraseless `sudo`
+access.
+
+If you'd save the above file as ``example.yaml``, you could run
+teuthology on it by saying::
+
+	./virtualenv/bin/teuthology example.yaml
+
+You can also pass the ``-v`` option, for more verbose execution. See
+``teuthology --help`` for more.
+
+
+Multiple config files
+---------------------
+
+You can pass multiple files as arguments to ``teuthology``. Each one
+will be read as a config file, and their contents will be merged. This
+allows you to e.g. share definitions of what a "simple 3 node cluster"
+is. The source tree comes with ``roles/3-simple.yaml``, so we could
+skip the ``roles`` section in the above ``example.yaml`` and then
+run::
+
+	./virtualenv/bin/teuthology roles/3-simple.yaml example.yaml
+
+
+Reserving target machines
+-------------------------
+
+Right now there is no automatic machine allocation and locking
+support.
+
+For the `sepia` cluster, use the Autotest web UI, lock the hosts you
+intend to use, and write a ``targets.yaml`` file yourself.
+
+Later, a utility will be written to create a similar yaml with as many
+hosts as you request, while taking care of the locking. (TODO)
+
+
+Tasks
+=====
+
+A task is a Python module in the ``teuthology.task`` package, with a
+callable named ``task``. It gets the following arguments:
+
+- ``ctx``: a context that is available through the lifetime of the
+  test run, and has useful attributes such as ``cluster``, letting the
+  task access the remote hosts. Tasks can also store their internal
+  state here. (TODO beware namespace collisions.)
+- ``config``: the data structure after the colon in the config file,
+  e.g. for the above ``cfuse`` example, it would be a list like
+  ``["client.0"]``.
+
+Tasks can be simple functions, called once in the order they are
+listed in ``tasks``. But sometimes, it makes sense for a task to be
+able to clean up after itself; for example, unmounting the filesystem
+after a test run. A task callable that returns a Python `context
+manager
+<http://docs.python.org/library/stdtypes.html#typecontextmanager>`__
+will have the manager added to a stack, and the stack will be unwound
+at the end of the run. This means the cleanup actions are run in
+reverse order, both on success and failure. A nice way of writing
+context managers is the ``contextlib.contextmanager`` decorator; look
+for that string in the existing tasks to see examples, and note where
+they use ``yield``.

TODO.org

@@ -0,0 +1,46 @@
+#+FILETAGS: :newdream:teuthology:todo:
+
+* TODO cleanup properly after a run
+for now, you can run something like this:
+dsh -m ubuntu@sepia70.ceph.dreamhost.com,ubuntu@sepia71.ceph.dreamhost.com,ubuntu@sepia72.ceph.dreamhost.com 'killall --quiet /tmp/cephtest/binary/usr/local/bin/cmon /tmp/cephtest/binary/usr/local/bin/cosd /tmp/cephtest/binary/usr/local/bin/cmds ; fusermount -u /tmp/cephtest/mnt.*; rm -rf /tmp/cephtest' 
+* TODO osd etc killing while test is running
+* TODO automatic host locking
+* TODO add task for kclient mount/umount task
+- kclient: [client.0]
+* TODO add task for rbd activation on cluster
+- rbd.activate:
+- rbd.activate: [client.0]
+* TODO add task for rbd image creation
+- rbd.create_image: [client.0]
+- rbd.create_image:
+    client.0:
+      image_size: 8192
+* TODO add task for rbd modprobe and dev create on client
+- rbd.modprobe: [client.0]
+- rbd.dev_create: [client.0]
+* TODO add task for mounting fs on top of rbd
+- rbd.mkfs: [client.0]
+- rbd.mount: [client.0]
+* TODO add full service rbd task that wraps the lower-level tasks
+- rbd:
+- rbd: [client.0]
+- rbd:
+    client.0:
+      image_size: 8192
+* TODO make cfuse task default to all clients
+- cfuse: [client.0]
+- cfuse:
+* TODO make kclient task default to all clients
+- kclient: [client.0]
+- kclient:
+* TODO more than one autotest task call
+- autotest:
+    client.0: [dbench, bonnie]
+- autotest:
+    client.0: [dbench, bonnie]
+* TODO MAYBE more defaults for autotest task, so you don't need to list all clients?
+- autotest:
+    default: dbench
+    client.42: [dbench, bonnie]
+- autotest: dbench
+- autotest: [dbench, bonnie]

dbench.yaml

@@ -1,14 +0,0 @@
-roles:
-- [mon.0, mds.0, osd.0]
-- [mon.1, osd.1]
-- [mon.2, client.0]
-targets:
-- ubuntu@sepia70.ceph.dreamhost.com
-- ubuntu@sepia71.ceph.dreamhost.com
-- ubuntu@sepia72.ceph.dreamhost.com
-tasks:
-- ceph:
-- cfuse: [client.0]
-- autotest:
-    client.0: [dbench]
-- interactive:

interactive.yaml

@@ -1,12 +0,0 @@
-roles:
-- [mon.0, mds.0, osd.0]
-- [mon.1, osd.1]
-- [mon.2, client.0]
-targets:
-- ubuntu@sepia70.ceph.dreamhost.com
-- ubuntu@sepia71.ceph.dreamhost.com
-- ubuntu@sepia72.ceph.dreamhost.com
-tasks:
-- ceph:
-- cfuse: [client.0]
-- interactive:

roles/3-simple.yaml

@@ -0,0 +1,4 @@
+roles:
+- [mon.0, mds.0, osd.0]
+- [mon.1, osd.1]
+- [mon.2, client.0]

teuthology/task/autotest.py

@@ -8,6 +8,23 @@ from orchestra import run
 log = logging.getLogger(__name__)
 
 def task(ctx, config):
+    """
+    Run an autotest test on the ceph cluster.
+
+    Only autotest client tests are supported.
+
+    The config is a mapping from role name to list of tests to run on
+    that client.
+
+    For example::
+
+        tasks:
+        - ceph:
+        - cfuse: [client.0, client.1]
+        - autotest:
+            client.0: [dbench]
+            client.1: [bonnie]
+    """
     assert isinstance(config, dict)
 
     log.info('Setting up autotest...')

teuthology/task/ceph.py

@@ -12,6 +12,15 @@ log = logging.getLogger(__name__)
 
 @contextlib.contextmanager
 def task(ctx, config):
+    """
+    Set up and tear down a Ceph cluster.
+
+    For example::
+
+        tasks:
+        - ceph:
+        - interactive:
+    """
     assert config is None
 
     log.info('Checking for old test directory...')

teuthology/task/cfuse.py

@@ -9,6 +9,18 @@ log = logging.getLogger(__name__)
 
 @contextlib.contextmanager
 def task(ctx, config):
+    """
+    Mount/unmount a ``cfuse`` client.
+
+    The config is expected to be a list of clients to do this
+    operation on. This lets you e.g. set up one client with ``cfuse``
+    and another with ``kclient``.
+
+        tasks:
+        - ceph:
+        - cfuse: [client.0]
+        - interactive:
+    """
     log.info('Mounting cfuse clients...')
     assert isinstance(config, list), \
         "task fuse automatic configuration not supported yet, list all clients"

teuthology/task/interactive.py

@@ -6,6 +6,23 @@ rlcompleter.__name__ # silence pyflakes
 readline.parse_and_bind('tab: complete')
 
 def task(ctx, config):
+    """
+    Run an interactive Python shell, with the cluster accessible via
+    the ``ctx`` variable.
+
+    Hit ``control-D`` to continue.
+
+    This is also useful to pause the execution of the test between two
+    tasks, either to perform ad hoc operations, or to examine the
+    state of the cluster. You can also use it to easily bring up a
+    Ceph cluster for ad hoc testing.
+
+    For example::
+
+        tasks:
+        - ceph:
+        - interactive:
+    """
     code.interact(
         banner='Ceph test interactive mode, use ctx to interact with the cluster, press control-D to exit...',
         # TODO simplify this

teuthology/task/nop.py

@@ -1,5 +1,10 @@
 def task(ctx, config):
     """
     This task does nothing.
+
+    For example::
+
+        tasks:
+        - nop:
     """
     pass