RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-01-29 14:34:40 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,523 Commits 215 Branches 394 Tags 1.3 GiB
9257b8706f
Commit Graph

2189 Commits

Author SHA1 Message Date
Loic Dachary
98e0617651 erasure-code: separate profile from pool creation 
Instead of being hard coded, the erasure code profile is read from yaml
as:

    erasure_code_profile:
      name: teuthologyprofile
      k: 2
      m: 1
      ruleset-failure-domain: osd

Which translates into

    ceph osd erasure-code-profile set teuthologyprofile \
        k=2 m=1 ruleset-failure-domain=osd

The semantic of k/m etc. depends on the erasure code plugin, but it is
common to use k as the dividing factor for each object and m as the
number of coding chunks.

Using a different set of parameters or a different plugin can
be done without modifying the teuthology code:

    erasure_code_profile:
      name: LRCprofile
      plugin: LRC
      mapping: __DD__DD
      layers: [[ "_cDD_cDD", "" ], [ "cDDD____", "" ], [ "____cDDD", ""]]

For backward compatibility, the default erasure code profile is set to

      k: 2
      m: 1
      ruleset-failure-domain: osd

which matches the previously hardcoded default of the corresponding
functions.

Signed-off-by: Loic Dachary <loic@dachary.org>
 2014-06-30 22:43:57 +02:00
Zack Cerza
84635d8b27 Merge pull request #233 from ceph/wip-background 
run something in the background
 2014-06-30 14:38:08 -06:00
Sage Weil
7e388c9f20 background_exec: run something in the background 
This is a contextmanager task that will run some command in the background
for the duration of any subsequent tasks, and kill it in the cleanup
phase.

Signed-off-by: Sage Weil <sage@inktank.com>
 2014-06-30 13:35:53 -07:00
Sage Weil
1cd8076543 daemon-helper: optional kill an entire process group 
Signed-off-by: Sage Weil <sage@inktank.com>
 2014-06-30 13:35:53 -07:00
Zack Cerza
5a61f449d1 Split up repo helper into separate functions 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-30 10:10:31 -06:00
John Spray
641a2943a7 task: update 'newfs' users to 'fs new' 
Signed-off-by: John Spray <john.spray@redhat.com>
 2014-06-30 10:40:18 +01:00
John Spray
8be756a065 task/ceph: Invoke newfs for CephFS as needed 
New ceph versions will no longer create the
CephFS pools and will not enable the filesystem
by default.  This change adds an explicit step
to the cluster initialization to create the
filesystem if it doesn't already exist.

Signed-off-by: John Spray <john.spray@inktank.com>
 2014-06-30 10:40:18 +01:00
Zack Cerza
6b94807595 Use corrected tube_name 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 14:42:18 -06:00
Zack Cerza
8583306769 Return corrected tube_name 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 14:41:58 -06:00
Zack Cerza
491999beef Use teuthology.beanstalk 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 14:34:39 -06:00
Zack Cerza
55b65d62f0 Document checkout_repo() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:21:53 -06:00
Zack Cerza
25a40677cc Drop a "fix" for a bug that probably never existed 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:18:05 -06:00
Zack Cerza
a4c3a02547 Add a note about teuthology scheduling 
Specifically, that we always schedule using the master branch.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:07 -06:00
Zack Cerza
f5bed55d47 Improve missing branch detection and logging 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:07 -06:00
Zack Cerza
abd359086c Add unit tests for repo_utils 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:07 -06:00
Zack Cerza
807d6fd715 Use 'ceph-qa-suite.git' for the repo name 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:07 -06:00
Zack Cerza
b16c48a8f8 Also handle teuthology repo checkouts 
Our shell script did this also.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:07 -06:00
Zack Cerza
88d26e4ae4 Handle ceph-qa-suite checkouts 
Previously we had an internally-maintained shell script that did this.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:06 -06:00
Zack Cerza
6e3e6693d1 Generalize error message 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:06 -06:00
Zack Cerza
484693c2e9 Fix linter errors 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:06 -06:00
Zack Cerza
93fd6b899f Move repo checkout code to new module repo_utils 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:06 -06:00
Zack Cerza
5e3c13de60 Use a more informative variable name for dest_path 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-27 10:14:06 -06:00
Zack Cerza
2e18fddb6f Only delete jobs that match the pattern 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
e7323da58b Swap order of job_id and run_name when deleting 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
f4321b0519 Skip the kernel stanza if the branch passed is '-' 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
ba1eae988e Add more docstrings 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
e1597a036f No need to explicitly return None here 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
667a223f03 Update unit tests to reflect renamed function 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:24 -06:00
Zack Cerza
7082a3dff0 Add a few more docstrings and a debug statement 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
743edd6b70 Fix a couple linter errors 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
173c7b84f6 Fix bug where the base yaml wasn't being merged 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
fd13a18c83 With dry_run, quote any individual args 
So that individual commands can be copied and pasted
in order to execute them individually.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
b6cb3f8cb7 Add lots of unit tests for teuthology.suite 
More to come...

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
91c7385988 Raise ScheduleFailError even if not sending email 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
356ab7e06d Do everything that schedule_suite.sh does 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
15317083f9 Add default results_email 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
89b38dd347 Don't schedule an email job if email isn't set 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
26fdef0a75 Split prepare_and_schedule() out of main() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
1be15edbcb Use the return value of schedule_suite() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:23 -06:00
Zack Cerza
19d73071d1 Don't reread the entire yaml file... 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
d92f538549 Remove unused functions 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
0aabafd65f 'and' is a thing 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
675f087b28 Take machine_type instead of worker 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
d14c44f4e2 Add functions for querying gitbuilder repos 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
4fed92c039 Silence connection pool logging 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
43505b2ace Only allow scheduling one suite per call. 
Also remove all traces of the extra and confusing term 'collection'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
f5af797ce3 Port to docopt 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
1ccbff8dbf Refactor collection scheduling out of main() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-25 12:54:22 -06:00
Zack Cerza
3134a32018 If archive_base is None, use config's 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-24 11:07:02 -06:00
Zack Cerza
5cd50a42be Allow killing jobs by passing a 'jobspec'. 
See teuthology-kill --help

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-20 11:39:47 -04:00
Zack Cerza
fc772d7354 Add a couple tests for teuthology.schedule 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-19 14:11:56 -04:00
Zack Cerza
f69f2dffa9 Automatically watch 'multi' tube if tube_name 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-10 14:40:37 -05:00
Zack Cerza
78dbc36931 Refactor teuthology.schedule... 
... to separate the assembling of the job config from the scheduling of
the job. Also port its argument parsing to docopt.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-09 18:31:01 -05:00
Zack Cerza
78610b03e3 Remote 'delete' and 'show' features 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-09 18:31:01 -05:00
Zack Cerza
b2f0ddad50 Remove unused import 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-09 18:30:37 -05:00
Zack Cerza
a405c228fc Merge pull request #268 from ceph/wip-5400 
Support installing ceph kernels from gitbuilder on rpm machines.
 2014-06-09 16:41:10 -05:00
Sandon Van Ness
3d1df3aa63 Better variable names. 
As suggested by review.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
 2014-06-09 13:43:45 -07:00
Zack Cerza
dbf984e9d8 Disable "fix" for corrupt repo detection 
It's causing more problems than it's solving at this point

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-09 11:46:19 -05:00
Zack Cerza
2e5c697f1d Use correct cwd 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-09 09:22:21 -05:00
Zack Cerza
e217488f9a Merge pull request #269 from ceph/wip-radosfix 
rados: Add hit_set_list missing from possible op weights
 2014-06-07 09:48:44 -05:00
Zack Cerza
b7c1e35114 Update to reflect relocated get_jobs() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-07 09:45:50 -05:00
David Zafman
875e2d240d rados: Add hit_set_list missing from possible op weights 
For the future help keep in sync with test/osd/TestRados.cc by listing
in order of op_types in that code.

Signed-off-by: David Zafman <david.zafman@inktank.com>
 2014-06-06 15:30:05 -07:00
Zack Cerza
2a7de82498 Move teuthology-ls's implementation to ls.py 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-06 16:15:15 -05:00
Zack Cerza
87e7ff0e2d Fix corrupt repo detection 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-06 12:01:33 -05:00
Sandon Van Ness
825db5057f Added requested comment. 
Signed-off-by: Sandon Van Ness <sandon@inktank.com>
 2014-06-05 12:07:47 -07:00
Sandon Van Ness
242d9de4e2 Remove unused variables and functions. 
Signed-off-by: Sandon Van Ness <sandon@inktank.com>
 2014-06-05 12:04:49 -07:00
Zack Cerza
cadcbf3914 Remove unused import 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-04 16:29:11 -05:00
Sandon Van Ness
ed3ec397ba Support installing ceph kernels from gitbuilder on rpm machines. 
Signed-off-by: Sandon Van Ness <sandon@inktank.com>
 2014-06-03 18:36:40 -07:00
Josh Durgin
3ae54ead8a Merge pull request #267 from ceph/wip-runxfstests 
wip-runxfstests

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
 2014-06-03 15:56:44 -07:00
Zack Cerza
0f04af0799 Add a timeout for unmounting 
Ten minutes seems like it might be longer than necessary, but I doubt
it's too short.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-03 12:39:38 -05:00
Alfredo Deza
60e6788ddf Merge pull request #265 from ceph/wip-orchestra 
Refactor teuthology.orchestra.run
 2014-06-03 09:12:21 -04:00
Zack Cerza
af73e3080c Remove some duplicate code. 
Well, not exactly duplicate, but functionally compatible. No sense in
maintaining two "walk the filesystem to assemble job information"
utilities.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-02 19:01:09 -05:00
Zack Cerza
ae0d26ae71 Try a different approach to zombie prevention 
The previous method I had committed had the unfortunate side-effect of
preventing teuthology job processes from giving their returncode to the
worker process. This method shouldn't do that.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-06-02 09:23:27 -05:00
Yehuda Sadeh
3968f6f744 radosgw-admin: adjust bucket link interface 
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
 2014-06-02 09:23:27 -05:00
Ilya Dryomov
86c60f7de0 rbd: change test_script to run_xfstests_krbd.sh 
Invoke run_xfstests.sh through a wrapper, which supplies an exclusion
list, in order to skip tests we don't want to run.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
 2014-05-31 15:16:52 +04:00
Ilya Dryomov
2551c528a2 rbd: recognize 'randomize' parameter 
randomize (default: false) randomizes xfstests test order.

Signed-off-by: Ilya Dryomov <ilya.dryomov@inktank.com>
 2014-05-31 15:11:38 +04:00
Zack Cerza
205c455fc1 Give daemons a five-minute timeout for stopping 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 16:39:16 -05:00
Zack Cerza
7035c17ec9 Add RemoteProcess.__repr__() 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 16:36:03 -05:00
Zack Cerza
eac2c2abbb Update users of the teuthology.orchestra.run APIs 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 16:05:23 -05:00
Zack Cerza
b386f5e5df Refactor teuthology.orchestra.run 
RemoteProcess behaves more like subprocess.Popen, with some important
differences.

A summary of the API changes:
* RemoteProcess.exitstatus is either an int or None; it is never a callable
  nor a gevent.AsyncResult.
* New method: RemoteProcess.execute()
* New method: RemoteProcess.poll()
* New method: RemoteProcess.wait()
* New attribute: RemoteProcess.returncode - alias to exitstatus
* New property: RemoteProcess.finished - added because returncode can be None
  if the connection was interrupted
* run.execute() is removed.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 16:05:23 -05:00
Zack Cerza
59ee17dc19 Avoid piles of zombies 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 09:59:36 -05:00
Zack Cerza
0fcfaee726 Add comment explaining 'preexec_fn=os.setpgrp' 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-30 09:56:27 -05:00
Zack Cerza
b26ba60565 Check for broken repos before updating 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-29 13:23:23 -05:00
Zack Cerza
abc722c8be More verbose logging for daemon restarts 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-29 00:37:27 -05:00
Zack Cerza
2363a17caa Bump log level to info 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-28 23:40:48 -05:00
Zack Cerza
c4783543e3 Don't block on teuthology-results processes 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-28 14:56:59 -05:00
Zack Cerza
a4a9a1f22f Correct logging when running teuthology-results 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-28 11:43:20 -05:00
Zack Cerza
baeeddbe76 Merge pull request #263 from qakapil/patch-two 
updated the get_system_type function
 2014-05-28 09:34:47 -05:00
ksharma
0b97eba364 updated the get_system_type function 
updated the get_system_type function
to add the rpm condition for SUSE.

Signed-off-by: ksharma@suse.com
 2014-05-28 16:31:56 +02:00
John Spray
6f8015135c tasks/mds_journal_migration: create. 
This is for testing the changes to CephFS
journal format, especially the migration
of journals from old to new format.

This is a teuthology task rather than being
a workunit script, because it involves
manipulating services and config during the
procedure.

Signed-off-by: John Spray <john.spray@inktank.com>
 2014-05-28 14:39:41 +01:00
John Spray
554f8c81e0 task/ceph: separate ceph.conf into function 
This is so that other tasks can modify ceph.conf
as needed.

Signed-off-by: John Spray <john.spray@inktank.com>
 2014-05-28 14:39:41 +01:00
John Spray
79dfe2cbef tasks: Use '_' instead of '-' in names 
Python modules are not allowed to contain hyphens
in the name.  Using hyphens only works here because
we're using low level __import__.

Change run_tasks to replace '-' with '_' when reading
configs, and rename the task modules to have valid
python module names.

Aside from general hygiene, the motivation to do this
is to allow one task to build on code in another
task by importing it.

Signed-off-by: John Spray <john.spray@inktank.com>
 2014-05-28 14:39:41 +01:00
Zack Cerza
d3c230071a Don't crash when an invalid branch is passed 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-22 12:26:50 -05:00
Zack Cerza
a6aebe2bbc Set results_server by default 
So that queued jobs get pushed to paddles

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-22 08:49:26 -05:00
Zack Cerza
3b382b7011 Process queued jobs synchronously 
Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-21 16:18:00 -05:00
Zack Cerza
08ace5465e Add exception hook to teuthology-worker 
Workers processes are dying occasionally, and this should cause the
exceptions to be logged.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
 2014-05-21 11:34:11 -05:00
Zack Cerza
d58d2c152e Merge pull request #258 from ceph/wip-fix-format-samba 
Fixed formatting and added a better variable for time out
 2014-05-19 18:33:22 -05:00
Yuri Weinstein
c8881bb7fa Fixed broken quotations 
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
 2014-05-19 16:18:48 -07:00
Yuri Weinstein
6e893ad219 Fixed formatting and added a better variable for time out 
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
 2014-05-19 16:03:30 -07:00
Samuel Just
216e5c9a80 task/: add populate_rbd_pool for quickly spinning up a pool with rbd stuff 
Signed-off-by: Samuel Just <sam.just@inktank.com>
 2014-05-19 15:04:54 -07:00