You can check out a working copy with

svn co https://ceph.svn.sourceforge.net/svnroot/ceph

The rest works essentially the same as CVS, but with svn instead of cvs.

There are a range of binary targets, mostly for ease of development and testing:

• fakesyn -- places all logical elements (MDS, client, etc.) in a single binary, with synchronous message delivery (for easy debugging!). Includes synthetic workload generation.
• fakefuse -- same as fakesyn, but mounts a single client via FUSE.
• newsyn -- starts up all logical elements using MPI. As with fakesyn, it includes synthetic workload generation.
• cosd -- standalone OSD
• cmon -- standalone monitor
• cmds -- standalone MDS
• cfuse -- standalone client, mountable via FUSE

For most development, fakesyn, fakefuse, and newsyn are sufficient.

Few quick steps to get things started:

1. Checkout, change into the ceph/ directory, and build. E.g.,

   svn co https://ceph.svn.sourceforge.net/svnroot/ceph
   cd ~/ceph/trunk/ceph
   make
   

2. Create a log/ dir for various runtime stats.

   mkdir log
   

3. Identify the EBOFS block devices. This is accomplished with symlinks (or actual files) in the dev/ directory. Devices can be identified by symlinks named after the hostname (e.g. osd.googoo-1), logical OSD number (e.g. osd4), or simply osd.all (in that order of preference). For example,

   mkdir dev
   ln -s /dev/sda3 dev/osd.all   # all nodes use /dev/sda3
   ln -s /dev/sda4 dev/osd0      # except osd0, which should use /dev/sd4
   

   These need not be "real" devices--they can be regular files too. To get going with fakesyn, for example,

   # create small "disks" for osd0-osd3
   for f in 0 1 2 3; do                                 # fakesyn defaults is 4 OSDs
   dd if=/dev/zero of=dev/osd$f bs=1048576 count=1024   # 1 GB each
   done
   
   # smaller devices for monitors too
   for f in 0 1 2 3 4 ; do
   dd if=/dev/zero of=dev/mon$f bs=1048576 count=10   # 10 MB each
   done
   

   Note that if your home/working directory is mounted via NFS, you'll want to symlink dev/ to a directory on a local disk.

A quick example, assuming you've set up "fake" EBOFS devices as above:

make fakesyn && ./fakesyn --mkfs --osd_pg_bits 4 --debug_ms 1 --debug_client 3 --syn rw 1 100000
# where those options mean:
#	--mkfs               # start with a fresh file system
#	--osd_pg_bits 4      # we only need a few PGs (we don't care about load balancing)
#	--debug_ms 1         # show message delivery
#	--debug_client 3     # show limited client stuff
#	--syn rw 1 100000    # write 1GB to a file in 100,000 byte chunks, then read it back

The full set of command line arguments can be found in config.cc.

The easiest route is fakefuse:

modprobe fuse  # make sure fuse module is loaded
mkdir mnt      # or whereever you want your mount point
make fakefuse && ./fakefuse --mkfs --debug_ms 1 mnt

You should be able to ls, copy files, or whatever else (in another terminal; fakefuse will stay in the foreground). Control-C will kill fuse and cause an orderly shutdown. Alternatively, fusermount -u mnt will unmount. If fakefuse crashes or hangs, you may need to kill -9 fakefuse and/or fusermount -u mnt to clean up. Overall, FUSE is pretty well-behaved.

If you're ready to start things up on multiple nodes (or even just multiple processes on the same node), newsyn is the easiest way to get things launched. It uses MPI to start up all the processes. Assuming you have MPICH2 (or similar) installed,

mpd &           # for a single host
mpiboot -n 10   # for multiple hosts (see MPICH docs)
make newsyn && mpiexec -l -n 10 ./newsyn --mkfs --nummds 1 --numosd 6 --numclient 20 --syn writefile 100 16384

You will probably want to make dev/osd.all a symlink to some block device that exists on every node you're starting an OSD on. Otherwise, you'll need a symlink (for "block device" file) for each osd. If you want to mount a distributed FS (instead of generating a synthetic workload), try

make newsyn && mpiexec -l -n 10 ./newsyn --mkfs --nummds 2 --numosd 6 --numclient 0     # 0 clients, just mds and osds
# in another terminal,
mkdir mnt
make cfuse && ./cfuse mnt
# and in yet another terminal,
ls mnt 
touch mnt/asdf   # etc

Currently, when the last client (cfuse instance, in this case) shuts down, the whole thing will shut down. Assuming things shut down cleanly, you should be able to start things up again without the --mkfs flag and recover the prior file system state.

Here's a somewhat crude table diagram that shows how the major pieces fit together. Ingore the MDS bits; that's mostly wrong.

	Application
	kernel
Application	FUSE glue
Client			MDS
Filer	ObjectCacher		MDLog	MDStore
	Objecter
		(message layer)
		OSD
		ObjectStore
		EBOFS	FakeStore
		BlockDevice
		Kernel POSIX interface
Key:	Network	Entity	Lib/module	Abstract interface	Kernel