You can check out a working copy (actually, clone the repository) with

git clone git://ceph.newdream.net/ceph.git

or

git clone http://ceph.newdream.net/git/ceph.git

To pull the latest,

git pull

You can browse the git repo at http://ceph.newdream.net/git.

There are a range of binary targets, mostly for ease of development and testing:

• fakesyn -- places all logical elements (MDS, client, etc.) in a single binary, with synchronous message delivery (for easy debugging!). Includes synthetic workload generation.
• fakefuse -- same as fakesyn, but mounts a single client via FUSE.
• newsyn -- starts up all logical elements using MPI. As with fakesyn, it includes synthetic workload generation.
• cosd -- standalone OSD
• cmon -- standalone monitor
• cmds -- standalone MDS
• cfuse -- standalone client, mountable via FUSE

For most development, fakesyn, fakefuse, and newsyn are sufficient.

Few quick steps to get things started. Note that these instructions assume either that you are running on one node, or have a shared directory (e.g. over NFS) mounted on each node.

1. Checkout, change into the ceph/src directory, and build. E.g.,

   git clone git://ceph.newdream.net/ceph.git
   cd ceph/src
   make mpi=no fuse=no
   

   (You can omit the mpi=no or fuse=no if you happen to have those installed.)
2. Create a log/ dir for various runtime stats.

   mkdir log
   

3. Identify the EBOFS block devices. This is accomplished with symlinks (or actual files) in the dev/ directory. Devices can be identified by symlinks named after the hostname (e.g. osd.googoo-1), logical OSD number (e.g. osd4), or simply osd.all (in that order of preference). For example,

   mkdir dev
   ln -s /dev/sda3 dev/osd.all   # all nodes use /dev/sda3
   ln -s /dev/sda4 dev/osd0      # except osd0, which should use /dev/sd4
   

   That is, when an osd starts up, it first looks for dev/osd$n, then dev/osd.all, in that order. These need not be "real" devices--they can be regular files too. To get going with fakesyn, for example, or to test a whole "cluster" running on the same node,

   # create small "disks" for osd0-osd3
   for f in 0 1 2 3; do                                 # default is 4 OSDs
   dd if=/dev/zero of=dev/osd$f bs=1048576 count=1024   # 1 GB each
   done
   

   Note that if your home/working directory is mounted via NFS or similar, you'll want to symlink dev/ to a directory on a local disk.

A quick example, assuming you've set up "fake" EBOFS devices as above:

make fakesyn && ./fakesyn --mkfs --debug_ms 1 --debug_client 3 --syn rw 1 100000
# where those options mean:
#	--mkfs               # start with a fresh file system
#	--debug_ms 1         # show message delivery
#	--debug_client 3     # show limited client stuff
#	--syn rw 1 100000    # write 1MB to a file in 100,000 byte chunks, then read it back

One the synthetic workload finishes, the synthetic client unmounts, and the whole system shuts down. The full set of command line arguments can be found in config.cc.

You can start up a the full cluster of daemons on a single host. Assuming you've created a set of individual files for each OSD's block device (the second option of #3 above), you can create a stop.sh script like

#!/bin/sh
killall cosd cmds cmon

and a start.sh script like

#!/bin/sh
./stop.sh
./mkmonmap 1.2.3.4:12345  # your IP here; any unused port will do
./cmon --mkfs --mon 0 &
./cosd --mkfs --osd 0 &
./cosd --mkfs --osd 1 &
./cosd --mkfs --osd 2 &
./cosd --mkfs --osd 3 &
./cmds &

Note that the IP you specify is for the monitor. This is the only fixed and static ip:port in the system. The rest of the cluster daemons bind to a random port and register themselves with the monitor.

The easiest route is fakefuse:

modprobe fuse  # make sure fuse module is loaded
mkdir mnt      # or whereever you want your mount point
make fakefuse && ./fakefuse --mkfs --debug_ms 1 mnt

You should be able to ls, copy files, or whatever else (in another terminal; fakefuse will stay in the foreground). Control-C will kill fuse and cause an orderly shutdown. Alternatively, fusermount -u mnt will unmount. If fakefuse crashes or hangs, you may need to kill -9 fakefuse and/or fusermount -u mnt to clean up. Overall, FUSE is pretty well-behaved. If you have the cluster daemon's already running (as above), you can mount via the standalone fuse client:

modprobe fuse
mkdir mnt
make cfuse && ./cfuse mnt

Any recent mainline kernel will do here.

$ cd linux
$ patch -p1 < ~/ceph/src/kernel/kconfig.patch
patching file fs/Kconfig
patching file fs/Makefile
$ cp ~/ceph/src/kernel/sample.uml.config .config
$ ln -s ~/ceph/src/kernel fs/ceph
$ ln -s ~/ceph/src/include/ceph_fs.h include/linux
$ make ARCH=um

I am using this x86_64 Debian UML root fs image, but any image will do (see http://user-mode-linux.sf.net) as long as the architecture (e.g. x86_64 vs i386) matches your host. Start up the UML instance with something like

./linux ubda=Debian-3.1-AMD64-root_fs mem=256M eth0=tuntap,,,1.2.3.4  # 1.2.3.4 is the _host_ ip

Note that if UML crashes/oopses/whatever, you can restart quick-and-dirty (up arrow + enter) with

reset ; killall -9 linux ; ./linux ubda=Debian-3.1-AMD64-root_fs mem=256M eth0=tuntap,,,1.2.3.4

You'll need to configure the network in UML with an unused IP. For my debian-based root fs image, this /etc/network/interfaces file does the trick:

iface eth0 inet static
        address 1.2.3.5     # unused ip in your host's netowrk
        netmask 255.0.0.0
        gateway 1.2.3.4     # host ip
auto eth0

Note that you need install uml-utilities (apt-get install uml-utilities on debian distros) and add yourself to the uml-net group on the host (or run the UML instance as root) for the network to start up properly.

Inside UML, you'll want an /etc/fstab line like

none            /host           hostfs  defaults        0       0

You can then load the kernel client module and mount from the UML instance with

insmod /host/path/to/ceph/src/kernel/ceph.ko
mount -t ceph 1.2.3.4:/ -o monport=12345,ip=1.2.3.5 mnt # 1.2.3.4 is host, 1.2.3.5 is uml ip

If you're ready to start things up on multiple nodes (or even just multiple processes on the same node), newsyn is the easiest way to get things launched. It uses MPI to start up all the processes. Assuming you have MPICH2 (or similar) installed,

mpd &           # for a single host
mpiboot -n 10   # for multiple hosts (see MPICH docs)
make newsyn && mpiexec -l -n 10 ./newsyn --mkfs --nummds 1 --numosd 6 --numclient 20 --syn writefile 100 16384

You will probably want to make dev/osd.all a symlink to some block device that exists on every node you're starting an OSD on. Otherwise, you'll need a symlink (for "block device" file) for each osd. If you want to mount a distributed FS (instead of generating a synthetic workload), try

make newsyn && mpiexec -l -n 10 ./newsyn --mkfs --nummds 2 --numosd 6 --numclient 0     # 0 clients, just mds and osds
# in another terminal,
mkdir mnt
make cfuse && ./cfuse mnt
# and in yet another terminal,
ls mnt 
touch mnt/asdf   # etc

Currently, when the last client (cfuse instance, in this case) shuts down, the whole thing will shut down. Assuming things shut down cleanly, you should be able to start things up again without the --mkfs flag and recover the prior file system state.

Here's a crude table diagram that shows how the major (user space) pieces fit together. Ingore the MDS bits; that's mostly wrong. FIXME: this links to the old Subversion repository.

	Application
	kernel
Application	FUSE glue
Client			MDS
Filer	ObjectCacher		MDLog	MDStore
	Objecter
		(message layer)
		OSD
		ObjectStore
		EBOFS	FakeStore
		BlockDevice
		Kernel POSIX interface
Key:	Network	Entity	Lib/module	Abstract interface	Kernel