You can check out a working copy (actually, clone the repository) with

git clone git://ceph.newdream.net/ceph.git

or

git clone http://ceph.newdream.net/git/ceph.git

To pull the latest,

git pull

You can browse the git repo at http://ceph.newdream.net/git.

There are a range of binary targets, mostly for ease of development and testing:

• cmon -- monitor
• cosd -- OSD storage daemon
• cmds -- MDS metadata server
• cfuse -- client, mountable via FUSE
• csyn -- client sythetic workload generator
• cmonctl -- control tool
• fakesyn -- places all logical elements (MDS, client, etc.) in a single binary, with synchronous message delivery (for easy debugging!). Includes synthetic workload generation.
• fakefuse -- same as fakesyn, but mounts a single client via FUSE.

Few quick steps to get things started. Note that these instructions assume either that you are running on one node, or have a shared directory (e.g. over NFS) mounted on each node.

1. Checkout, change into the ceph/src directory, and build. E.g.,

   git clone git://ceph.newdream.net/ceph.git
   cd ceph
   ./autogen.sh
   ./configure   # of CXXFLAGS="-g" ./configure to disable optimizations (for debugging)
   cd src
   make
   

2. Create a log/ dir for various runtime stats.

   mkdir log
   

3. Identify the EBOFS block devices. This is accomplished with symlinks (or actual files) in the dev/ directory. Devices can be identified by symlinks named after the hostname (e.g. osd.googoo-1), logical OSD number (e.g. osd4), or simply osd.all (in that order of preference). For example,

   mkdir dev
   ln -s /dev/sda3 dev/osd.all   # all nodes use /dev/sda3
   ln -s /dev/sda4 dev/osd0      # except osd0, which should use /dev/sd4
   

   That is, when an osd starts up, it first looks for dev/osd$n, then dev/osd.all, in that order. These need not be "real" devices--they can be regular files too. To get going with fakesyn, for example, or to test a whole "cluster" running on the same node,

   # create small "disks" for osd0-osd3
   for f in 0 1 2 3; do                                 # default is 4 OSDs
   dd if=/dev/zero of=dev/osd$f bs=1048576 count=1024   # 1 GB each
   done
   

   Note that if your home/working directory is mounted via NFS or similar, you'll want to symlink dev/ to a directory on a local disk.

You can start up a the full cluster of daemons on a single host. Assuming you've created a set of individual files for each OSD's block device (the second option of #3 above), there is a start.sh and stop.sh script that will start up on port 12345.

One caveat here is that the ceph daemons need to know what IP they are reachable at; they determine that by doing a lookup on the machine's hostname. Since many/most systems map the hostname to 127.0.0.1 in /etc/hosts, you either need to change that (the easiest approach, usually) or add a --bind 1.2.3.4 argument to cmon/cosd/cmds to help them out.

Note that the monitor has the only fixed and static ip:port in the system. The rest of the cluster daemons bind to a random port and register themselves with the monitor.

The easiest route is fakefuse:

modprobe fuse  # make sure fuse module is loaded
mkdir mnt      # or whereever you want your mount point
make fakefuse && ./fakefuse --mkfs --debug_ms 1 mnt

You should be able to ls, copy files, or whatever else (in another terminal; fakefuse will stay in the foreground). Control-C will kill fuse and cause an orderly shutdown. Alternatively, fusermount -u mnt will unmount. If fakefuse crashes or hangs, you may need to kill -9 fakefuse and/or fusermount -u mnt to clean up. Overall, FUSE is pretty well-behaved. If you have the cluster daemon's already running (as above), you can mount via the standalone fuse client:

modprobe fuse
mkdir mnt
make cfuse && ./cfuse mnt

Any recent mainline kernel will do here.

$ cd linux
$ patch -p1 < ~/ceph/src/kernel/kconfig.patch
patching file fs/Kconfig
patching file fs/Makefile
$ cp ~/ceph/src/kernel/sample.uml.config .config
$ ln -s ~/ceph/src/kernel fs/ceph
$ ln -s ~/ceph/src/include/ceph_fs.h include/linux
$ make ARCH=um

I am using this x86_64 Debian UML root fs image, but any image will do (see http://user-mode-linux.sf.net) as long as the architecture (e.g. x86_64 vs i386) matches your host. Start up the UML guest instance with something like

./linux ubda=Debian-3.1-AMD64-root_fs mem=256M eth0=tuntap,,,1.2.3.4  # 1.2.3.4 is the _host_ ip

Note that if UML crashes/oopses/whatever, you can restart quick-and-dirty (up arrow + enter) with

reset ; killall -9 linux ; ./linux ubda=Debian-3.1-AMD64-root_fs mem=256M eth0=tuntap,,,1.2.3.4

You'll need to configure the network in UML with an unused IP. For my debian-based root fs image, this /etc/network/interfaces file does the trick:

iface eth0 inet static
        address 1.2.3.5     # unused ip in your host's netowrk for the uml guest
        netmask 255.0.0.0
        gateway 1.2.3.4     # host ip
auto eth0

Note that you need install uml-utilities (apt-get install uml-utilities on debian distros) and add yourself to the uml-net group on the host (or run the UML instance as root) for the network to start up properly.

Inside UML, you'll want an /etc/fstab line like

none            /host           hostfs  defaults        0       0

You can then load the kernel client module and mount from the UML instance with

insmod /host/path/to/ceph/src/kernel/ceph.ko
mount -t ceph 1.2.3.4:/ mnt  # 1.2.3.4 is host

A quick example, assuming you've set up "fake" EBOFS devices as above:

make fakesyn && ./fakesyn --mkfs --debug_ms 1 --debug_client 3 --syn rw 1 100000
# where those options mean:
#	--mkfs               # start with a fresh file system
#	--debug_ms 1         # show message delivery
#	--debug_client 3     # show limited client stuff
#	--syn rw 1 100000    # write 1MB to a file in 100,000 byte chunks, then read it back

One the synthetic workload finishes, the synthetic client unmounts, and the whole system shuts down. The full set of command line arguments can be found in config.cc.