Last updated: 2016-11-21

The FreeBSD build will build most of the tools in Ceph.
Note that the (kernel) RBD dependant items will not work since FreeBSD does not 
have RBD (yet)

I started looking into Ceph, because the HAST solution with CARP and 
ggate did not really do what I was looking for. But I'm aiming for 
running a Ceph storage cluster on storage nodes that are running ZFS.
In the end the cluster would be running bhyve on RBD disk that are stored in 
Ceph.

Progress from last report:
==========================

Most important change:
 -	All test run to completion for the current selection of 
	tools. This is only the case for "My Fork" repository. Some of the 
	commits need to be pulled into the HEAD
 -	As of now Cmake is the only way of building Ceph
    - And testing would be best done thru ctest.
 -	Reworked threading/polling code for the simple socket code.
	Now uses a selfpipe, instead of using an odd shutdown() signaling 
	Linux feature.
 -	Modified the EventKqueue code to work around the "feature" that 
	starting threads destroys the the kqueue handles. 
 -  ceph-disk should now be able to support FileStore on a ZFS disk.
	The main reason that it needs to be ZFS is for xattribute: Size and number.
	
Build Prerequisites
===================

	Compiling and building Ceph is tested on 12-CURRENT, but I guess/expect 
	11-RELEASE will also work. And Clang is at 3.8.0.
	It uses the CLANG toolset that is available, 3.7 is no longer tested, 
	but was working when that was with 11-CURRENT. 
	Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to 
	compile everything

The following setup will get things running for FreeBSD:
	
	This all require root privilidges.

 -	Install bash and link it in /bin
	sudo pkg install bash
	sudo ln -s /usr/local/bin/bash /bin/bash
 -	Need to add one compatability line to 
	/usr/include/errno.h
	#define    ENODATA         87              /* Attribute not found */
	(Otherwise some cython compiles will fail.)
 -  getopt is used by several testscripts but it requires more than what 
	the native getopt(1) delivers. So best is to install getopt from ports
	and remove/replace the getopt in /usr/bin.

Getting the FreeBSD work on Ceph:
=================================

 - 	cd "place to work on this"
	git clone https://github.com/wjwithagen/ceph.git
	cd ceph
	git checkout wip-wjw-freebsd-cmake

Building Ceph
=============
 - 	Go and start building
	./do_freebsd.sh
	
Parts not (yet) included:
=========================

 -	RBD
	Rados Block Devices is implemented in the Linux kernel
	It seems that there used to be a userspace implementation first.
	And perhaps ggated could be used as a template since it does some of 
	the same, other than just between 2 disks. And it has a userspace 
	counterpart.
 -	BlueStore.
	FreeBSD and Linux have different AIO API, and that needs to be made 
	compatible Next to that is there discussion in FreeBSD about 
	aio_cancel not working for all devicetypes
 -	CephFS
	Cython tries to access an internal field in dirent which does not 
	compile

Tests that verify the correct working of the above are also excluded 
from the testset

Tests not (yet) include:
=======================

 -  None, although some test can fail if running tests in parallel and there is
    not enough swap. Then tests will start to fail in strange ways.
 
Task to do:
===========
 -	Build an automated test platform that will build ceph/master on 
	FreeBSD and report the results back to the Ceph developers. This will 
	increase the maintainability of the FreeBSD side of things. 
	Developers are signalled that they are using Linux-isms that will not
	compile/run on FreeBSD Ceph has several projects for this: Jenkins, 
	teuthology, pulpito, ...
	But even just a while { compile } loop and report the build data on a 
	static webpage would do for starters.

 -	Run integration tests to see if the FreeBSD daemons will work with a 
	Linux Ceph platform.

 -	Compile and test the user space RBD (Rados Block Device).

 -	Investigate and see if an in-kernel RBD device could be developed a la 
	'ggate'

 -	Investigate the keystore, which could be kernel embedded on Linux an 
	currently prevents building Cephfs and some other parts.

 -	Scheduler information is not used atm, because the schedulers work 
	rather different. But at a certain point in time, this would need some 
	attention:
	in: ./src/common/Thread.cc

 -	Integrate the FreeBSD /etc/rc.d init scripts in the Ceph stack. Both 
	for testing, but mainly for running Ceph on production machines.