mirror of https://github.com/ceph/ceph
124 lines
4.5 KiB
Plaintext
124 lines
4.5 KiB
Plaintext
|
|
Last updated: 2016-11-21
|
|
|
|
The FreeBSD build will build most of the tools in Ceph.
|
|
Note that the (kernel) RBD dependant items will not work since FreeBSD does not
|
|
have RBD (yet)
|
|
|
|
I started looking into Ceph, because the HAST solution with CARP and
|
|
ggate did not really do what I was looking for. But I'm aiming for
|
|
running a Ceph storage cluster on storage nodes that are running ZFS.
|
|
In the end the cluster would be running bhyve on RBD disk that are stored in
|
|
Ceph.
|
|
|
|
Progress from last report:
|
|
==========================
|
|
|
|
Most important change:
|
|
- All test run to completion for the current selection of
|
|
tools. This is only the case for "My Fork" repository. Some of the
|
|
commits need to be pulled into the HEAD
|
|
- As of now Cmake is the only way of building Ceph
|
|
- And testing would be best done thru ctest.
|
|
- Reworked threading/polling code for the simple socket code.
|
|
Now uses a selfpipe, instead of using an odd shutdown() signaling
|
|
Linux feature.
|
|
- Modified the EventKqueue code to work around the "feature" that
|
|
starting threads destroys the the kqueue handles.
|
|
- ceph-disk should now be able to support FileStore on a ZFS disk.
|
|
The main reason that it needs to be ZFS is for xattribute: Size and number.
|
|
|
|
Build Prerequisites
|
|
===================
|
|
|
|
Compiling and building Ceph is tested on 12-CURRENT, but I guess/expect
|
|
11-RELEASE will also work. And Clang is at 3.8.0.
|
|
It uses the CLANG toolset that is available, 3.7 is no longer tested,
|
|
but was working when that was with 11-CURRENT.
|
|
Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to
|
|
compile everything
|
|
|
|
The following setup will get things running for FreeBSD:
|
|
|
|
This all require root privilidges.
|
|
|
|
- Install bash and link it in /bin
|
|
sudo pkg install bash
|
|
sudo ln -s /usr/local/bin/bash /bin/bash
|
|
- Need to add one compatability line to
|
|
/usr/include/errno.h
|
|
#define ENODATA 87 /* Attribute not found */
|
|
(Otherwise some cython compiles will fail.)
|
|
- getopt is used by several testscripts but it requires more than what
|
|
the native getopt(1) delivers. So best is to install getopt from ports
|
|
and remove/replace the getopt in /usr/bin.
|
|
|
|
Getting the FreeBSD work on Ceph:
|
|
=================================
|
|
|
|
- cd "place to work on this"
|
|
git clone https://github.com/wjwithagen/ceph.git
|
|
cd ceph
|
|
git checkout wip-wjw-freebsd-cmake
|
|
|
|
Building Ceph
|
|
=============
|
|
- Go and start building
|
|
./do_freebsd.sh
|
|
|
|
Parts not (yet) included:
|
|
=========================
|
|
|
|
- RBD
|
|
Rados Block Devices is implemented in the Linux kernel
|
|
It seems that there used to be a userspace implementation first.
|
|
And perhaps ggated could be used as a template since it does some of
|
|
the same, other than just between 2 disks. And it has a userspace
|
|
counterpart.
|
|
- BlueStore.
|
|
FreeBSD and Linux have different AIO API, and that needs to be made
|
|
compatible Next to that is there discussion in FreeBSD about
|
|
aio_cancel not working for all devicetypes
|
|
- CephFS
|
|
Cython tries to access an internal field in dirent which does not
|
|
compile
|
|
|
|
Tests that verify the correct working of the above are also excluded
|
|
from the testset
|
|
|
|
Tests not (yet) include:
|
|
=======================
|
|
|
|
- None, although some test can fail if running tests in parallel and there is
|
|
not enough swap. Then tests will start to fail in strange ways.
|
|
|
|
Task to do:
|
|
===========
|
|
- Build an automated test platform that will build ceph/master on
|
|
FreeBSD and report the results back to the Ceph developers. This will
|
|
increase the maintainability of the FreeBSD side of things.
|
|
Developers are signalled that they are using Linux-isms that will not
|
|
compile/run on FreeBSD Ceph has several projects for this: Jenkins,
|
|
teuthology, pulpito, ...
|
|
But even just a while { compile } loop and report the build data on a
|
|
static webpage would do for starters.
|
|
|
|
- Run integration tests to see if the FreeBSD daemons will work with a
|
|
Linux Ceph platform.
|
|
|
|
- Compile and test the user space RBD (Rados Block Device).
|
|
|
|
- Investigate and see if an in-kernel RBD device could be developed a la
|
|
'ggate'
|
|
|
|
- Investigate the keystore, which could be kernel embedded on Linux an
|
|
currently prevents building Cephfs and some other parts.
|
|
|
|
- Scheduler information is not used atm, because the schedulers work
|
|
rather different. But at a certain point in time, this would need some
|
|
attention:
|
|
in: ./src/common/Thread.cc
|
|
|
|
- Integrate the FreeBSD /etc/rc.d init scripts in the Ceph stack. Both
|
|
for testing, but mainly for running Ceph on production machines.
|