Interested in working on Ceph?

We are actively seeking experienced C/C++ and Linux kernel developers who are interested in helping turn Ceph into a stable production-grade storage system. Competitive salaries, benefits, etc. If interested, please contact sage at newdream dot net.

Welcome

Ceph is a distributed network file system designed to provide excellent performance, reliability, and scalability. Ceph fills two significant gaps in the array of currently available file systems:
  1. Robust, open-source distributed storage -- Ceph is released under the terms of the LGPL, which means it is free software (as in speech and beer). Ceph will provide a variety of key features that are generally lacking from existing open-source file systems, including seamless scalability (the ability to simply add disks to expand volumes), intelligent load balancing, and efficient, easy to use snapshot functionality.
  2. Scalability -- Ceph is built from the ground up to seamlessly and gracefully scale from gigabytes to petabytes and beyond. Scalability is considered in terms of workload as well as total storage. Ceph is designed to handle workloads in which tens thousands of clients or more simultaneously access the same file, or write to the same directory--usage scenarios that bring typical enterprise storage systems to their knees.
Here are some of the key features that make Ceph different from existing file systems that you may have worked with:
  1. Seamless scaling -- A Ceph filesystem can be seamlessly expanded by simply adding storage nodes (OSDs). However, unlike most existing file systems, Ceph proactively migrates data onto new devices in order to maintain a balanced distribution of data. This effectively utilizes all available resources (disk bandwidth and spindles) and avoids data hot spots (e.g., active data residing primarly on old disks while newer disks sit empty and idle).
  2. Strong reliability and fast recovery -- All data in Ceph is replicated across multiple OSDs. If any OSD fails, data is automatically re-replicated to other devices. However, unlike typical RAID systems, the replicas for data on each disk are spread out among a large number of other disks, and when a disk fails, the replacement replicas are also distributed across many disks. This allows recovery to proceed in parallel (with dozens of disks copying to dozens of other disks), removing the need for explicit "spare" disks (which are effectively wasted until they are needed) and preventing a single disk from becoming a "RAID rebuild" bottleneck.
  3. Adaptive MDS -- The Ceph metadata server (MDS) is designed to dynamically adapt its behavior to the current workload. As the size and popularity of the file system hierarchy changes over time, that hierarchy is dynamically redistributed among available metadata servers in order to balance load and most effectively use server resources. (In contrast, current file systems force system administrators to carve their data set into static "volumes" and assign volumes to servers. Volume sizes and workloads inevitably shift over time, forcing administrators to constantly shuffle data between servers or manually allocate new resources where they are currently needed.) Similarly, if thousands of clients suddenly access a single file or directory, that metadata is dynamically replicated across multiple servers to distribute the workload.
For more information about the underlying architecture of Ceph, please see the Overview. This project is based on a substantial body of research conducted by the Storage Systems Research Center at the University of California, Santa Cruz over the past few years that has resulted in a number of publications.

Current Status

Ceph is roughly alpha quality, and is under very active development. Ceph is not yet suitable for any uses other than testing and review. The file system is mountable and more or less usable using a FUSE-based client, and development is underway on a native Linux kernel client. Many features are planned but not yet implemented, including snapshots.

The Ceph project is actively seeking participants. If you are interested in using Ceph, or contributing to its development, please join the mailing list.

Please feel free to contact me with any questions or comments.