mirror of
https://github.com/ceph/ceph
synced 2024-12-16 08:26:25 +00:00
102 lines
4.7 KiB
ReStructuredText
102 lines
4.7 KiB
ReStructuredText
===========================================
|
|
Hard Disk and File System Recommendations
|
|
===========================================
|
|
|
|
Hard Disk Prep
|
|
==============
|
|
|
|
Ceph aims for data safety, which means that when the application receives notice
|
|
that data was written to the disk, that data was actually written to the disk.
|
|
For old kernels (<2.6.33), disable the write cache if the journal is on a raw
|
|
disk. Newer kernels should work fine.
|
|
|
|
Use ``hdparm`` to disable write caching on the hard disk::
|
|
|
|
sudo hdparm -W 0 /dev/hda 0
|
|
|
|
In production environments, we recommend running OSDs with an operating system
|
|
disk, and a separate disk(s) for data. If you run data and an operating system
|
|
on a single disk, create a separate partition for your data before configuring
|
|
your OSD cluster.
|
|
|
|
|
|
File Systems
|
|
============
|
|
|
|
Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying file
|
|
system for:
|
|
|
|
- Internal object state
|
|
- Snapshot metadata
|
|
- RADOS Gateway Access Control Lists (ACLs).
|
|
|
|
Ceph OSDs rely heavily upon the stability and performance of the underlying file
|
|
system. The underlying file system must provide sufficient capacity for XATTRs.
|
|
File system candidates for Ceph include B tree and B+ tree file systems such as:
|
|
|
|
- ``btrfs``
|
|
- ``XFS``
|
|
|
|
If you are using ``ext4``, mount your file system to enable XATTRs. You must also
|
|
add the following line to the ``[osd]`` section of your ``ceph.conf`` file. ::
|
|
|
|
filestore xattr use omap = true
|
|
|
|
.. warning:: XATTR limits.
|
|
|
|
The RADOS Gateway's ACL and Ceph snapshots easily surpass the 4-kilobyte limit
|
|
for XATTRs in ``ext4``, causing the ``ceph-osd`` process to crash. Version 0.45
|
|
or newer uses ``leveldb`` to bypass this limitation. ``ext4`` is a poor file
|
|
system choice if you intend to deploy the RADOS Gateway or use snapshots on
|
|
versions earlier than 0.45.
|
|
|
|
.. tip:: Use ``xfs`` initially and ``btrfs`` when it is ready for production.
|
|
|
|
The Ceph team believes that the best performance and stability will come from
|
|
``btrfs.`` The ``btrfs`` file system has internal transactions that keep the
|
|
local data set in a consistent state. This makes OSDs based on ``btrfs`` simple
|
|
to deploy, while providing scalability not currently available from block-based
|
|
file systems. The 64-kb XATTR limit for ``xfs`` XATTRS is enough to accommodate
|
|
RDB snapshot metadata and RADOS Gateway ACLs. So ``xfs`` is the second-choice
|
|
file system of the Ceph team in the long run, but ``xfs`` is currently more
|
|
stable than ``btrfs``. If you only plan to use RADOS and ``rbd`` without
|
|
snapshots and without ``radosgw``, the ``ext4`` file system should work just fine.
|
|
|
|
FS Background Info
|
|
==================
|
|
|
|
Before ``ext3``, ``ReiserFS`` was the only journaling file system available for
|
|
Linux. However, ``ext3`` doesn't provide Extended Attribute (XATTR) support.
|
|
While ``ext4`` provides XATTR support, it only allows XATTRs up to 4kb. The
|
|
4kb limit is not enough for RADOS GW ACLs, snapshots, and other features. As of
|
|
version 0.45, Ceph provides a ``leveldb`` feature for ``ext4`` file systems
|
|
that stores XATTRs in excess of 4kb in a ``leveldb`` database.
|
|
|
|
The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly
|
|
scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``.
|
|
Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that
|
|
they are more robust when recovering from crashes, power outages, etc. These
|
|
filesystems journal all of the changes they will make before performing writes.
|
|
|
|
``XFS`` was developed for Silicon Graphics, and is a mature and stable
|
|
filesystem. By contrast, ``btrfs`` is a relatively new file system that aims
|
|
to address the long-standing wishes of system administrators working with
|
|
large scale data storage environments. ``btrfs`` has some unique features
|
|
and advantages compared to other Linux filesystems.
|
|
|
|
``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation
|
|
timestamps and checksums that verify metadata integrity, so it can detect
|
|
bad copies of data and fix them with the good copies. The copy-on-write
|
|
capability means that ``btrfs`` can support snapshots that are writable.
|
|
``btrfs`` supports transparent compression and other features.
|
|
|
|
``btrfs`` also incorporates multi-device management into the file system,
|
|
which enables you to support heterogeneous disk storage infrastructure,
|
|
data allocation policies. The community also aims to provide ``fsck``,
|
|
deduplication, and data encryption support in the future. This compelling
|
|
list of features makes ``btrfs`` the ideal choice for Ceph clusters.
|
|
|
|
.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write
|
|
.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems
|
|
.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system
|