mirror of https://github.com/ceph/ceph
135 lines
7.1 KiB
ReStructuredText
135 lines
7.1 KiB
ReStructuredText
=======================
|
|
Zoned Storage Support
|
|
=======================
|
|
|
|
http://zonedstorage.io
|
|
|
|
Zoned Storage is a class of storage devices that enables host and storage
|
|
devices to cooperate to achieve higher storage capacities, increased throughput,
|
|
and lower latencies. The zoned storage interface is available through the SCSI
|
|
Zoned Block Commands (ZBC) and Zoned Device ATA Command Set (ZAC) standards on
|
|
Shingled Magnetic Recording (SMR) hard disks today and is also being adopted for
|
|
NVMe Solid State Disks with the upcoming NVMe Zoned Namespaces (ZNS) standard.
|
|
|
|
This project aims to enable Ceph to work on zoned storage drives and at the same
|
|
time explore research problems related to adopting this new interface. The
|
|
first target is to enable non-overwrite workloads (e.g. RGW) on host-managed SMR
|
|
(HM-SMR) drives and explore cleaning (garbage collection) policies. HM-SMR
|
|
drives are high capacity hard drives with the ZBC/ZAC interface. The longer
|
|
term goal is to support ZNS SSDs, as they become available, as well as overwrite
|
|
workloads.
|
|
|
|
The first patch in these series enabled writing data to HM-SMR drives. This
|
|
patch introduces ZonedFreelistManger, a FreelistManager implementation that
|
|
passes enough information to ZonedAllocator to correctly initialize state of
|
|
zones by tracking the write pointer and the number of dead bytes per zone. We
|
|
have to introduce a new FreelistManager implementation because with zoned
|
|
devices a region of disk can be in three states (empty, used, and dead), whereas
|
|
current BitmapFreelistManager tracks only two states (empty and used). It is
|
|
not possible to accurately initialize the state of zones in ZonedAllocator by
|
|
tracking only two states. The third planned patch will introduce a rudimentary
|
|
cleaner to form a baseline for further research.
|
|
|
|
Currently we can perform basic RADOS benchmarks on an OSD running on an HM-SMR
|
|
drives, restart the OSD, and read the written data, and write new data, as can
|
|
be seen below.
|
|
|
|
Please contact Abutalib Aghayev <agayev@psu.edu> for questions.
|
|
|
|
::
|
|
|
|
$ sudo zbd report -i -n /dev/sdc
|
|
Device /dev/sdc:
|
|
Vendor ID: ATA HGST HSH721414AL T240
|
|
Zone model: host-managed
|
|
Capacity: 14000.520 GB (27344764928 512-bytes sectors)
|
|
Logical blocks: 3418095616 blocks of 4096 B
|
|
Physical blocks: 3418095616 blocks of 4096 B
|
|
Zones: 52156 zones of 256.0 MB
|
|
Maximum number of open zones: no limit
|
|
Maximum number of active zones: no limit
|
|
52156 / 52156 zones
|
|
$ MON=1 OSD=1 MDS=0 sudo ../src/vstart.sh --new --localhost --bluestore --bluestore-devs /dev/sdc --bluestore-zoned
|
|
<snipped verbose output>
|
|
$ sudo ./bin/ceph osd pool create bench 32 32
|
|
pool 'bench' created
|
|
$ sudo ./bin/rados bench -p bench 10 write --no-cleanup
|
|
hints = 1
|
|
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
|
|
Object prefix: benchmark_data_h0.cc.journaling712.narwhal.p_29846
|
|
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
|
|
0 0 0 0 0 0 - 0
|
|
1 16 45 29 115.943 116 0.384175 0.407806
|
|
2 16 86 70 139.949 164 0.259845 0.391488
|
|
3 16 125 109 145.286 156 0.31727 0.404727
|
|
4 16 162 146 145.953 148 0.826671 0.409003
|
|
5 16 203 187 149.553 164 0.44815 0.404303
|
|
6 16 242 226 150.621 156 0.227488 0.409872
|
|
7 16 281 265 151.384 156 0.411896 0.408686
|
|
8 16 320 304 151.956 156 0.435135 0.411473
|
|
9 16 359 343 152.401 156 0.463699 0.408658
|
|
10 15 396 381 152.356 152 0.409554 0.410851
|
|
Total time run: 10.3305
|
|
Total writes made: 396
|
|
Write size: 4194304
|
|
Object size: 4194304
|
|
Bandwidth (MB/sec): 153.333
|
|
Stddev Bandwidth: 13.6561
|
|
Max bandwidth (MB/sec): 164
|
|
Min bandwidth (MB/sec): 116
|
|
Average IOPS: 38
|
|
Stddev IOPS: 3.41402
|
|
Max IOPS: 41
|
|
Min IOPS: 29
|
|
Average Latency(s): 0.411226
|
|
Stddev Latency(s): 0.180238
|
|
Max latency(s): 1.00844
|
|
Min latency(s): 0.108616
|
|
$ sudo ../src/stop.sh
|
|
$ # Notice the lack of "--new" parameter to vstart.sh
|
|
$ MON=1 OSD=1 MDS=0 sudo ../src/vstart.sh --localhost --bluestore --bluestore-devs /dev/sdc --bluestore-zoned
|
|
<snipped verbose output>
|
|
$ sudo ./bin/rados bench -p bench 10 rand
|
|
hints = 1
|
|
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
|
|
0 0 0 0 0 0 - 0
|
|
1 16 61 45 179.903 180 0.117329 0.244067
|
|
2 16 116 100 199.918 220 0.144162 0.292305
|
|
3 16 174 158 210.589 232 0.170941 0.285481
|
|
4 16 251 235 234.918 308 0.241175 0.256543
|
|
5 16 316 300 239.914 260 0.206044 0.255882
|
|
6 15 392 377 251.206 308 0.137972 0.247426
|
|
7 15 458 443 252.984 264 0.0800146 0.245138
|
|
8 16 529 513 256.346 280 0.103529 0.239888
|
|
9 16 587 571 253.634 232 0.145535 0.2453
|
|
10 15 646 631 252.254 240 0.837727 0.246019
|
|
Total time run: 10.272
|
|
Total reads made: 646
|
|
Read size: 4194304
|
|
Object size: 4194304
|
|
Bandwidth (MB/sec): 251.558
|
|
Average IOPS: 62
|
|
Stddev IOPS: 10.005
|
|
Max IOPS: 77
|
|
Min IOPS: 45
|
|
Average Latency(s): 0.249385
|
|
Max latency(s): 0.888654
|
|
Min latency(s): 0.0103208
|
|
$ sudo ./bin/rados bench -p bench 10 write --no-cleanup
|
|
hints = 1
|
|
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
|
|
Object prefix: benchmark_data_h0.aa.journaling712.narwhal.p_64416
|
|
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
|
|
0 0 0 0 0 0 - 0
|
|
1 16 46 30 119.949 120 0.52627 0.396166
|
|
2 16 82 66 131.955 144 0.48087 0.427311
|
|
3 16 123 107 142.627 164 0.3287 0.420614
|
|
4 16 158 142 141.964 140 0.405177 0.425993
|
|
5 16 192 176 140.766 136 0.514565 0.425175
|
|
6 16 224 208 138.635 128 0.69184 0.436672
|
|
7 16 261 245 139.967 148 0.459929 0.439502
|
|
8 16 301 285 142.468 160 0.250846 0.434799
|
|
9 16 336 320 142.189 140 0.621686 0.435457
|
|
10 16 374 358 143.166 152 0.460593 0.436384
|
|
|