RepoMirrors/ceph
1
0
Fork 0
mirror of https://github.com/ceph/ceph synced 2025-01-02 00:52:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

compressor/lz4: rebuild if buffer is not contiguous

Browse Source 
In older versions of lz4 (specifically < 1.8.2) bit errors
can be introduced when compressing from fragmented memory.  The lz4
bug was fixed by this lz4 commit:

commit af127334670a5e7b710bbd6adb71aa7c3ef0cd72
Author: Yann Collet <cyan@fb.com>
Date:   Sat May 5 18:24:11 2018 -0700

    fixed frametest error

    The error can be reproduced using following command :
    ./frametest -v -i100000000 -s1659 -t31096808

    It's actually a bug in the stream LZ4 API,
    when starting a new stream
    and providing a first chunk to complete with size < MINMATCH.
    In which case, the chunk becomes a dictionary.
    No hash was generated and stored,
    but the chunk is accessible as default position 0 points to dictStart,
    and position 0 is still within MAX_DISTANCE.
    Then, next attempt to read 32-bits from position 0 fails.

    The issue would have been mitigated by starting from index 64 KB,
    effectively eliminating position 0 as too far away.

    The proper fix is to eliminate such "dictionary" as too small.
    Which is what this patch does.

This is a workaround to rebuild our input buffer into a continguos buffer
if it is not already contiguous.

Fixes: https://tracker.ceph.com/issues/39525
Signed-off-by: Sage Weil <sage@redhat.com>
This commit is contained in:
Sage Weil 2020-02-27 09:30:27 -06:00
parent 8c8fb59dcf
commit 8e80f5f678
1 changed files with 12 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
src/compressor/lz4/LZ4Compressor.h
View File

@ -36,6 +36,18 @@ class LZ4Compressor : public Compressor {
}
int compress(const bufferlist &src, bufferlist &dst) override {
// older versions of liblz4 introduce bit errors when compressing
// fragmented buffers. this was fixed in lz4 commit
// af127334670a5e7b710bbd6adb71aa7c3ef0cd72, which first
// appeared in v1.8.2.
//
// workaround: rebuild if not contiguous.
if (!src.is_contiguous()) {
bufferlist new_src = src;
new_src.rebuild();
return compress(new_src, dst);
}
#ifdef HAVE_QATZIP
if (qat_enabled)
return qat_accel.compress(src, dst);