The messages of ioctl-test related build is not aligned with other
commands:
[LD] hash-speedtest
[LD] hash-vectest
[CC] ioctl-test.o
[LD] ioctl-test
Signed-off-by: David Sterba <dsterba@suse.com>
The test programs are not build by default but we want at least the
build coverage, so add them. Also make the build parallel.
Signed-off-by: David Sterba <dsterba@suse.com>
There are some stale headers that we don't need and the int types are
using the kernel types and pull kerncompat.h. As this is a basic header
that should minimize dependencies use the standard int types.
Signed-off-by: David Sterba <dsterba@suse.com>
Now that there are more implementations for the hashes test them all on
the vectors if the CPU supports that.
Signed-off-by: David Sterba <dsterba@suse.com>
Add test vectors that are longer that the internal block length.
Accelerated implementations may not be used on the short or unpaded
blocks but we need to test them as well.
Signed-off-by: David Sterba <dsterba@suse.com>
Copy sha256-x86.c from https://github.com/noloader/SHA-Intrinsics, that
uses the compiler intrinsics to implement the update step with the
native x86_64 instructions.
To avoid dependencies of the reference code and the x86 version, check
runtime support only if the compiler also supports -msha.
Signed-off-by: David Sterba <dsterba@suse.com>
Benchmark all accelerated implementations if the CPU supports them. Set
the level before each test, expecting that the implementation switches
the implementation dynamically.
Signed-off-by: David Sterba <dsterba@suse.com>
Copy AVX2 implementation from https://github.com/sneves/blake2-avx2 .
Though this is marked experimental, libsodium uses this version.
Signed-off-by: David Sterba <dsterba@suse.com>
Copy implementation from https://github.com/BLAKE2/BLAKE2, add runtime
detection of SSE2 and add the switch function.
Signed-off-by: David Sterba <dsterba@suse.com>
Add support for run-time detection of CPU features on x86_64 to allow
selection of accelerated implementations of hash algorithms.
When possible use the compiler builtin (works on gcc and clang).
The SHA extensions can't be detected by __builtin_cpu_supports and the
__cpuid/__cpuidex macros are not consistently provided in all supported
gcc and clang versions. Copy the __cpuidex and call it manually for the
SHA extensions. Complete list https://en.wikipedia.org/wiki/CPUID .
Signed-off-by: David Sterba <dsterba@suse.com>
[WARNING]
Clang 15.0.7 warns about several unused variables:
kernel-shared/zoned.c:829:6: warning: variable 'num_sequential' set but not used [-Wunused-but-set-variable]
u32 num_sequential = 0, num_conventional = 0;
^
cmds/scrub.c:1174:6: warning: variable 'n_skip' set but not used [-Wunused-but-set-variable]
int n_skip = 0;
^
mkfs/main.c:493:6: warning: variable 'total_block_count' set but not used [-Wunused-but-set-variable]
u64 total_block_count = 0;
^
image/main.c:2246:6: warning: variable 'bytenr' set but not used [-Wunused-but-set-variable]
u64 bytenr = 0;
^
[CAUSE]
Most of them are just straightforward set but not used variables.
The only exception is total_block_count, which has commented out code
relying on it.
[FIX]
Just remove those variables, and for @total_block_count, also remove the
comments.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[WARNING]
Clang 15.0.7 gives the following warning:
image/main.c:95:2: warning: field '' with variable sized type 'union metadump_struct::(anonymous at image/main.c:95:2)' not at the end of a struct or class is a GNU extension [-Wgnu-variable-sized-type-not-at-end]
union {
^
[CAUSE]
The union contains meta_cluster, which variable sized:
struct meta_cluster {
struct meta_cluster_header header;
struct meta_cluster_item items[];
} __attribute__ ((__packed__));
Thus clang gives above warning since it's a GNU extension.
[FIX]
Just move the union to the end of the structure.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[FALSE ALERT]
Unlike gcc, clang doesn't really understand the comments, thus it's
reportings tons of fall through related errors:
cmds/reflink.c:124:3: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
case 'r':
^
cmds/reflink.c:124:3: note: insert '__attribute__((fallthrough));' to silence this warning
case 'r':
^
__attribute__((fallthrough));
cmds/reflink.c:124:3: note: insert 'break;' to avoid fall-through
case 'r':
^
break;
[CAUSE]
Although gcc is fine with /* fallthrough */ comments, clang is not.
[FIX]
So just introduce a fallthrough macro to handle the situation properly,
and use that macro instead.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[FALSE ALERT]
Clang 15.0.7 gives the following false alert in get_dev_extent_len():
kernel-shared/extent-tree.c:3328:2: warning: variable 'div' is used uninitialized whenever switch default is taken [-Wsometimes-uninitialized]
default:
^~~~~~~
kernel-shared/extent-tree.c:3332:24: note: uninitialized use occurs here
return map->ce.size / div;
^~~
kernel-shared/extent-tree.c:3311:9: note: initialize the variable 'div' to silence this warning
int div;
^
= 0
And one in btrfs_stripe_length() too:
kernel-shared/volumes.c:2781:2: warning: variable 'stripe_len' is used uninitialized whenever switch default is taken [-Wsometimes-uninitialized]
default:
^~~~~~~
kernel-shared/volumes.c:2785:9: note: uninitialized use occurs here
return stripe_len;
^~~~~~~~~~
kernel-shared/volumes.c:2754:16: note: initialize the variable 'stripe_len' to silence this warning
u64 stripe_len;
^
= 0
[CAUSE]
Clang doesn't really understand what BUG_ON() means, thus in that
default case, we won't get uninitialized value but crash directly.
[FIX]
Silent the errors by assigning the default value properly using the
value of SINGLE profile.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[FALSE ALERT]
With clang 15.0.7, there is a false alert on uninitialized value in
ctree.c:
kernel-shared/ctree.c:3418:13: warning: variable 'offset' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
} else if (ret < 0) {
^~~~~~~
kernel-shared/ctree.c:3428:41: note: uninitialized use occurs here
write_extent_buffer(eb, &subvol_id_le, offset, sizeof(subvol_id_le));
^~~~~~
kernel-shared/ctree.c:3418:9: note: remove the 'if' if its condition is always true
} else if (ret < 0) {
^~~~~~~~~~~~~
kernel-shared/ctree.c:3380:22: note: initialize the variable 'offset' to silence this warning
unsigned long offset;
^
= 0
kernel-shared/ctree.c:3418:13: warning: variable 'eb' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
} else if (ret < 0) {
^~~~~~~
kernel-shared/ctree.c:3429:26: note: uninitialized use occurs here
btrfs_mark_buffer_dirty(eb);
^~
kernel-shared/ctree.c:3418:9: note: remove the 'if' if its condition is always true
} else if (ret < 0) {
^~~~~~~~~~~~~
kernel-shared/ctree.c:3378:26: note: initialize the variable 'eb' to silence this warning
struct extent_buffer *eb;
^
= NULL
[CAUSE]
The original code is handling the return value from
btrfs_insert_empty_item() like this:
ret = btrfs_insert_empty_item();
if (ret >= 0) {
/* Do something for it. */
} else if (ret == -EEXIST) {
/* Do something else. */
} else if (ret < 0) {
/* Error handling. */
}
But the problem is, the last one check is always true if we can reach
there.
Thus clang is providing the hint to remove the if () check.
[FIX]
Normally we prefer to do error handling first, so move the error
handling first so we don't need the if () else if () chain.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before decompressing, we zero out the content of the entire output buffer,
so that we don't get any garbage after the last byte of data. We do this
for all compression algorithms. However zstd, at least with libzstd 1.5.2
on Debian (version 1.5.2+dfsg-1), the decompression routine can end up
touching the content of the output buffer beyond the last valid byte of
decompressed data, resulting in a corruption.
Example reproducer:
$ cat test.sh
#!/bin/bash
DEV=/dev/sdj
MNT=/mnt/sdj
rm -f /tmp/send.stream
umount $DEV &> /dev/null
mkfs.btrfs -f $DEV &> /dev/null || echo "MKFS failed!"
mount -o compress=zstd $DEV $MNT
# File foo is not sector size aligned, 127K.
xfs_io -f -c "pwrite -S 0xab 0 3" \
-c "pwrite -S 0xcd 3 130042" \
-c "pwrite -S 0xef 130045 3" $MNT/foo
# Now do an fallocate that increases the size of foo from 127K to 128K.
xfs_io -c "falloc 0 128K " $MNT/foo
btrfs subvolume snapshot -r $MNT $MNT/snap
btrfs send --compressed-data -f /tmp/send.stream $MNT/snap
echo -e "\nFile data in the original filesystem:\n"
od -A d -t x1 $MNT/snap/foo
umount $MNT
mkfs.btrfs -f $DEV &> /dev/null || echo "MKFS failed!"
mount $DEV $MNT
btrfs receive --force-decompress -f /tmp/send.stream $MNT
echo -e "\nFile data in the new filesystem:\n"
od -A d -t x1 $MNT/snap/foo
umount $MNT
Running the reproducer gives:
$ ./test.sh
(...)
File data in the original filesystem:
0000000 ab ab ab cd cd cd cd cd cd cd cd cd cd cd cd cd
0000016 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
*
0130032 cd cd cd cd cd cd cd cd cd cd cd cd cd ef ef ef
0130048 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
0131072
At subvol snap
File data in the new filesystem:
0000000 ab ab ab cd cd cd cd cd cd cd cd cd cd cd cd cd
0000016 cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd cd
*
0130032 cd cd cd cd cd cd cd cd cd cd cd cd cd ef ef ef
0130048 cd cd cd cd 00 00 00 00 00 00 00 00 00 00 00 00
0130064 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
0131072
The are 4 bytes with a value of 0xcd instead of 0x00, at file offset
127K (130048).
Fix this by explicitly zeroing out the part of the output buffer that was
not used after decompressing with zstd.
The decompression of compressed extents, sent when using the send v2
stream, happens in the following cases:
1) By explicitly passing --force-decompress to the receive command, as in
the reproducer above;
2) Calling the BTRFS_IOC_ENCODED_WRITE ioctl failed with -ENOTTY, meaning
the kernel on the receiving side is old and does not implement that
ioctl;
3) Calling the BTRFS_IOC_ENCODED_WRITE ioctl failed with -ENOSPC;
4) Calling the BTRFS_IOC_ENCODED_WRITE ioctl failed with -EINVAL.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Move the source file to own directory so it can be further split and
refactored. File needs to be renamed to main.c so the build magic works.
Signed-off-by: David Sterba <dsterba@suse.com>
By default print how many subvolumes are considered for checks, either
found or specified on the command line. Once a subvolume is removed from
the list print the progress from the total count.
$ btrfs subvolume sync /path
Waiting for 130 subvolumes
Subvolume id 256 is gone (1/130)
Subvolume id 257 is gone (2/130)
...
Subvolume id 384 is gone (129/130)
Subvolume id 385 is gone (130/130)
Signed-off-by: David Sterba <dsterba@suse.com>
Use the configured widths and print padding directly instead of the
embedded printf format and fixed width strings.
Signed-off-by: David Sterba <dsterba@suse.com>
To make option formatting a bit easier so the spacing is unified add
macros and formatting helpers.
Usage in the help text:
OPTLINE("-o value", "description")
Internally the option and description are delimiters by chars that are
not part of normal text, the formatter separates that and uses fixed
with for output. The description text can be of any length, multi-line
text should still end up as one token (i.e. newline without ',' between).
Signed-off-by: David Sterba <dsterba@suse.com>
Input must be prepared and output must be interpreted under the C
locale, which requires special precautions on the client’s side.
Pull-request: #561
Author: Christopher Yeleighton <ne01026@shark.2a.pl>
Signed-off-by: David Sterba <dsterba@suse.com>
There are two "bg_reclaim_threshold" under the sysfs directory. One is at
/sys/fs/btrfs/<UUID>/ and sets the threshold to start the auto reclaim
thread. The other one is
at/sys/fs/btrfs/<UUID>/allocations/{data,metadata,system} and sets the
threshold to reclaim a block group.
These two options have the same name but they are calculated against
different metrics. The former is a percentage of allocated (for a device
extent) space on total device space, and the latter is a percentage of
reclaimable space on a block group's zone capacity.
Add description for per-space_info bg_reclaim_threshold to distinguish
these two same name configurations.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[PROBLEM]
Btrfs check original mode output is not that reader friendly already, it
even includes pointer output:
backref 15353727729664 parent 1140559929556992 not referenced back 0xc9133d70
tree backref 15353727729664 parent 14660022714368 not found in extent tree
incorrect global backref count on 15353727729664 found 3 wanted 2
backpointer mismatch on [15353727729664 16384]
In above case, the "0xc9133d70" is completely useless, as it's a pointer
for the tree_backref structure.
And the term "backref" is quite abused in above case.
[ENHANCEMENT]
To enhance the situation, let's use some output format from lowmem mode
instead.
Now above example will be changed to:
tree extent[15353727729664, 16384] parent 1140559929556992 has no tree block found
tree extent[15353727729664, 16384] parent 14660022714368 has no backref item in extent tree
incorrect global backref count on 15353727729664 found 3 wanted 2
backpointer mismatch on [15353727729664 16384]
And some example for data backrefs:
data extent[12845056, 1048576] bytenr mimsmatch, extent item bytenr 12845056 file item bytenr 0
data extent[12845056, 1048576] referencer count mismatch (root 5 owner 257 offset 0) wanted 1 have 0
data extent[14233600, 12288] referencer count mismatch (parent 42139648) wanted 0 have 1
data extent[14233600, 12288] referencer count mismatch (root 5 owner 307 offset 0) wanted 0 have 1
data extent[14233600, 12288] referencer count mismatch (parent 30507008) wanted 0 have 1
Furthermore, the original function print_tree_backref_error() is a mess
already, here we clean it up by exacting all the error output into a
dedicated helper, print_backref_error(), so the function itself only
need to find out errors.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Prior to version 2.38, libblkid fails to detect zoned mode's superblock
location resulting in blkid failing to detect btrfs on zoned block
devices. This patch suggest to the user to upgrade libblkid if it
detects a version lower then 2.38.
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The copyright config variable can be used for a real copyright but we
don't have that for the manual pages and it was a stub. The date of page
generation time is added to the page anyway.
Signed-off-by: David Sterba <dsterba@suse.com>
This patch will add a dedicated section for btrfs sysfs interfaces.
It will include:
- Directory layout explanation
Including:
* Description
* Introduced in which kernel version
- Files explanation
Including:
* RW/RO type
* Description
* Introduced in which kernel version
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Concrete use-case:
- I want a static build to run in a small test VM.
- RedHat regrettably does not provide util-linux static libraries
- like libuuid.a and libblkid.a.
- So, one has to build them from source.
- Installing these to /usr is lame.
- This change lets me do make btrfs.static EXTRA_LDFLAGS=-L/path/to/util-linux-2.38/.libs
Pull-request: #563
Author: Alexey <snarkmaster@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Per user report on https://old.reddit.com/r/btrfs/comments/107fnw1/btrfs_filesystem_mkswapfile_results_in_an/
the swapfile header does not contain the correct number of pages that
matches the file size and the activated swapfile is only 1GiB:
# btrfs filesystem mkswapfile -s 10g swapfile
# swapon swapfile
# cat /proc/swaps
Filename Type Size Used Priority
/swap/swapfile file 1048572 0 -2
A workaround is to run 'mkswap swapfile' before activation. Proper fix
is to calculate the number of (fixed size) 4K pages available for the
swap.
Issue: #568
Signed-off-by: David Sterba <dsterba@suse.com>
