Although we already have a pretty good array defined for all
super/compat_ro/incompat flags, we still rely on a manually defined mask
to do the printing.
This can lead to easy de-sync between the definition and the flags.
Change it to automatically iterate through the array to calculate the
flags, and add the remaining super flags.
Pull-request: #810
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report that a canceled checksum conversion (still
experimental feature) resulted in unexpected super flags:
csum_type 0 (crc32c)
csum_size 4
csum 0x14973811 [match]
bytenr 65536
flags 0x1000000001
( WRITTEN |
CHANGING_FSID_V2 )
magic _BHRfS_M [match]
While for a filesystem under checksum conversion it should have either
CHANGING_DATA_CSUM or CHANGING_META_CSUM.
[CAUSE]
It turns out that, due to btrfs-progs keeps its own extra flags inside
its own ctree.h headers, not the shared uapi headers, we have
conflicting super flags:
kernel-shared/uapi/btrfs_tree.h:#define BTRFS_SUPER_FLAG_METADUMP_V2 (1ULL << 34)
kernel-shared/uapi/btrfs_tree.h:#define BTRFS_SUPER_FLAG_CHANGING_FSID (1ULL << 35)
kernel-shared/uapi/btrfs_tree.h:#define BTRFS_SUPER_FLAG_CHANGING_FSID_V2 (1ULL << 36)
kernel-shared/ctree.h:#define BTRFS_SUPER_FLAG_CHANGING_DATA_CSUM (1ULL << 36)
kernel-shared/ctree.h:#define BTRFS_SUPER_FLAG_CHANGING_META_CSUM (1ULL << 37)
Note that CHANGING_FSID_V2 is conflicting with CHANGING_DATA_CSUM.
[FIX]
Cross port the proper updated uapi headers into btrfs-progs, and remove
the definition from ctree.h.
This would change the value for CHANGING_DATA_CSUM and
CHANGING_META_CSUM, but considering they are experimental features, and
kernel would reject them anyway, the damage is not that huge and we can
accept such change before exposing it to end users.
Pull-request: #810
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a long existing failure in my local VM that with any newer
kernel (6.x) the test case misc/004 would fail with ENOSPC during
balance:
[TEST] misc-tests.sh
[TEST/misc] 004-shrink-fs
failed: /home/adam/btrfs-progs/btrfs balance start -mconvert=single -sconvert=single -f /home/adam/btrfs-progs/tests/mnt
test failed for case 004-shrink-fs
make: *** [Makefile:547: test-misc] Error 1
[CAUSE]
With more testing, it turns out that just before the balance, the
filesystem still have several empty data block groups.
The reason is the new default discard=async behavior, as it also changes
the empty block groups to be async, this leave the empty block groups
there, resulting no extra space for the convert balance.
[FIX]
I do not understand why for loop block devices we also enable
discard, but at least disable discard for the test case so that we can
ensure the empty block groups get cleaned up properly.
Pull-request: #809
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There's no obvious reason why there's -O1 instead -O2 which is commonly
used on distro builds. -O1 was enabled in c1690a3832 ("Switch to -O1
for optimizations to enable FORTIFY_SOURCE") for the fortify checks.
Signed-off-by: David Sterba <dsterba@suse.com>
Due to unknown cause the libbtrfsutil and libbtrfs are not built with
sanitizer libraries and the ASAN test does not succeed. This needs to be
analyzed why, for now disable it so CI can continue.
$ make D=asan test-libbtrfsutil
[PY] libbtrfsutil
==235341==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
Signed-off-by: David Sterba <dsterba@suse.com>
The test case always fails in my VM, with the following error:
$ sudo TEST=038\* make test-misc
[TEST] misc-tests.sh
[TEST/misc] 038-backup-root-corruption
Backup 2 not overwritten
test failed for case 038-backup-root-corruption
After more debugging, it turns out that there is nothing wrong except
the final check:
[ "$main_root_ptr" -ne "$backup_new_root_ptr" ] || _fail "Backup 2 not overwritten"
The _fail() is only triggered if the previous check returns false, which
is completely the opposite.
Furthermore on the github CI, the kernel commits 2 instead of 1
transaction, resulting the next slot never to match the current
generation/tree root.
The two bugs combined, github CI always passses the test case, while
for my VM which does the expected one transaction, it would always fail.
Fix it by:
- Use a proper "if [] then; fi" block to check the tree root bytenr
- Use the generation diff to calculate the expected backup root slot
- Log the full super block dump for debug usage
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There is a bug report that for fuzzed image
bko-155621-bad-block-group-offset.raw, "btrfs check --mode=lowmem
--repair" would lead to an endless loop.
Unlike original mode, lowmem mode relies on the backref walk to properly
go through each root, but unfortunately inside __add_inline_refs() we
doesn't handle unknown backref types correctly, causing it never moving
forward thus deadloop.
Fix it by erroring out to prevent an endless loop.
Issue: #788
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There is a bug report that with UBSAN enabled, fuzz/006 test case
crashes.
It turns out that the image bko-154021-invalid-drop-level.raw has
invalid dir items, that the name/data len is beyond the item.
And if we try to read beyond the eb boundary, UBSAN got triggered.
Normally in kernel tree-checker would reject such metadata in the first
place, but in btrfs-progs we can not be that strict or we cannot do a
lot of repair.
So here just enhance print_dir_item() to do extra sanity checks for
data/name len before reading the contents.
Issue: #805
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
ASAN build (make D=asan) detects a memory leak in
btrfs-corrupt-block inside debug_corrupt_sector().
This can be reproduced by fsck/013 test case.
The cause is pretty simple, we just malloc a sector and forgot to free
it.
Issue: #806
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Executing the script inside the directories as './test.sh' is not
supposed to work but could happen accidentally. With an exit after
attempting to source the we can fix that. Not all cases have been fixed
in f6bbe06c08 ("btrfs-progs: tests: add protection against running
out of test suite").
Signed-off-by: David Sterba <dsterba@suse.com>
The test case verifies behavior of ext4 unwritten extents:
- Create a unwritten (preallocated) extent on ext4
- Fill the on-disk extent with random garbage
This is to make sure if btrfs tries to read the on-disk data, it would
definitely get some garbage.
As I found sometimes mkfs.ext4 can fill the unused bg with zeros.
- Fill the preallocated file range with some data
This is to make sure btrfs-convert can handle mixed written and
unwritten ranges.
- Save the checksum of the file.
- Convert the fs
- Verify the checksum
For older btrfs-convert, there would be only one regular file extent,
and reading the file would read out some garbage and cause checksum to
mismatch.
For the fixed btrfs-convert, we punch holes for unwritten extents,
thus only the written part would be read out and match the checksum.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report that btrfs-convert cannot handle unwritten extents
(EXT2_EXTENT_FLAGS_UNINIT set, which is pretty much the same as
BTRFS_FILE_EXTENT_PREALLOC), which can cause the converted image to have
incorrect contents.
[CAUSE]
Currently we use ext2fs_block_iterate2() to go through all data extents
of an ext2 inode, but it doesn't provide the info on if the range is
unwritten or not.
Thus for unwritten extents, the results btrfs would just treat it as
regular extents, and read the contents from disk other than setting the
contents to zero.
[FIX]
Instead of the ext2fs_block_iterate2(), here we follow the debugfs'
"dump_extents" command, to use ext2fs_extent_*() helpers to go through
every data extent of the inode, that's if the inode supports the
EXT4_EXTENTS_FL flag.
Now we can properly get the info of which extents are unwritten, and use
holes to replace those unwritten extents.
Reported-by: Yordan <y16267966@gmail.com>
Link: https://lore.kernel.org/all/d34c7d77a7f00c93bea6a4d6e83c7caf.mailbg@mail.bg/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
If a btrfs filesystem had dev-replace ran in the past, even it's already
finished, btrfstune would refuse to change its csum:
WARNING: Experimental build with unstable or unfinished features
WARNING: Switching checksums is experimental, do not use for valuable data!
Proceed to switch checksums
ERROR: running dev-replace detected, please finish or cancel it.
ERROR: btrfstune failed
[CAUSE]
The current dev-replace detection is only checking if we have
DEV_REPLACE item in device tree.
However DEV_REPLACE item will also exist even if a dev-replace finished,
so the existing check can not handle such case at all.
[FIX]
If an dev-replace item is found, further check the state of the item to
prevent false alerts.
Issue: #798
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This is inspired by a recent bug that csum change doesn't detect
finished dev-replace.
At the time of that csum change patch, there is no print-tree to
show the content of btrfs_dev_replace_item thus contributes to the bug.
Add the new output for btrfs_dev_replace_item, and the example looks
like this:
item 1 key (0 DEV_REPLACE 0) itemoff 16171 itemsize 72
src devid -1 cursor left 1179648000 cursor right 1179648000 mode ALWAYS
state FINISHED write errors 0 uncorrectable read errors 0
start time 1717282771 (2024-06-02 08:29:31)
stop time 1717282771 (2024-06-02 08:29:31)
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
With the change of minimal number of zones, mkfs-tests/030-zoned-rst now
fails because the loopback device is 2GB and can contain 8x 256MB zones.
Use the nullb helpers to choose a smaller zone size.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add test for mkfs.btrfs zone reset behavior to check if
- it resets all the zones without "-b" option
- it detects an active zone outside of the FS range
- it do not reset a zone outside of the range
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Even with "mkfs.btrfs -b", mkfs.btrfs resets all the zones on the device.
Limit the reset target within the specified length.
Also, we need to check that there is no active zone outside of the FS
range. Having an active zone outside FS reduces the number of zones btrfs
can write simultaneously. Technically, we can still scan all the device
zones and keep active zones outside FS intact and try to live with the
limited active zones. But, that will make btrfs operations harder.
It is generally bad idea to use "-b" on a non-test usage on a device with
active zone limit in the first place. You really need to take care that FS
and outside the FS goes over the limit. That means you'll never be able to
use zones outside the FS anyway.
So, until there is a strong request for that, I don't think it's worthwhile
to do so.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
While "byte_count" is eventually rounded down to sectorsize at make_btrfs()
or btrfs_add_to_fs_id(), it would be better round it down first and do the
size checks not to confuse the things.
Also, on a zoned device, creating a filesystem whose size is not aligned
to the zone boundary can be confusing. Round it down further to the zone
boundary.
The size calculation with a source directory is also tweaked to be aligned.
device_get_partition_size_fd_stat() must be aligned down not to exceed the
device size. And, btrfs_mkfs_size_dir() should have return sectorsize aligned
size. So, add an UASSERT for it.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Currently, we check if a device is larger than 5 zones to determine we can
create btrfs on the device or not. Actually, we need more zones to create
DUP block groups, so it fails with "ERROR: not enough free space to
allocate chunk". Implement proper support for non-SINGLE profile.
Also, current code does not ensure we can create tree-log BG and data
relocation BG, which are essential for the real usage. Count them as
requirement too.
The calculation for a regular btrfs is also adjusted to use dev_stripes
style.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
We are going to implement a better minimum size calculation for the zoned
mode. Move the current logic to btrfs_min_dev_size() and unify the size
checking path.
Also, convert "int mixed" to "bool mixed" while at it.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
test_minimum_size() already checks if each device can host the initial
block groups. There is no need to check if the first device can host the
initial system chunk again.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
block_count and dev_block_count are counting the size in bytes. And,
comparing them with e.g, "min_dev_size" is confusing. Rename them to
represent the unit better.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The dependency generation and change tracking was completely broken.
Partly after changes to move the generated files to .deps/.
- the generic rule did work as it could not reflect that dependency file
is not next to the source file
- inclusion of dependency files directive never found them because the
list was not updated to look to .deps, also there were some ancient
files mentioned
- the dependency generation commands were run before each target,
slowing down build
How it works now:
- dependencies are created before each .c file is compiled, so that way
any changes to the file get translated to the dependency files
- missing dependencies are create after first run
- there's single dependency file for all build types (box, static, both)
- same as before, 'make clean' will also delete the dependency files
Signed-off-by: David Sterba <dsterba@suse.com>
After patch "btrfs-progs: qgroup: handle stale qgroup deletion more
accurately" cleaning stale qgroups may not happen if the subvolume
cleaning is still in progress. Update the test so it's able to handle
that.
Signed-off-by: David Sterba <dsterba@suse.com>
Run tests with enabled sanitizers. There are still known problems with
leaks that will make the whole fail. This needs to be fixed before the
workflow can be enabled for devel or master.
Signed-off-by: David Sterba <dsterba@suse.com>
Use unaligned access helper for code that potentially or actually
accesses data that come from on-disk structures. This is for image or
chunk restore. This may pessimize some cases but is in general safer on
strict alignment architectures and has no effect on other architectures.
Related issue #770.
Signed-off-by: David Sterba <dsterba@suse.com>
The search header is used for extracting data from buffer returned by
the SEARCH_TREE ioctl and needs special access helpers as there are no
guarantees about alignment.
With -fsanitize=alignment this still leads to an error because address
of the members is taken, regardless of the unaligned access method is
used (both temporary memcpy to a structure or the packed struct cast).
Add another hint to compiler that the structure is special and add the
packed attribute. This fixes the sanitizer error.
Signed-off-by: David Sterba <dsterba@suse.com>
There's a lot of places with unsafe access to data that come from a
search buffer, which is packed and the structures there are not
guaranteed to be aligned, also accessing the on-disk format structures.
- search header - this is an in-memory buffer with a series of on-disk
structures, no alignment must be assumed
- anything that's not a byte buffer must be accessed as an unaligned
buffer (the exceptions are name-like buffers)
Signed-off-by: David Sterba <dsterba@suse.com>
We will need generic helpers for unaligned access with LE->CPU
conversion, so add them. Should be use for potentially unaligned read
from tree search buffer.
Signed-off-by: David Sterba <dsterba@suse.com>
Fix parsing of send stream, properly access potentially unaligned data.
This can happen on hosts with strict alignment (ARM v5 or v6).
Related issue: #770
Signed-off-by: David Sterba <dsterba@suse.com>
There will be an unplanned update to libbtrfs (fixing send/receive
stream parsing and unaligned data access). The current ABI is frozen and
won't change but at least the patch level should change. Update the build
to create all links up to the major.minor.patch. Until now it was just
major.minor:
- libbtrfs.so -> libbtrfs.so.0.1.2
- libbtrfs.so.0 -> libbtrfs.so.0.1.2
- libbtrfs.so.0.1 -> libbtrfs.so.0.1.2
- libbtrfs.so.0.1.2
Signed-off-by: David Sterba <dsterba@suse.com>
Same fix a previous commit, unaligned access on strict alignment hosts
could produce wrong results (reported on send/receive and arm5). As
libbtrfs has own copy of the code fix it here too, replacing leXX_to_cpu
with get_unaligned_leXX where appropriate. This means any access to raw
buffers that get cast to a structure.
Issue: #770
Signed-off-by: David Sterba <dsterba@suse.com>
There's a report:
ERROR: Failed to send/receive subvolume: .../testbackup.20240330T1102 -> .../testbackup.20240330T1102
ERROR: ... Command execution failed (exitcode=1)
ERROR: ... sh: btrfs send '.../testbackup.20240330T1102' | ssh user@host.lan 'sudo -n btrfs receive '\''...'\'''
ERROR: ... invalid tlv in cmd tlv_type = 816
This is send/receive between arm64 and armv5el hosts, with btrfs-progs
6.2.1. Last known working version is 5.16. This looked like another
custom protocol extension by NAS vendors but this was a false trace and
this is indeed a bug in stream parsing after changes to the v2 protocol.
The most likely explanation is that the armv5 host requires strict
alignment for reads (32bit type must be 4 byte aligned) but the way the
raw data buffer is mapped to the cmd structure in read_cmd() does not
guarantee that.
Issue: #770
Fixes: aa1ca3789e ("btrfs-progs: receive: support v2 send stream DATA tlv format")
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report that, with very weird mount status, there can be
some mount source which can not be accessed:
/path/dev/exports fs 500G 57G 444G 12% /path/dev/exports
Strace shows we can not access the above mount source:
131065 stat("/path/dev/exports", 0x7ffed17b8e20) = -1 EACCES (Permission denied)
And lead to failed mount check:
131065 write(2, "ERROR: ", 7) = 7
131065 write(2, "cannot check mount status of /de"..., 56) = 56
131065 write(2, "\n", 1) = 1
[CAUSE]
The mount check is based on libblkid, which gives the mount source, and
for non-btrfs mounts, we call path_is_reg_or_block_device() to check if
we even need to continue checking.
But in above case, the mount source is another fs, and we can not access
the source.
So we error out causing the check_mounted() to return error.
[FIX]
There is never any guarantee we can access the mount source, but on the
other hand, I do not want to ignore all access failure for the mount
source.
Let test_status_for_mkfs() to only skip check_mounted() error if
@force_overwrite is true.
This would still keep the old strict checks on whether the target is
already mounted, but if the end user really knows that certain mount
source do not need to be checked, they can always pass "-f" option to
skip the false alerts.
Link: https://bugzilla.suse.com/show_bug.cgi?id=1223799
Reported-by: Jiri Belka <jiri.belka@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Recent patches updated stale qgroup handling, using 'unlinked' and
'dropped' where we otherwise use 'deleted' and 'cleaned'.
Signed-off-by: David Sterba <dsterba@suse.com>
Currently `btrfs qgroup show` command shows any 0 level qgroup without a
root backref as `<stale>`, which is not correct.
There are several more cases:
- Under deletion
The subvolume is not yet full dropped, but unlinked.
In that case we would not have a root backref item, but the qgroup is
not stale.
- Squota space holder
This is for squota mode, that a fully dropped subvolume still have
extents accounting on the already-gone subvolume.
In this case it's not stale either, and future accounting relies on
it.
This patch would add above special cases, and add an extra `SPECIAL
PATHS` section to explain all the cases, including `<stale>`.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The current stale qgroup deletion doesn't handle the following cases at
all:
- It doesn't detect stale qgroups correctly
The current check is using the root backref, which means unlinked but
not yet fully dropped subvolumes would mark its corresponding qgroups
stale.
This is incorrect. The real stale check should be based on the root
item, not root backref.
- Squota non-empty but stale qgroups
Such qgroups can not and should not be deleted, as future accounting
still require them.
- Full accounting mode, stale qgroups but not empty
Since qgroup numbers are inconsistent already, it's common to have
such stale qgroups with non-zero numbers.
Now it's dependent on the kernel to determine whether such qgroup can
be deleted.
Address the above problems:
- Do root_item based detection
So that btrfs_qgroup::stale would properly indicate if there is a
subvolume root item for the qgroup.
- Do not attempt to delete squota stale but non-empty qgroups
- Attempt to delete stale but non-empty qgroups for full accounting mode
And deletion failure would not count as an error.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo
David Sterba
Naohiro Aota
Aidan Gibson