Add make command line variable TEST_FROM that takes a glob from where to
start the test sequence. Update docs and fix some trivial typos.
Signed-off-by: David Sterba <dsterba@suse.com>
The build is fine on BE but endianness was broken which prevents mkfs
and anything else on existing images.
Signed-off-by: David Sterba <dsterba@suse.com>
Run release tests after pushing master branch to verify that everything
is ok, also trigger the static build so there are binaries built with
the new version tag.
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Command `btrfs scrub start -B` and `btrfs scrub status` are reporting
very different results for "Total to scrub":
$ sudo btrfs scrub start -B /mnt/btrfs/
scrub done for c107ef62-0a5d-4fd7-a119-b88f38b8e084
Scrub started: Mon Jun 5 07:54:07 2023
Status: finished
Duration: 0:00:00
Total to scrub: 1.52GiB
Rate: 0.00B/s
Error summary: no errors found
$ sudo btrfs scrub status /mnt/btrfs/
UUID: c107ef62-0a5d-4fd7-a119-b88f38b8e084
Scrub started: Mon Jun 5 07:54:07 2023
Status: finished
Duration: 0:00:00
Total to scrub: 12.00MiB
Rate: 0.00B/s
Error summary: no errors found
This can be very confusing for end users.
[CAUSE]
It's the function print_fs_stat() handling the "Total to scrub" output.
For `btrfs scrub start` command, we use the used bytes (aka, the total
used dev extents of a device) for output.
This is not really accurate, as the chunks may be mostly empty just like
the following:
$ btrfs fi df /mnt/btrfs/
Data, single: total=1.01GiB, used=9.06MiB
System, DUP: total=40.00MiB, used=64.00KiB
Metadata, DUP: total=256.00MiB, used=1.38MiB
GlobalReserve, single: total=22.00MiB, used=0.00B
Thus we're reporting 1.5GiB to scrub (1.01GiB + 40MiB * 2 + 256MiB * 2).
But in reality, we only scrubbed 12MiB
(9.06MiB + 64KiB * 2 + 1.38MiB * 2).
[FIX]
Instead of using the used dev-extent bytes of a device, go with proper
scrubbed bytes for each device.
This involves print_fs_stat() and print_scrub_dev() called inside
scrub_start().
Now the output should match each other.
Issue: #636
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report that on s390x big endian systems, mkfs.btrfs just
fails:
$ mkfs.btrfs -f ~/test.img
btrfs-progs v6.3.1
Invalid mapping for 1081344-1097728, got 17592186044416-17592190238720
Couldn't map the block 1081344
ERROR: cannot read chunk root
ERROR: open ctree failed
[CAUSE]
The error is caused by wrong endian conversion. The s390x is a big
endian architecture:
$ lscpu
Byte Order: Big Endian
While checking the offending @disk_key and @key inside
btrfs_read_sys_array(), we got:
2301 while (cur_offset < array_size) {
(gdb)
2304 if (cur_offset + len > array_size)
(gdb)
2307 btrfs_disk_key_to_cpu(&key, disk_key);
(gdb)
2310 sb_array_offset += len;
(gdb) print *disk_key
$2 = {objectid = 281474976710656, type = 228 '\344', offset = 17592186044416}
(gdb) print key
$3 = {objectid = 281474976710656, type = 228 '\344', offset = 17592186044416}
(gdb)
Now we can see, @disk_key is indeed in the little endian, but @key is
not converted to the CPU native endian.
Furthermore, if we step into the help btrfs_disk_key_to_cpu(), it shows
we're using little endian version:
(gdb) step
btrfs_disk_key_to_cpu (disk_key=0x109fcdb, cpu_key=0x3ffffff847f)
at ./kernel-shared/accessors.h:592
592 memcpy(cpu_key, disk_key, sizeof(struct btrfs_key));
[FIX]
The kernel accessors.h checks if __LITTLE_ENDIAN is defined or not, but
that only works inside kernel.
In user space, __LITTLE_ENDIAN and __BIG_ENDIAN are both provided by
kerncompat.h that should have been included already.
Instead we should check __BYTE_ORDER against __LITTLE_ENDIAN to
determine our endianness.
With this change, s390x build works as expected now.
Issue: #639
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The remount workflow could cause some problems so make a note about it.
Recommend the umount/mount step by default. A seeding device used for
e.g. a root filesystem that gets updated and has snapshots is a real
world example where the space consumed by unreclaimed deleted snapshots
would hurt.
Pull-request: #462
Author: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
mkfs.btrfs help message for --quiet is 'no message except errors' so
we probably ought to silence this as well in the quiet case.
Author: Dominique Martinet <dominique.martinet@atmark-techno.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Other pages list the latest version first, do that here as well. Also
reorder index so the features are first before version changelogs.
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
We got some test failures related to btrfs-convert with subpage, e.g.
btrfs/012, the failure would cause the following dmesg:
BTRFS warning (device nvme0n1p7): v1 space cache is not supported for page size 16384 with sectorsize 4096
BTRFS error (device nvme0n1p7): open_ctree failed
[CAUSE]
v1 space cache has tons of hard coded PAGE_SIZE usage, and considering
v2 space cache is going to replace it (which is already the new default
since v5.15 btrfs-progs), thus for btrfs subpage support, we just simply
reject the v1 space cache, and utilize v2 space cache when possible.
But there is special catch in btrfs-convert, although we're specifying
v2 space cache as the new default for btrfs-convert, it doesn't really
follow the specification at all.
Thus the converted filesystem will still go v1 space cache.
[FIX]
It can be a huge change to btrfs-convert to make the initial btrfs image
to support v2 cache.
Thus this patch would change the fs at the final stage, just before we
finalize the btrfs.
This patch would drop all the v1 cache created, then call
btrfs_create_free_space_tree() to populate the free space tree and
commit the superblock with needed compat_ro flags.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add the DEF_SUPER_FLAG_ENTRY for CHANGING_FSID_V2 to our btrfs-progs'
print-tree.c, as it is currently missing in the dump-super output, which
was too confusing.
Before:
flags 0x1000000001
( WRITTEN )
After:
flags 0x1000000001
( WRITTEN |
CHANGING_FSID_V2 )
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Print the source directory for --rootdir and if --shrink is used. With
-vv then print the individual files as added:
$ mkfs.btrfs --rootdir dir --shrink -vv img
...
Rootdir from: Documentation
ADD: /btrfs-progs/Documentation/btrfs-check.rst
...
ADD: /btrfs-progs/Documentation/btrfs-send.rst
Shrink: yes
Label: (null)
UUID: 40d3a16f-02d8-40d7-824b-239cee528093
...
The 'Rootdir from' is printed before the files are added so there's now
message before the files are added which could take some time.
Issue: #627
Signed-off-by: David Sterba <dsterba@suse.com>
Move files from m4/ to config that is also used for build and we can
reduce the number of toplevel directories.
Signed-off-by: David Sterba <dsterba@suse.com>
If I run autoreconf -fi in btrfs-progs repo in VirtualBox, it fails with
possibly undefined macro AC_DEFINE. To fix this, I added m4 as the macro
directory in configure.ac as follows. Adding that line fixes the error
when running autoreconf.
Versions:
autoreconf 2.69
automake 1.15.1
btrfs-progs v6.3-fba31d63
openSUSE 15.5
VirtualBox 6.1.40
configure.ac:7: error: possibly undefined macro: AC_DEFINE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:63: error: possibly undefined macro: AC_MSG_ERROR
configure.ac:276: error: possibly undefined macro: AC_MSG_WARN
autoreconf: /usr/bin/autoconf failed with exit status: 1
Issue: #632
Signed-off-by: David Sterba <dsterba@suse.com>
More complete test coverage:
- json an string table formatters
- fuzz tests (no mount)
- libbtrfs build test
- libbtrfsutil python test
- ioctl build test
- hash tests
Signed-off-by: David Sterba <dsterba@suse.com>
The links to repositories and contributors is hard to find as it's below
the feature list. Move it up as it is for overview.
Signed-off-by: David Sterba <dsterba@suse.com>
There are reports that json output of 'qgroup show' crashes due to
internal error when printing the limit values:
INTERNAL ERROR: unknown unit base, mode 2304
btrfs(internal_error+0x10a)[0x5605c37ce48a]
btrfs(pretty_size_snprintf+0x5c)[0x5605c37d105c]
btrfs(fmt_print+0x44e)[0x5605c37d178e]
btrfs(+0x7ed1d)[0x5605c3800d1d]
btrfs(main+0x8f)[0x5605c379beff]
/lib64/libc.so.6(+0x27bb0)[0x7f83924ddbb0]
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f83924ddc79]
btrfs(_start+0x25)[0x5605c379d405]
common/units.c:82: pretty_size_snprintf: Assertion `0` failed, value 0
btrfs(+0x1d4b1)[0x5605c379f4b1]
btrfs(pretty_size_snprintf+0x7b)[0x5605c37d107b]
btrfs(fmt_print+0x44e)[0x5605c37d178e]
btrfs(+0x7ed1d)[0x5605c3800d1d]
btrfs(main+0x8f)[0x5605c379beff]
/lib64/libc.so.6(+0x27bb0)[0x7f83924ddbb0]
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f83924ddc79]
btrfs(_start+0x25)[0x5605c379d405]
This is caused by "size" format that requires the unit mode, but it was not
specified and some stack value used. As json prints the raw values, use
the plain %llu format.
Link: https://bugzilla.opensuse.org/show_bug.cgi?id=1206960
Link: https://bugzilla.opensuse.org/show_bug.cgi?id=1209136#c15
Signed-off-by: David Sterba <dsterba@suse.com>
Currently "btrfstune --csum" allows us to change the csum to the same
one, this is good for testing but not good for end users, as if the end
user interrupts it, they have to resume the change (even it's to the
same csum type) until it finished, or kernel would reject such fs.
Furthermore, we never change the super block csum type until we
completely changed the csum type, thus for resume cases, the fs would
still show up as using the old csum type thus won't cause any problem
resuming.
So here we just reject the csum conversion if the target csum type is
the same as the existing csum type.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When the csum conversion is interrupted when changing data csum
objectid, we should just resume the objectid conversion.
This situation can be detected by comparing the old and new csum items.
They should both exist but doesn't intersect (interrupted halfway), or
only new csum items exist (interrupted after we have deleted old csums).
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
If the csum conversion is interrupted when old csums are being deleted,
we should resume by continue deleting the old csums.
The function delete_old_data_csums() can handle half deleted cases
already.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We have a very rare chance to hit a fs with empty csum tree but still
has CHANGING_DATA_CSUM flag.
The window is very small, but it's still possible, so handle it by
jumping directly to metadata csum change.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There are two possible situations where there is no new csum items at
all:
- We just inserted csum change item
This means all csums are really old csum type, and we can start
the conversion from the beginning, only need to skip the csum change
item insert.
- We finished data csum conversion but not yet started metadata
conversion
This means all csums are already new csum type, and we can resume
by starting changing metadata csums.
To distinguish the two cases, we need to read the first sector, and
verify the data content against both csum types.
If the csum matches with old csum type, we resume from generating new
data csum.
If the csum matches with new csum type, we resume from rewriting
metadata csum.
If the csum doesn't match either csum type, we have some big problems
then.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
If a csum change is interrupted at new data csum generation stage, we
can detect such situation by checking the old and new csum items.
At the new data csum generation stage, old csums are untouched, and only
new csums items (with different objectid) are inserted into the csum
tree.
Thus the old csum items should cover a larger range, while the new csum
items should be a subset of the old csums.
The resume part would start by re-generating the remaining part, then go
through the conversion stages.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For interrupted metadata checksum change, we only need to call the same
change_meta_csums().
Since we don't have any record on the last converted metadata, thus we
have to go through all metadata anyway.
And the existing change_meta_csums() has already implemented the needed
checks to skip converted metadata.
Since we're here, also implement all the surrounding checks, like making
sure the new target csum type matches the interrupted one.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
Doing the following csum change in a row, it would fail:
# mkfs.btrfs -f --csum crc32c $dev
# btrfstune --csum sha256 $dev
# btrfstune --csum crc32c $dev
# btrfstune --csum sha256 $dev
WARNING: Experimental build with unstable or unfinished features
WARNING: Switching checksums is experimental, do not use for valuable data!
Proceed to switch checksums
ERROR: failed to insert csum change item: File exists
ERROR: failed to generate new data csums: File exists
WARNING: reserved space leaked, flag=0x4 bytes_reserved=16384
extent buffer leak: start 30572544 len 16384
extent buffer leak: start 30441472 len 16384
WARNING: dirty eb leak (aborted trans): start 30441472 len 16384
[CAUSE]
During every csum change operation, btrfstune would insert an temporaray
csum change item into root tree.
But unfortunately after the conversion btrfstune doesn't properly delete
the csum change item, result the following items in the root tree:
item 10 key (CSUM_CHANGE TEMPORARY_ITEM 0) itemoff 13423 itemsize 0
temporary item objectid CSUM_CHANGE offset 0
target csum type crc32c (0)
item 11 key (CSUM_CHANGE TEMPORARY_ITEM 2) itemoff 13423 itemsize 0
temporary item objectid CSUM_CHANGE offset 2
target csum type sha256 (2)
Thus at the last conversion try to go back to SHA256, we failed to
insert the same item, and caused the above error.
[FIX]
After finishing the metadata csum conversion, do a proper removal of the
csum item.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Since commit c8593f65cbf3 ("btrfs-progs: sync tree-checker.[ch] from
kernel"), btrfs-progs can do the kernel level tree block checks, which
is not really sutiable for dump-tree.
Under a lot of cases, we're using dump-tree tool to debug to collect the
details from end users.
If it's a bitflip causing a rejection, we would be unable to determine
the cause.
So this patch would add OPEN_CTREE_SKIP_LEAF_ITEM_CHECKS for dump-tree.
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The csum change for metadata is like uuid-change, we go with in-place
csum update without any COW.
During the rewrite, we will manually check the csum (both old and new)
for each tree block.
And only rewrite the csum if the tree block matches its old csum.
(For tree block matches its new csum, we need to do nothing).
And when everything is done, just update the superblock to reflect the
csum type change.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
At this stage, the csum tree should only contain the temporary csum
items (CSUM_CHANGE, EXTENT_CSUM, logical), and no more old csum items.
Now we can convert those temporary csum items back to regular csum items
by changing their key objectids back to EXTENT_CSUM.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The new helper function, delete_old_data_csums(), would delete the old
data csums while keep the new one untouched.
Since the new data csums have a key objectid (-13) smaller than the
old data csums (-10), we can safely delete from the tail of the btree.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch would modify btrfs_csum_file_block() to handle csum type
other than the one used in the current fs.
The new data checksum would use a different objectid (-13) to
distinguish with the existing one (-10).
This needs to change tree-checker to skip the item size checks,
since new csum can be larger than the original csum.
After this stage, the resulted csum tree would look like this:
item 0 key (CSUM_CHANGE EXTENT_CSUM 13631488) itemoff 8091 itemsize 8192
range start 13631488 end 22020096 length 8388608
item 1 key (EXTENT_CSUM EXTENT_CSUM 13631488) itemoff 7067 itemsize 1024
range start 13631488 end 14680064 length 1048576
Note the itemsize is 8 times the original one, as the original csum is
CRC32, while target csum is SHA256, which is 8 times the size.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch introduces a new helper function,
read_verify_one_data_sector(), to do the data read and checksum
verification (against the old csum).
This data would be later re-used to generate a new csum.
And since we're introduce the helper function, we also build the
skeleton to iterate the data extents using the old csum tree.
This method is much better compared to iterating using extent tree,
which has no directly indicator on whether the data extent has csum or
not (nodatasum or preallocated).
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The overall idea is to make sure no running operations (balance,
dev-replace, dirty log) for the fs before csum change.
And also reject half converted csums for now.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The existing attempt for changing csum types is as the following:
- Create a new temporary csum root
- Generate new data csums into the temporary csum root
- Drop the old csum tree and make the temporary one as csum root
- Change the checksums for metadata in-place
Unfortunately after some experiments, the csum root switch method has a
big pitfall, the backref items in extent tree.
Those backref items still point back to the old tree, meaning without a
lot of extra tricks, the extent tree would be corrupted.
Thus we have to go a new single tree variant:
- Generate new data csums into the csum root
The new data csums would have a different objectid to distinguish
them.
- Drop the old data csum items
- Change the key objectids of the new csums
- Change the checksums for metadata in-place
This means unfortunately we have to revert most of the old code, and
update the temporary item format.
The new temporary item would only record the target csum type.
At every stage we have a method to determine the progress, thus no need
for an item, but in the future it's still open for change.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add structures and prototypes. Leave out undefined types (time64_t,
block_group_rsv) for now. Kernel 6.4-rc1.
Signed-off-by: David Sterba <dsterba@suse.com>
The send.h for libbtrfs has been separated some time ago so we're now
free to keep up with kernel, 6.4-rc1.
Signed-off-by: David Sterba <dsterba@suse.com>
