The send.h is exported for libbtrfs and includes ctree.h but it lacks
the in-tree/out-of-tree ifdef switch. This would be a problem once
ctree.h moves.
Signed-off-by: David Sterba <dsterba@suse.com>
The test fails inside travis, check for a multiple-profile detection.
This was verified to work on a proper host so the workaround is in place
let the whole suite pass.
Signed-off-by: David Sterba <dsterba@suse.com>
The variable @root is only set but not utilized, while we only utilize
@root1.
Replace @root1 with @root, then remove the @root1.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Since it's inclusion in b3751c131a ("btrfs-progs: docs: update
btrfs-man5"), this option has never been available in kernel as
usebackuproot is a one-time mount option.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Now if an ENOSPC error happened, the free space report would help user
to determine if it's a real ENOSPC or a bug in convert.
The reported free space is the calculated free space, which doesn't
include super block space, nor merged data chunks.
The free space is always smaller than the reported available space of
the original fs, as we need extra padding space for used space to avoid
too fragmented data chunks.
The output would be:
$ ./btrfs-convert /dev/sda
create btrfs filesystem:
blocksize: 4096
nodesize: 16384
features: extref, skinny-metadata (default)
checksum: crc32c
free space report:
total: 10737418240
free: 0 (0.00%)
ERROR: unable to create initial ctree: No space left on device
WARNING: an error occurred during conversion, the original filesystem is not modified
Signed-off-by: Qu Wenruo <wqu@suse.com>
[ put total, free to separate lines ]
Signed-off-by: David Sterba <dsterba@suse.com>
The original fs is not touched until we migrate the super blocks.
Under most error cases, we fail before that thus the original fs is
still safe.
So change the error message according the stages we failed to reflect
that.
Signed-off-by: Qu Wenruo <wqu@suse.com>
[ adjust wording of messages ]
Signed-off-by: David Sterba <dsterba@suse.com>
This patch will enhance the error handling of ext2_copy_inodes by:
- Return more meaningful error number
Instead of -1 (-EPERM), now return -EIO for ext2 calls error, and
proper error number from btrfs calls.
- Commit transaction if ext2fs_open_inode_scan() failed
- Call ext2fs_close_inode_scan() on error
- Hunt down the BUG_ON()s
- Add error messages for transaction related calls
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add a helpful hint in the README to encourage contributors to install an
editorconfig plugin. This should help maintain source file consistency
in the long term (eg tabs instead of spaces).
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
[ update coding style references, reword ]
Signed-off-by: David Sterba <dsterba@suse.com>
Not all contributors work on projects that use linux kernel coding
style. This commit adds a basic editorconfig [0] to assist contributors
with managing configuration.
[0]: https://editorconfig.org/
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
Signed-off-by: David Sterba <dsterba@suse.com>
Calculate average fanout between levels:
Levels: 4
Total nodes: 289048
On level 0: 288054
On level 1: 989 (avg fanout 291)
On level 2: 4 (avg fanout 247)
On level 3: 1 (avg fanout 4)
Signed-off-by: David Sterba <dsterba@suse.com>
The new test case will test whether btrfs-convert can handle 64G ext*
fs.
This exercise the cctx->total_bytes calculation where multiplying 4K
(unsigned int) and 16777216 (u32) could lead to bit overflow for
unsigned int and got 0, and screw up later free space calculation.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit "btrfs-progs: convert: prevent 32bit overflow for
cctx->total_bytes" added an assert to ensure that cctxx.total_bytes did
not overflow, but this ASSERT calls assert_trace, which expects a long
value.
By converting the u64 to long overflows in a 32bit machine, leading the
assert_trace to be triggered since cctx.total_bytes turns to zero.
Fix this problem by comparing the cctx.total_bytes with zero when
calling ASSERT.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When convert is called on a 64GiB ext4 fs, it fails like this:
$ btrfs-convert /dev/loop0p1
create btrfs filesystem:
blocksize: 4096
nodesize: 16384
features: extref, skinny-metadata (default)
checksum: crc32c
creating ext2 image file
ERROR: missing data block for bytenr 1048576
ERROR: failed to create ext2_saved/image: -2
WARNING: an error occurred during conversion, filesystem is partially created but not finalized and not mountable
Btrfs-convert also corrupts the source fs:
$ LANG=C e2fsck /dev/loop0p1 -f
e2fsck 1.45.6 (20-Mar-2020)
Resize inode not valid. Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 3681 has zero dtime. Fix<y>? yes
Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes
Inode 3744 was part of the orphaned inode list. FIXED.
Deleted inode 3745 has zero dtime. Fix<y>? yes
Inode 3747 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear<y>? yes
...
[CAUSE]
After some debugging, the first strange behavior is, the value of
cctx->total_bytes is 0 in ext2_open_fs().
It turns out that, the value assign for cctx->total_bytes could lead to
bit overflow for the unsigned int value.
And that 0 cctx->total_bytes leads to various problems for later free
space calculation.
For example, in calculate_available_space(), we use cctx->total_bytes to
ensure we won't create a data chunk beyond device end:
cue_len = min(cctx->total_bytes - cur_off, cur_len);
If that cur_offset is also 0, we will create a cache_extent with 0 size,
which could cause a lot of problems for cache tree search.
[FIX]
Do manual casting for the multiply operation, so we could got a real u64
result. The fix will be applied to all supported fses (ext* and
reiserfs).
Reported-by: Christian Zangl <coralllama@gmail.com>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Kernel patch ff51bf02d107 ("btrfs: block-group: fix free-space bitmap
threshold") is needed to exercise both branches of the test, this can be
detected by lack of the objectid and offset parsed from the dump.
Issue: #288
Signed-off-by: David Sterba <dsterba@suse.com>
For workarounds or known missing support add helper to print a
notification about that, when _not_run is not suitable.
Signed-off-by: David Sterba <dsterba@suse.com>
- add missing raid1c34 profiles to the list of supported profiles
- document defaults change in 5.8
- update wording
Signed-off-by: David Sterba <dsterba@suse.com>
The single profile is better suited as default for data on multiple
devices. Switch from RAID0 because:
- it's easier to convert to other profiles, as single consumes some
chunks per device, but RAID0 has chunks on all devices regardless of
the used space
- RAID0 has no redundancy and compared one disk failure affects many
files due to striping, while with single the chances are a bit higher
that complete files are stored on one device
- when the device sizes are not equal and not even close to equal, the
maximum achievable size with RAID0 is size of the smallest device due
to striping, with single it's the sum of all device sizes
The changed defaults could affect scripts and deployments that rely on
the old values, but given the number of possible profiles for multiple
devices let's hope that they're specified explicitly in majority of
cases.
Issue: #270
Signed-off-by: David Sterba <dsterba@suse.com>
Extract the defaults for data and metadata profiles to a header and
use the symbolic names instead of hardcoding the profiles.
Signed-off-by: David Sterba <dsterba@suse.com>
The node/leaf stats have been calculated but never displayed. Moreover,
a more detailed information about counts on each level can be useful,
add it to the output of tree-stats.
Example output:
Levels: 3
Total nodes: 25692
On level 0: 25601
On level 1: 90
On level 2: 1
Issue: #266
Signed-off-by: David Sterba <dsterba@suse.com>
The options have been deprecated for a long time, the kernel mount
options are being removed too (in 5.9).
Signed-off-by: David Sterba <dsterba@suse.com>
The option -A was used long time ago for debugging and marked as
obsolete since 4.14.1. Remove the option and set the alloc start to the
default value 1MiB.
Signed-off-by: David Sterba <dsterba@suse.com>
There's still some interest in the btrfs-debugfs tool, make it work with
python v3 until we have a replacement.
Issue: #261
Signed-off-by: Lakshmipathi <lakshmipathi.ganapathi@collabora.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Many subcommands have their own verbosity options that are being
superseded by the global options. Update the help text to reflect that
where applicable.
Signed-off-by: David Sterba <dsterba@suse.com>
Add a test case to check if the converted fs has device extent beyond
boundary.
The disk layout of source ext4 fs needs some extents to make them
allocated at the very end of the fs. The script is from the original
reporter.
Also, since the existing convert tests always uses 512M as device size,
which is not suitable for this test case, make it to grab the existing
device size to co-operate with this test case.
Reported-by: Jiachen YANG <farseerfc@gmail.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
The following script could lead to corrupted btrfs fs after
btrfs-convert:
fallocate -l 1G test.img
mkfs.ext4 test.img
mount test.img $mnt
fallocate -l 200m $mnt/file1
fallocate -l 200m $mnt/file2
fallocate -l 200m $mnt/file3
fallocate -l 200m $mnt/file4
fallocate -l 205m $mnt/file1
fallocate -l 205m $mnt/file2
fallocate -l 205m $mnt/file3
fallocate -l 205m $mnt/file4
umount $mnt
btrfs-convert test.img
The result btrfs will have a device extent beyond its boundary:
pening filesystem to check...
Checking filesystem on test.img
UUID: bbcd7399-fd5b-41a7-81ae-d48bc6935e43
[1/7] checking root items
[2/7] checking extents
ERROR: dev extent devid 1 physical offset 993198080 len 85786624 is beyond device boundary 1073741824
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 913960960 bytes used, error(s) found
total csum bytes: 891500
total tree bytes: 1064960
total fs tree bytes: 49152
total extent tree bytes: 16384
btree space waste bytes: 144885
file data blocks allocated: 2129063936
referenced 1772728320
[CAUSE]
Btrfs-convert first collect all used blocks in the original fs, then
slightly enlarge the used blocks range as new btrfs data chunks.
However the enlarge part has a problem, that it doesn't take the device
boundary into consideration.
Thus it caused device extents and data chunks to go beyond device
boundary.
[FIX]
Just to extra check before inserting data chunks into
btrfs_convert_context::data_chunk.
Reported-by: Jiachen YANG <farseerfc@gmail.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[WARNING]
When compiling btrfs-progs, the following warning pops up:
In file included from /usr/include/stdio.h:867,
from ./kerncompat.h:22,
from common/fsfeatures.c:17:
In function 'printf',
inlined from 'process_features' at common/fsfeatures.c:192:4,
inlined from 'btrfs_process_runtime_features' at common/fsfeatures.c:205:2:
/usr/include/bits/stdio2.h:107:10: warning: '%s' directive argument is null [-Wformat-overflow=]
107 | return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This only occur with default make parameters. If compiling with D=1, the
warning just disappears.
The involved tool chain is:
- GCC 10.1.0
[CAUSE]
The offending code is:
static void process_features(u64 flags, enum feature_source source)
{
...
if (flags & feat->flag) {
printf("Turning ON incompat feature '%s': %s\n",
feat->name, feat->desc);
}
...
}
Currently, there is no runtime/fs feature without a name nor
description. So we shouldn't hit a feature with NULL as name nor
description.
This looks like a bug in GCC though.
[WORKAROUND]
However can workaround it by doing an explicit check on feat->name and
feat->desc to teach GCC not to do a wrong warning.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[WARNING]
When compiling btrfs-progs, there is one warning from convert ext2 code:
convert/source-ext2.c: In function 'ext2_open_fs':
convert/source-ext2.c:91:44: warning: pointer targets in passing argument 1 of 'strndup' differ in signedness [-Wpointer-sign]
91 | cctx->volume_name = strndup(ext2_fs->super->s_volume_name, 16);
| ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
| |
| __u8 * {aka unsigned char *}
In file included from ./kerncompat.h:25,
from convert/source-ext2.c:19:
/usr/include/string.h:175:35: note: expected 'const char *' but argument is of type '__u8 *' {aka 'unsigned char *'}
175 | extern char *strndup (const char *__string, size_t __n)
| ~~~~~~~~~~~~^~~~~~~~
The toolchain involved is:
- GCC 10.1.0
- e2fsprogs 1.45.6
[CAUSE]
Obviously, in the offending e2fsprogs, the volume label is using u8,
which is unsigned char, not char.
/*078*/ __u8 s_volume_name[EXT2_LABEL_LEN]; /* volume name, no NUL? */
[FIX]
Just do a forced conversion to suppress the warning is enough.
I don't think we need to apply -Wnopointer-sign yet.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
