When a ext2 fs filled with a 57M file, it's possible that convert fails
with assert in add_merge_cache_extent().
The problem is that the ext2 used space just takes some of the second
superblock.
And due to a bug in reserving superblock space, it corrupted used space
tree and cause assert.
Fix in by doing better used space merging for case where superblock
range is inside the ext2 used space.
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For new btrfs-convert, it's less restrict for metadata chunk allocation.
While the may_rollback() function is still following the restrict 1:1
mapping check for all chunks, it will not allow some new convert image
to be rolled back.
Add new per extent check for new convert to allow it to be rolled back.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Cleanup all the old btrfs-convert facilities, including:
1) btrfs_convert_operations->alloc/free/test_extents*
No need to do non-standard extent allocation.
After init_btrfs() everything can be done by normal routine.
Now only 4 functions are needed in btrfs_convert_operations.
1) open_fs
2) read_used_space
3) copy_inodes
4) close_fs
2) fs_info->extent_ops
Same as above.
3) Old init_btrfs(), create_image(), create_file_image_range()
Replaced with newer and cleaner one.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before this patch, btrfs-convert only rely on large enough initial
system/metadata chunk size to ensure no newer system/meta chunk will be
created.
But that's not safe enough. So add two new members in fs_info,
avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
chunks to be created before init_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add support to rollback new btrfs-convert.
The support is quite easy unlike the new convert behavior, which in fact
makes the rollback less restricted.
The core is, rollback must support both old and new convert result.
For old convert, all fs image extents can be covered by any chunk type.
But any extents inside reserved range must be covered by chunk.
For new convert, all fs image extents are covered by data chunk.
Allowing any condition will just make another fail to pass.
So make the convert condition a little less restricted to allow both can
be converted.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function do_convert_v2() to do new convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ use __strncpy_null for label ]
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new init_btrfs_v2() function for later newer do_convert().
Since we have good enough chunk allocation, a lot of wired chunk hack
won't ever be used.
We only need to insert data chunks and create needed subvolume.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance record_file_blocks() to handle reserved ranges.
Old file system can use the space in btrfs reserved ranges.
So we could not use the bytenr of old filesystem directly.
Thanks to previous patches, we have a full fs image in convert_root, and
it has already relocated the blocks in reserved ranges.
So here we just search the convert_root to get correct disk_bytenr and
use it.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, migrate_reserved_ranges() to migrate used fs
data in btrfs reserved space.
Unlike old implementation, which will need to relocate all the
complicated csum and reference relocation, previous patches already
ensure such reserved ranges won't be allocated. So here we only need
copy these data out and create new extent/csum/reference.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Use new function, create_convert_image_v2() to create snapshot of old
filesystem.
Unlike old function which is called after copying all inodes, this
function need to be called before copying inodes.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Btrfs_record_file_extent() has some small problems like:
1) Can't handle overlapping extents
2) May create extent larger than BTRFS_MAX_EXTENT_SIZE
So enhance it using previously added facilites.
This is used for later btrfs-convert, as for new convert, we create
saved image first, then copy inode.
Which will also cause extent overlapping.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new function, btrfs_search_overlap_extent() to find the first
overlapping extent.
It's useful for later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, make_convert_data_chunks(), to build up data
chunks for convert.
It will call a modified version of btrfs_alloc_data_chunk() to force
data chunks to covert all known ext* data.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, setup_temporary_extent_tree() to build a
temporary extent tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, setup_temp_csum_tree(), to setup temporary
csum tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introudce new function, setup_temp_fs_tree(), to setup temporary fs tree
for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, setup_temp_device_tree(), to setup temporary
device tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, setup_temp_chunk_root(), to initialize temporary
chunk root for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, setup_temp_tree_root(), to initialize temporary
tree root for make_btrfs_v2().
The new function will setup tree root at metadata chunk and ensure data
won't be written into metadata chunk.
Also, new make_btrfs_v2() will have a much better code structure than
old make_btrfs().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new function, setup_temp_super(), to setup temporary super
for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ use __strncpy_null in setup_temp_super ]
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function make_convert_btrfs() for convert.
This new function will have the following features:
1) Allocate temporary sb/metadata/system chunk, avoiding old used data
2) More structured functions
No more over 1000 lines function, better function split and code
reuse
This will finally replace current make_btrfs(), but now only used for
convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new function, calculate_available_space() to get available
space cache_tree data_chunks cache_tree.
Unlike old implementation, this function will do the new work:
1) batch used ext* data space.
To ensure data chunks will recovery them all.
And restore the result into mkfs_cfg->convert_data_chunks for later
use.
2) avoid SB and reserved space at chunk level
Both batched data space or free space will not cover reserved space,
like SB or the first 1M.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before we do real convert, we need to read and build up used space cache
tree for later data/meta separate chunk layout.
This patch will iterate all used blocks in ext2 filesystem and record it
into cctx->used cache tree, for later use.
This provides the very basic of later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The new convert framework copies code from current dumpe2fs, which uses
BIGALLOC feature introduced in e2fsprogs v1.42.
While there are a lot of enterprise distributions which are still using
v1.41 e2fsprogs, this will cause compile error for them.
This patch introduces backward compatibility for new convert framework,
by manually introduce macros for ext2 BIGALLOC feature.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We really use root only for mount/umount and access to the ext2_saved
image (that has 0600). Also switch to common variable so we can use
helpers.
Signed-off-by: David Sterba <dsterba@suse.com>
New convert introduced simpler chunk/extent allocation algorithm, at the
cost of complex backup superblock migration codes.
Use specially built ext2 images to test if btrfs-convert can convert and
rollback images without problem.
All these special ext2 image have blocks/holes across 2nd btrfs backup
superblock.
The naming of test image is like the following:
|<------superblock migration range----->|
64M 64M + 64K
|-Data--|-Data--|/Hole//|-Data--|/Hole//|-Data--|--Data--| = drdhdhdrd
These test cases should check all typical layouts and make sure new
convert works.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add support for custom convert test scripts, just like fsck tests.
Instead of generic convert tests, we need more specifically created images
for new convert tests.
This patch provide the needed infrastructure for later convert test
cases.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
- If -c is set, filename argument is ignored.
- Describe about -h option
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Usage is only printed if -h option is set. However it's nice to
do it when wrong option is set or the number of argument is wrong.
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Remove the following build error.
====================================
$ make btrfs-crc
[CC] btrfs-crc.o
[LD] btrfs-crc
btrfs-crc.o: In function `usage':
/home/sat/src/btrfs-progs/btrfs-crc.c:26: multiple definition of `usage'
help.o:/home/sat/src/btrfs-progs/help.c:125: first defined here
collect2: error: ld returned 1 exit status
Makefile:294: recipe for target 'btrfs-crc' failed
make: *** [btrfs-crc] Error 1
=====================================
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We know nodesize should not overflow with the shift, but le'ts make the
code correct if the resulting type can store the full value.
Resolves-coverity-id: 1358120
Signed-off-by: David Sterba <dsterba@suse.com>
Replace old and not so informal srand()/rand() calls to new random
number API.
Including btrfs-corrupt-block(main user), btrfs-image and btrfs-crc.
Some tests like dir-test/random-test/quick-test is not modified.
As random-test itself can't even pass build.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David has reported some quite chaos usage of pseudo random numbers.
Like using static srand seed, or even calling rand() without setting
seed correctly.
The new pseudo random API will initialize the random seed on its first
calling and use uniformly distributed pseudo random number generator as
backend.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ renamed variables and functions, added prefixes ]
Signed-off-by: David Sterba <dsterba@suse.com>
For skinny metadata case, btrfs-corrupt-block can't corrupt a metadata
extent due to the type check missing BTRFS_METADATA_ITEM_KEY.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
New convert framework uses new and simpler chunk layout, while the cost
is the more complex superblock range migration logical, compared to old
convert.
Enhance the convert test script to create file which will takes up 2nd
backup superblock space, to ensure the superblock migration is working
as expected.
Suggested-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Previously btrfs-image restore would set the chunk items to have 1 stripe,
even if the chunk is dup. If you use btrfsck on the restored file system,
some dev_extent will not find any relative chunk stripe, and the
bytes-used of dev_item will not equal to the dev_extents's total_bytes.
This patch store a additional physical just for the dup case when build
the in-memory chunk-tree.
Currently btrfsck on the restored file system, only single and dup is no
problem. raid* support should be added in the future.
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
