The new convert treats the convert image as a normal file, without any
special flags and permissions.
This is different from original code:
1) Permission changed from 0400 to 0600
2) Inode lacks READONLY flag
This makes we can read-write mount the ext2 image and cause rollback
failure.
Follow old code behavior, use 0400 permission and add back READONLY
flag to fix it.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When copying inode, if there is a file referring part of a hole range,
convert will fail.
The problem is, when calculating real extent bytenr, it doesn't check if
the original extent is a hole.
In case the orinal extent is a hole, we still calculate bytenr using
file_pos - found_extent_file_pos, causing non-zero value, and later
btrfs_record_file_extent() detects that we are pointing to non-exist
extent and aborts convert.
Fix it by checking the disk_bytenr before calculating real disk bytenr.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
New convert doesn't insert holes for superblock migration range.
Unlike the old design, which only relocates 4K (superblock size) to
other places.
In the new design, to make sure convert can handle different page sizes
and align chunks bytenr, we relocate the whole 64K range.
And if there is only a 4K used block inside 64K superblock migration
range, it will make converted the fs have discontiguous file extents.
This patch will fix it by inserting needed holes to avoid such
discontinuous error.
Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
New convert has several bugs with backup superblock migration
1) Backup superblocks are not migrated due to incorrect condition
Two wrong checks cause backup superblocks not to be migrated at all
2) Converted ext* image doesn't keep hole for backup superblocks
Since we are creating file extents according to tmp_used, which has
wiped out backup superblock ranges.
In that case, later superblock migration will fail, since migration
will insert file extent range into ext* image.
Fix above bugs will make convert on ext2 image filled about 100M data
successful.
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When a ext2 fs filled with a 57M file, it's possible that convert fails
with assert in add_merge_cache_extent().
The problem is that the ext2 used space just takes some of the second
superblock.
And due to a bug in reserving superblock space, it corrupted used space
tree and cause assert.
Fix in by doing better used space merging for case where superblock
range is inside the ext2 used space.
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For new btrfs-convert, it's less restrict for metadata chunk allocation.
While the may_rollback() function is still following the restrict 1:1
mapping check for all chunks, it will not allow some new convert image
to be rolled back.
Add new per extent check for new convert to allow it to be rolled back.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Cleanup all the old btrfs-convert facilities, including:
1) btrfs_convert_operations->alloc/free/test_extents*
No need to do non-standard extent allocation.
After init_btrfs() everything can be done by normal routine.
Now only 4 functions are needed in btrfs_convert_operations.
1) open_fs
2) read_used_space
3) copy_inodes
4) close_fs
2) fs_info->extent_ops
Same as above.
3) Old init_btrfs(), create_image(), create_file_image_range()
Replaced with newer and cleaner one.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before this patch, btrfs-convert only rely on large enough initial
system/metadata chunk size to ensure no newer system/meta chunk will be
created.
But that's not safe enough. So add two new members in fs_info,
avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
chunks to be created before init_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add support to rollback new btrfs-convert.
The support is quite easy unlike the new convert behavior, which in fact
makes the rollback less restricted.
The core is, rollback must support both old and new convert result.
For old convert, all fs image extents can be covered by any chunk type.
But any extents inside reserved range must be covered by chunk.
For new convert, all fs image extents are covered by data chunk.
Allowing any condition will just make another fail to pass.
So make the convert condition a little less restricted to allow both can
be converted.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function do_convert_v2() to do new convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ use __strncpy_null for label ]
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new init_btrfs_v2() function for later newer do_convert().
Since we have good enough chunk allocation, a lot of wired chunk hack
won't ever be used.
We only need to insert data chunks and create needed subvolume.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance record_file_blocks() to handle reserved ranges.
Old file system can use the space in btrfs reserved ranges.
So we could not use the bytenr of old filesystem directly.
Thanks to previous patches, we have a full fs image in convert_root, and
it has already relocated the blocks in reserved ranges.
So here we just search the convert_root to get correct disk_bytenr and
use it.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, migrate_reserved_ranges() to migrate used fs
data in btrfs reserved space.
Unlike old implementation, which will need to relocate all the
complicated csum and reference relocation, previous patches already
ensure such reserved ranges won't be allocated. So here we only need
copy these data out and create new extent/csum/reference.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Use new function, create_convert_image_v2() to create snapshot of old
filesystem.
Unlike old function which is called after copying all inodes, this
function need to be called before copying inodes.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function, make_convert_data_chunks(), to build up data
chunks for convert.
It will call a modified version of btrfs_alloc_data_chunk() to force
data chunks to covert all known ext* data.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function make_convert_btrfs() for convert.
This new function will have the following features:
1) Allocate temporary sb/metadata/system chunk, avoiding old used data
2) More structured functions
No more over 1000 lines function, better function split and code
reuse
This will finally replace current make_btrfs(), but now only used for
convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new function, calculate_available_space() to get available
space cache_tree data_chunks cache_tree.
Unlike old implementation, this function will do the new work:
1) batch used ext* data space.
To ensure data chunks will recovery them all.
And restore the result into mkfs_cfg->convert_data_chunks for later
use.
2) avoid SB and reserved space at chunk level
Both batched data space or free space will not cover reserved space,
like SB or the first 1M.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before we do real convert, we need to read and build up used space cache
tree for later data/meta separate chunk layout.
This patch will iterate all used blocks in ext2 filesystem and record it
into cctx->used cache tree, for later use.
This provides the very basic of later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The new convert framework copies code from current dumpe2fs, which uses
BIGALLOC feature introduced in e2fsprogs v1.42.
While there are a lot of enterprise distributions which are still using
v1.41 e2fsprogs, this will cause compile error for them.
This patch introduces backward compatibility for new convert framework,
by manually introduce macros for ext2 BIGALLOC feature.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There's a mix of opencoded strncpy + null termination, strncpy, memcppy
without termination etc. Unify them and use the helper.
Resolves-coverity-id: 1357105
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce 3 new members for btrfs_convert_context:
1) struct cache_tree used
Records accurate byte ranges which are used by old filesystem.
This will be used to create old filesystem image.
2) struct cache_tree data_chunks
Records batched ranges which must be covered by data chunks.
The bytenr range is optimized to meet all the chunk requirement.
3) u64 total_bytenr
Records how large the filesystem is in bytenr.
Yes, we can calculate it easy, but that's for old blocks based
filesystem.
This will make it more friendly for extent based filesystem.
And later cctx->block_counts and may be removed
And 2 for mkfs_config:
1) char *chunk_uuid.
Used as temporary chunk_uuid (unparsed) string for later
make_convert_btrfs()
2) u64 super_bytenr
Records the new temporary super bytenr after make_btrfs().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Push out common convert operations into function pointers that we can
ultimately allow other file systems to use to provide their own
conversions.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Btrfs has changed to delete subvolume/snapshot asynchronously, which
means that after umount itself, if we've already deleted 'ext2_saved',
rollback can still be completed.
So this adds a check for ROOT_BACKREF before checking ROOT_ITEM since
ROOT_BACKREF is immediately not in the btree after
ioctl(BTRFS_IOC_SNAP_DESTROY) returns.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Reviewed-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ updated error messages ]
Signed-off-by: David Sterba <dsterba@suse.com>
When testing under libguestfs, btrfs-convert will never succeed to fix
chunk map, and always fails.
But in that case, it's already a mountable btrfs.
So better to info user with different error message for that case.
The root cause of it is still under investigation.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Check nodesize against features, not only sectorsize.
In fact, one of the btrfs-convert and mkfs differs in the nodesize
check.
This patch also provides the basis for later btrfs-convert fix.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We need to call ext2fs_close_inode_scan to release resources from
ext2fs_open_inode_scan.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The sequence, transid and reserved fields of inode were writen to disk
with uninitizlized value, this patch fixes it.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Coverity complains that the fslabel might be longer than the superblock
buffer, down in do_convert.
The label is at most 255 bytes, terminated by zero. Use buffers of the
right size.
Resolves-coverity-id: 1320911
Signed-off-by: David Sterba <dsterba@suse.com>
In btrfs-convert main(), strdup() allocates memory to fslabel but that
memory is not freed. We could fix it by adding free() calls to every
return point, but that would make the code messy because there are
several return paths.
So I fix it by changing the code using strdup() with local array and
strncpy().
And btrfs-convert main() guarantees that string length of fslabel is not
to exceed 'BTRFS_LABEL_SIZE', so it's enough to use strcpy() instead of
strncpy() to copy fslabel in do_convert().
Signed-off-by: Byongho Lee <bhlee.kernel@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
No need to close timer handle afain in subthread-close-callback,
it is closed by task_stop() automatically.
This patch also add a fflush() to make log output on time.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
As convert implement its own alloc extent, avoid such metadata problem
too.
Reported-by: Chris Murphy <lists@colorremedies.com>
Reported-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
block_iterate_proc uses the libext2fs error return codes when it doesn't
need them to return the error. We can push this out to __block_iterate_proc
and allow the reiserfs converter to share the code.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Adding a dirent consists of multiple steps that will need to be taken by
any converter.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
EXT2_FT_REG_FILE and BTRFS_FT_REG_FILE happen to share the same value, but
btrfs_insert_dir_item expects the btrfs version.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
In preparation to extend to other file systems, rename ext2_root to
image_root.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Hand off initialization of blk_iterate_data to init_blk_iterate_data and
pass the structure when the elements are the only values passed anyway.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Curretnly there are no features set for the converted filesystem and
it's not possible to request it like in mkfs.
Add new option -O|--features, use -O list-all to show all that are
supported for convert.
Note: from now on, convert without any -O option will use the same
defaults as mkfs. The original behaviour was rather confusing.
Convert is now more verbose about the parameters of the btrfs filesytem.
Signed-off-by: David Sterba <dsterba@suse.cz>
The file sys/acl.h is part of libacl and if the development package is
not installed, build of btrfs-convert fails.
We do not link against libacl nor use the functions provided by libacl. The
ACL_* values are directly read from the extN data, so it's more part of the
on-disk format rather than an interface to libacl.
The dependency on libacl is completely dropped.
Reported-by: Hugo Mills <hugo@carfax.org.uk>
Signed-off-by: David Sterba <dsterba@suse.cz>
We're not using it anywhere. The best practice is to add enums with
values > 255 for the long options, option index counting is error prone.
Signed-off-by: David Sterba <dsterba@suse.cz>
