Introduce new function, make_convert_data_chunks(), to build up data
chunks for convert.
It will call a modified version of btrfs_alloc_data_chunk() to force
data chunks to covert all known ext* data.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce new function make_convert_btrfs() for convert.
This new function will have the following features:
1) Allocate temporary sb/metadata/system chunk, avoiding old used data
2) More structured functions
No more over 1000 lines function, better function split and code
reuse
This will finally replace current make_btrfs(), but now only used for
convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new function, calculate_available_space() to get available
space cache_tree data_chunks cache_tree.
Unlike old implementation, this function will do the new work:
1) batch used ext* data space.
To ensure data chunks will recovery them all.
And restore the result into mkfs_cfg->convert_data_chunks for later
use.
2) avoid SB and reserved space at chunk level
Both batched data space or free space will not cover reserved space,
like SB or the first 1M.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Before we do real convert, we need to read and build up used space cache
tree for later data/meta separate chunk layout.
This patch will iterate all used blocks in ext2 filesystem and record it
into cctx->used cache tree, for later use.
This provides the very basic of later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The new convert framework copies code from current dumpe2fs, which uses
BIGALLOC feature introduced in e2fsprogs v1.42.
While there are a lot of enterprise distributions which are still using
v1.41 e2fsprogs, this will cause compile error for them.
This patch introduces backward compatibility for new convert framework,
by manually introduce macros for ext2 BIGALLOC feature.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There's a mix of opencoded strncpy + null termination, strncpy, memcppy
without termination etc. Unify them and use the helper.
Resolves-coverity-id: 1357105
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce 3 new members for btrfs_convert_context:
1) struct cache_tree used
Records accurate byte ranges which are used by old filesystem.
This will be used to create old filesystem image.
2) struct cache_tree data_chunks
Records batched ranges which must be covered by data chunks.
The bytenr range is optimized to meet all the chunk requirement.
3) u64 total_bytenr
Records how large the filesystem is in bytenr.
Yes, we can calculate it easy, but that's for old blocks based
filesystem.
This will make it more friendly for extent based filesystem.
And later cctx->block_counts and may be removed
And 2 for mkfs_config:
1) char *chunk_uuid.
Used as temporary chunk_uuid (unparsed) string for later
make_convert_btrfs()
2) u64 super_bytenr
Records the new temporary super bytenr after make_btrfs().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Push out common convert operations into function pointers that we can
ultimately allow other file systems to use to provide their own
conversions.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Btrfs has changed to delete subvolume/snapshot asynchronously, which
means that after umount itself, if we've already deleted 'ext2_saved',
rollback can still be completed.
So this adds a check for ROOT_BACKREF before checking ROOT_ITEM since
ROOT_BACKREF is immediately not in the btree after
ioctl(BTRFS_IOC_SNAP_DESTROY) returns.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Reviewed-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ updated error messages ]
Signed-off-by: David Sterba <dsterba@suse.com>
When testing under libguestfs, btrfs-convert will never succeed to fix
chunk map, and always fails.
But in that case, it's already a mountable btrfs.
So better to info user with different error message for that case.
The root cause of it is still under investigation.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Check nodesize against features, not only sectorsize.
In fact, one of the btrfs-convert and mkfs differs in the nodesize
check.
This patch also provides the basis for later btrfs-convert fix.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We need to call ext2fs_close_inode_scan to release resources from
ext2fs_open_inode_scan.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The sequence, transid and reserved fields of inode were writen to disk
with uninitizlized value, this patch fixes it.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Coverity complains that the fslabel might be longer than the superblock
buffer, down in do_convert.
The label is at most 255 bytes, terminated by zero. Use buffers of the
right size.
Resolves-coverity-id: 1320911
Signed-off-by: David Sterba <dsterba@suse.com>
In btrfs-convert main(), strdup() allocates memory to fslabel but that
memory is not freed. We could fix it by adding free() calls to every
return point, but that would make the code messy because there are
several return paths.
So I fix it by changing the code using strdup() with local array and
strncpy().
And btrfs-convert main() guarantees that string length of fslabel is not
to exceed 'BTRFS_LABEL_SIZE', so it's enough to use strcpy() instead of
strncpy() to copy fslabel in do_convert().
Signed-off-by: Byongho Lee <bhlee.kernel@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
No need to close timer handle afain in subthread-close-callback,
it is closed by task_stop() automatically.
This patch also add a fflush() to make log output on time.
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
As convert implement its own alloc extent, avoid such metadata problem
too.
Reported-by: Chris Murphy <lists@colorremedies.com>
Reported-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
block_iterate_proc uses the libext2fs error return codes when it doesn't
need them to return the error. We can push this out to __block_iterate_proc
and allow the reiserfs converter to share the code.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Adding a dirent consists of multiple steps that will need to be taken by
any converter.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
EXT2_FT_REG_FILE and BTRFS_FT_REG_FILE happen to share the same value, but
btrfs_insert_dir_item expects the btrfs version.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
In preparation to extend to other file systems, rename ext2_root to
image_root.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Hand off initialization of blk_iterate_data to init_blk_iterate_data and
pass the structure when the elements are the only values passed anyway.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Curretnly there are no features set for the converted filesystem and
it's not possible to request it like in mkfs.
Add new option -O|--features, use -O list-all to show all that are
supported for convert.
Note: from now on, convert without any -O option will use the same
defaults as mkfs. The original behaviour was rather confusing.
Convert is now more verbose about the parameters of the btrfs filesytem.
Signed-off-by: David Sterba <dsterba@suse.cz>
The file sys/acl.h is part of libacl and if the development package is
not installed, build of btrfs-convert fails.
We do not link against libacl nor use the functions provided by libacl. The
ACL_* values are directly read from the extN data, so it's more part of the
on-disk format rather than an interface to libacl.
The dependency on libacl is completely dropped.
Reported-by: Hugo Mills <hugo@carfax.org.uk>
Signed-off-by: David Sterba <dsterba@suse.cz>
We're not using it anywhere. The best practice is to add enums with
values > 255 for the long options, option index counting is error prone.
Signed-off-by: David Sterba <dsterba@suse.cz>
We've passed blocksize to prepare_system_chunk and used it to read and
write superblock. While this does not cause a bug (SUPER_INFO is
blocksize ie. page size on most arches), we should really use the
correct size.
Signed-off-by: David Sterba <dsterba@suse.cz>
Before this patch, ext*_image is always set NODATACSUM inode flag.
However btrfs-convert will set normal file with DATACUSM flag by
default, and generate checksum for regular file extent.
Now, a regular file extent is shared by a btrfs file inode with DATACSUM
and ext*_image with NODATACSUM, and it has checksum in csum tree.
This will cause btrfsck complain about odd checksum, since ext*_image is
set NODATACSUM but has checksum generated from regular file extent.
This patch makes convert completely obey datacsum setting, meaning
btrfs-convert will generate csum for every file extent by default.
Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Allow btrfs-convert to use nodesizes other than 4096. It defaults to
max(16384, pagesize), like mkfs.
Signed-off-by: Sebastian Thorarensen <sebth@naju.se>
Signed-off-by: David Sterba <dsterba@suse.cz>
With progress turned on by default we should be able to disable it
as well.
Reported-by: Jérôme Poulin <jeromepoulin@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Agreed by several people, showing progress by default makes sense as
conversion is a one-time and long running action.
Signed-off-by: David Sterba <dsterba@suse.cz>
glibc 2.10+ (5+ years old) enables all the desired features:
_XOPEN_SOURCE 700, __XOPEN2K8, POSIX_C_SOURCE, DEFAULT_SOURCE; with a
single _GNU_SOURCE define in the makefile alone. For portability to
other libc implementations (e.g. dietlibc) _XOPEN_SOURCE=700 is also
defined.
This also resolves Debian bug report filed by Michael Tautschnig -
"Inconsistent use of _XOPEN_SOURCE results in conflicting
declarations". Whilst I was not able to reproduce the results, the
reported fact is that _XOPEN_SOURCE set to 500 in one set of files
(e.g. cmds-filesystem.c) generates/defines different struct stat from
other files (cmds-replace.c).
This patch thus cleans up all feature defines, and sets them at a
consistent level.
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747969
Signed-off-by: Dimitri John Ledkov <dimitri.j.ledkov@intel.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
This fixes various compilation errors where PATH_MAX and XATTR_SIZE_MAX
were missing. To my knowledge, this should have no bad side effects.
Signed-off-by: Merlijn Wajer <merlijn@wizzup.org>
Signed-off-by: David Sterba <dsterba@suse.cz>
Run fstests: btrfs/012 will fail with message:
unable to do rollback
It is because the rollback function checks sequentially each piece of space
to map to a certain block group. If some piece doesn't, rollback refuses to continue.
After kernel commit:
commit 47ab2a6c689913db23ccae38349714edf8365e0a
Btrfs: remove empty block groups automatically
Empty block groups are removed, so there are possible gaps:
|--block group 1--| |--block group 2--|
^
|
gap
So the piece of space of the gap belongs to a removed empty block group,
and rollback should detect this case, and feel free to continue.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
When converting a sparse ext* filesystem, btrfs-convert adds checksum extents
for empty extents, whose disk_bytenr = 0, and this can end up with some weird
problems, one of them is the failure of reading free space cache inode on
mounting converted btrfs.
The fix is simple, just to skip making checksum on empty extents.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Bug-Debian: http://bugs.debian.org/539433
Bug-Debian: http://bugs.debian.org/583768
Authors:
Luca Bruno <lucab@debian.org>
Alexander Kurtz <kurtz.alex@googlemail.com>
Daniel Baumann <daniel.baumann@progress-technologies.net>
Signed-off-by: Dimitri John Ledkov <xnox@debian.org>
Signed-off-by: David Sterba <dsterba@suse.cz>
Fix (at least one user-visible) typos: it's its, not it's.
Signed-off-by: Holger Hoffstätte <holger.hoffstaette@googlemail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
The @force parameter for function @do_rollback is never checked
or used, remove it.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
The -d, -i, -n options make no sense to rollback.
Check the improper usages such as:
# btrfs-convert -r -d <dev>
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Since this patch:
btrfs-progs: move the check_argc_* functions into utils.c
All tools including the independent tools(e.g. btrfs-image, btrfs-convert)
can share the convenience of the check_argc_* functions, so this patch
adopt the argc check functions globally.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Allow the specification of the filesystem UUID at mkfs time.
Non-unique unique IDs are rejected. This includes attempting
to re-mkfs with the same UUID; if you really want to do that,
you can mkfs with a new UUID, then re-mkfs with the one you
wanted.
(Implemented only for mkfs.btrfs, not btrfs-convert).
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
[converted help to asciidoc]
Signed-off-by: David Sterba <dsterba@suse.cz>
For btrfs-convert, btrfstune, btrfs rescue, they report "device busy"
when given a device that does not actually exist e.g.
# btrfstune -x abcdefg (this device does not exist)
$ ...device busy...
We deal with this case by add "ret < 0" error check when
judging the return value of check_mounted.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
a clean up patch, the BTRFS_STRIPE_LEN is been duplicated across
btrfs-progs, the kernel defines it in volume.h so do the same
for progs.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
free(3) already checks the pointer for NULL, no need to do it
on your own. This patch make the change globally.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
Now we set @refs to 2 on creating a new extent buffer, meanwhile we
allocate the needed free space, but we don't give enough free_extent_buffer()
to reduce the eb's references to zero so that the eb can finally be freed,
so the problem is we has decrease the referene count of backrefs to zero, which
ends up releasing the space occupied by the eb, and this space can be allocated
again for something else(another eb or disk), usually a crash(core dump) will
occur, I've hit a crash in rb_insert() because another eb re-use the space while
the original one is floating around.
We should do the same thing as the kernel code does, it's necessary to initialize
@refs to 1 instead of 2, this helps us get rid of the above problem.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
So I needed to add a flag to not try to read block groups when doing
--init-extent-tree since we could hang there, but that meant adding a whole
other 0/1 type flag to open_ctree_fs_info. So instead I've converted it all
over to using a flags setting and added the flag that I needed. This has been
tested with xfstests and make test. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
mkfs -r wasn't creating chunks properly, making it very difficult to
allocate space for anything except tiny filesystems.
This changes it around to use more of the generic infrastructure, and
to do actual logical->physical block number translation.
It also allocates space to the files in smaller extents (max 1MB), which
keeps the allocator from trying to allocate an extent bigger than a
single chunk.
It doesn't quite support multi-device mkfs -r yet, but is much closer.
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In principle, link_subvol() can be given an abitrary string as the name
of the saved subvolume. It copies it into a fixed-size stack buffer and
then uses it as dirent names without testing its length.
This limits its length to BTRFS_NAME_LEN. This was found by static
analsys.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Before this change, passing -O skinny-metadata to mkfs.btrfs would
only set the skinny metadata incompat flag in the super block after
the filesystem was created. This change makes mkfs.btrfs directly
create a filesystem with only skinny extents for metadata.
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
sparse can freak out when <linux/fs.h> is included because it redefines
approximately a gazillion symbols already found in <sys/mount.h>:
/usr/include/linux/fs.h:203:9: warning: preprocessor token MS_RDONLY redefined
/usr/include/sys/mount.h:37:9: this was the original definition
Happily, we don't actually need to include the low-level <linux/fs.h>
for anything. One assumes it was just carried over from kernel space.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
__CHECKER__ is only for the type juggling used to tell sparse which
types need conversion between address spaces. It is not OK to use to
change the code that gets checked to avoid bugs elsewhere in the build
infrastructure. We want to check the code that builds when the checker
isn't enabled.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Mark many functions as static, and remove any resulting dead code.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Port of commit b3b4aa7 to userspace.
parameter tree root it's not used since commit
5f39d397dfbe140a14edecd4e73c34ce23c4f9ee ("Btrfs: Create extent_buffer
interface for large blocksizes")
This gets userspace a tad closer to kernelspace by removing
this unused parameter that was all over the codebase...
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Instead of aborting with a BUG_ON() statement, return a
negated errno code. Also updated mkfs and convert tools
to print a nicer error message when make_btrfs() returns
an error.
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Some codes still use the cpu_to_lexx instead of the
BTRFS_SETGET_STACK_FUNCS declared in ctree.h.
Also added some BTRFS_SETGET_STACK_FUNCS for btrfs_header and
btrfs_super.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
For any btrfs-$FOO executable, rename the main source file from
$FOO.c to to btrfs-$FOO.c
This makes it slightly more obvious what's building what,
and allows us to write a default rule in the Makefile for
these tools.
(also add btrfs-calc-size to the list of objects to remove
on make clean)
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
