When btrfs-image run on a mounted filesystem,
the undergoing fs operations may change what you have imaged a while ago.
In this case, give a warning to remind the user that he may not
get a consistent image he wants.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
[tweaked the message]
Signed-off-by: David Sterba <dsterba@suse.cz>
The btrfs-image requires at least 2 args to run,
one for the source dev/file, the other for the target dev/file.
This patch depends on patch:
btrfs-progs: move the check_argc_* functions into utils.c
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Don't bother free the buffer if the malloc failed.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Steps to reproduce:
# mkfs.btrfs -f <dev1>
# btrfs-image <dev1> <image_file>
# btrfs-image -r -o <image_file> <dev2>
# btrfs check <dev2>
btrfs check output:
: read block failed check_tree_block
: Couldn't read tree root
: Couldn't open file system
The btrfs-image should not mess with the chunk tree under the old_restore way.
The new restore way was introduced by:
commit d6f7e3da0d
Btrfs-progs: make btrfs-image restore with a valid chunk tree V2
...
And the following commit enhanced the new restore on the valid chunk tree
building stuff:
commit ef2a8889ef
Btrfs-progs: make image restore with the original device offsets
...
But the second commit should not effect the old_restore way since the
old_restore way doesn't try to build a valid chunk tree at all.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
If the malloc above fails, the btrfs-image will exit directly
without any error messages.
Now just return the ENOMEM errno and let the caller prompt the
error message.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Handle the malloc failure for dump_worker in the same way as
the restore worker.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
When btrfs-image failed to create an image, the invalid output file
had better be deleted to prevent being used mistakenly in the future.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
For btrfs-image,
dump may not come with option '-o'
-r may not come with option '-c', '-s', '-w', dev_cnt != 1
-m may not come with dev_cnt < 2
All of the above should be regarded as invalid combinations,
and the usage will show up.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
The btrfs-image support multiple devices with -m specified.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
The value of variable leaf in while loop don't have to be set
for every round. Just move it outside.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
It's 32 bits as defined in ctree.h, but the struct had it as 64 bits.
Found using MemorySanitizer.
Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Reviewed-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
A mdrestore_struct was being written to without its mutex being held.
This race was found with ThreadSanitizer; the relevant part of the report
looks like this:
WARNING: ThreadSanitizer: data race (pid=18828)
Write of size 8 at 0x7fffffc3d088 by main thread:
#0 build_chunk_tree .../btrfs-progs/btrfs-image.c:2233
#1 __restore_metadump .../btrfs-progs/btrfs-image.c:2294
#2 restore_metadump .../btrfs-progs/btrfs-image.c:2345
#3 main .../btrfs-progs/btrfs-image.c:2545
Previous read of size 8 at 0x7fffffc3d088 by thread T1 (mutexes: write M0):
#0 restore_worker .../btrfs-progs/btrfs-image.c:1636
Location is stack of main thread.
Mutex M0 created at:
#0 pthread_mutex_init ??:0
#1 mdrestore_init .../btrfs-progs/btrfs-image.c:1766
#2 __restore_metadump .../btrfs-progs/btrfs-image.c:2286
#3 restore_metadump .../btrfs-progs/btrfs-image.c:2345
#4 main .../btrfs-progs/btrfs-image.c:2545
Thread T1 (tid=18830, running) created by main thread at:
#0 pthread_create ??:0
#1 mdrestore_init .../btrfs-progs/btrfs-image.c:1784
#2 __restore_metadump .../btrfs-progs/btrfs-image.c:2286
#3 restore_metadump .../btrfs-progs/btrfs-image.c:2345
#4 main .../btrfs-progs/btrfs-image.c:2545
Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
If pthread_create fails in mdrestore_init, then number of threads
created could be less then num of threads option. Hence pass number of
successful pthread_create calls to mdrestore_destroy, so that we don't
call pthread_join on IDs not present when pthread_create fails.
metadump_init already had this fixed, but repeats code from
metadump_destroy. Reuse metadump_destroy by passing number of threads
created (successful pthread_create calls) and save repeated cleaup
code. Had to move metadump_destroy before metadump_init for obvious
reasons.
Signed-off-by: Rakesh Pandit <rakesh@tuxera.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Remove unused variable in btrfs-image.c (update_super) and update man
page documentation about -r option. Running btrfsck on a restored
image produces missing chunk information. This is because by default,
btrfs-image fixes up chunk tree to use 1 stripe pointing to the
primary device. This in turns results in btrfsck making some noise.
$ ./mkfs.btrfs /dev/sdb2 -f
$ ./btrfs-image /dev/sdb2 btrfs_image_output
$ ./btrfs-image -r btrfs_image_output disk-image
$ ./btrfsck disk-image
Device extent[1, 29360128, 8388608] didn't find the relative chunk.
Device extent[1, 1111490560, 1073741824] didn't find the relative chunk.
Ideally btfsck should be updated to reflect this default behavior and
not through these messages, but it isn't harmful and can be done
later.
Signed-off-by: Rakesh Pandit <rakesh@tuxera.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
During restoring of image (-r using btrfs-image) we zero out RAID
profile in chunk type but forget to save BTRFS_BLOCK_GROUP_DUP if
present. This results in some false messages being printed by btrfsck.
$ ./mkfs.btrfs /dev/sdb2 -f
$ ./btrfs-image /dev/sdb2 btrfs_image_output
$ ./btrfs-image -r btrfs_image_output disk-image
$ ./btrfsck disk-image
Checking filesystem on disk-image
UUID: e644be2d-7701-4bd4-8804-7487f560d2a7
checking extents
Chunk[256, 228, 20971520]: length(8388608), offset(20971520), type(2) mismatch with block group[20971520, 192, 8388608]: offset(8388608), objectid(20971520), flags(34)
Chunk[256, 228, 29360128]: length(1073741824), offset(29360128), type(4) mismatch with block group[29360128, 192, 1073741824]: offset(1073741824), objectid(29360128), flags(36)
Block group[20971520, 8388608] (flags = 34) didn't find the relative chunk.
Block group[29360128, 1073741824] (flags = 36) didn't find the relative chunk.
Even though ./btrfsck on /dev/sdb2 seemed fine. This is due to type
mismatch above and type mismatch occured because we zero'ed out
BTRFS_BLOCK_GROUP_DUP while handling chunk trees.
Signed-off-by: Rakesh Pandit <rakesh@tuxera.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
So I needed to add a flag to not try to read block groups when doing
--init-extent-tree since we could hang there, but that meant adding a whole
other 0/1 type flag to open_ctree_fs_info. So instead I've converted it all
over to using a flags setting and added the flag that I needed. This has been
tested with xfstests and make test. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
This was found by static analysis.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
search_for_chunk_blocks() allocates a fixed-size buffer and then reads
arbitrary u32 sized buffers in to it. Instead let's fail if the item is
bigger than the buffer. This was found by static analysis.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
find_collision() allocates name_len bytes for its sub array so the index
must be less than name_len. This was found by static analysis.
Signed-off-by: Zach Brown <zab@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Remove unused eb parameter from btrfs_item_nr, unused since introduced
in 7777e63b42
Signed-off-by: Ross Kirk <ross.kirk@gmail.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Signed-off-by: "Chris West (Faux)" <git@goeswhere.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The variable "buf" passed into find_collision() as parameter "name"
should be freed on unsuccessful returns.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Port of commit b3b4aa7 to userspace.
parameter tree root it's not used since commit
5f39d397dfbe140a14edecd4e73c34ce23c4f9ee ("Btrfs: Create extent_buffer
interface for large blocksizes")
This gets userspace a tad closer to kernelspace by removing
this unused parameter that was all over the codebase...
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Alexandre pointed out that his xattrs have sensitive information in them as
well, so fix btrfs-image to zero out the data part of xattrs that we find.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Reviewed-by: Filipe Manana <fdmanana@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Some codes still use the cpu_to_lexx instead of the
BTRFS_SETGET_STACK_FUNCS declared in ctree.h.
Also added some BTRFS_SETGET_STACK_FUNCS for btrfs_header and
btrfs_super.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
This adds a 'btrfs-image -m' option, which let us restore an image that
is built from a btrfs of multiple disks onto several disks altogether.
This aims to address the following case,
$ mkfs.btrfs -m raid0 sda sdb
$ btrfs-image sda image.file
$ btrfs-image -r image.file sdc
---------
so we can only restore metadata onto sdc, and another thing is we can
only mount sdc with degraded mode as we don't provide informations of
another disk. And, it's built as RAID0 and we have only one disk,
so after mount sdc we'll get into readonly mode.
This is just annoying for people(like me) who're trying to restore image
but turn to find they cannot make it work.
So this'll make your life easier, just tap
$ btrfs-image -m image.file sdc sdd
---------
then you get everything about metadata done, the same offset with that of
the originals(of course, you need offer enough disk size, at least the disk
size of the original disks).
Besides, this also works with raid5 and raid6 metadata image.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
As for skinny metadata, key.offset stores levels rather than extent length.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
I noticed a slight problem with btrfs-image, since it was building a chunk tree
by setting the physical offset of the stripes to the same as the logical offset
it created this problem where the super block was now mapped into the file
system differently than it was before. This isn't a huge deal except that we
also carry along the free space cache with us, which is setup with the idea that
super at physical X is at logical Y. So this would make the free space checker
in fsck freak out because it would see that the cache says that the super block
is free space, and that the area where it thought the super block was located is
in fact used. In the mount case we'd end up overwriting real metadata with
backup super blocks. So we need to maintain the physical offsets in our
stripes. This is a huge pain because we store the logical bytenrs of all of our
metadata. This patch scans the entire image looking for chunk tree blocks and
builds an in memory chunk tree so we can write logical blocks to their physical
offsets. With this patch we no longer have the problems I described above.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
We've had a few users who wouldn't (or couldn't) provide us btrfs-images because
we maintain the file names when making an image. So introduce a sanitize
option. There are two uses, one that is fast and the other that is dog slow.
The fast way just generates garbage that's equal in length to the original name.
The slow way will try and find a crc32c collision for the file name that is also
the same length. Finding a crc32c collision for the file name "btrfs-progs" on
my box without CPU crc32c support takes a little more than 3 minutes, and a
little less than 2 minutes for my box that has CPU crc32c support, so it's a
lengthy and CPU intensive process.
The idea is that we use -s for most cases, and then only use -ss when we need
the file system tree to be somewhat sane. I could probably do a better job
about finding collisions, but I'll have to revist that later. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
btrfs-image is assuming that every metadata block is the same size. The
super is a special snowflake though. It's 4K even on 64K filesystems.
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
When working with a user with a broken file system I noticed I wasn't able to
read some of the blocks properly from the restored image. This is because his
extent tree was corrupt and was missing references to some of the blocks, which
means they weren't copied into the image when he generated it. So add a -w
option which will walk all of the trees manually and copy them into the image.
This way we can run fsck against a complete file system image and fix any bugs
in fsck. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
This fixes up the progs to properly deal with skinny metadata. This adds the -x
option to mkfs and btrfstune for enabling the skinny metadata option. This also
makes changes to fsck so it can properly deal with the skinny metadata entries.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Previously btrfs-image would set a METADUMP flag and would make one big system
chunk to cover the entire file system in the super in order to get around the
unpleasant business of having to adjust the chunk tree. This meant that you
could use the progs stuff on a restored file system, which is great for testing
btrfsck and other such things. But we want to be able to run the tree log
replay on a file system that is not able to run the tree log replay. So in
order to do this we need to fixup the super's chunk array and the chunk tree
itself. This is pretty easy since we restore using the logical offsets of the
metadata, so we just have to set the chunk items to have 1 stripe and have the
stripes point at the primary device and then use the logical offset of the chunk
as the physical offset. With this patch I can restore a file system image that
had a tree log and mount the file system and have the log be replayed
successfully. This patch also gives you the -o option in case you want the old
restore way, in the case where we want to make sure the system chunks as they
were given to us are correct. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
A lot of tree log replay bugs are because of strange space cache setups, so make
btrfs-image scrape the free space cache as well so we can better replicate what
a user is seeing if they have a tree log bug or anything related to free space
cache. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Currently btrfs-image doesn't copy the tree logs, which doesn't help me when
we're trying to debug log replay bugs. Since we don't have entries in the
extent root for the blocks we have to walk down all of the trees in order to
copy them. With this patch I can image a file system with a tree log and it
works fine. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
We use BUG_ON() everywhere in btrfs-image. When users are going to use this
command things are pretty dire, so I'd rather have really good error messages so
I know what happened rather than figure out which one of the 20 BUG_ON()'s made
the stupid thing exit early. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
"mdresotre" sounds like it might be French, but most
likely it's a misspelling of mdrestore. Fix it.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Zach Brown <zab@redhat.com>
This commit introduces a new kind of back reference for btrfs metadata.
Once a filesystem has been mounted with this commit, IT WILL NO LONGER
BE MOUNTABLE BY OLDER KERNELS.
The new back ref provides information about pointer's key, level and in which
tree the pointer lives. This information allow us to find the pointer by
searching the tree. The shortcoming of the new back ref is that it only works
for pointers in tree blocks referenced by their owner trees.
This is mostly a problem for snapshots, where resolving one of these fuzzy back
references would be O(number_of_snapshots) and quite slow. The solution used
here is to use the fuzzy back references in the common case where a given tree
block is only referenced by one root, and use the full back references when
multiple roots have a reference
This patch adds btrfs image tool. The image tool is
a debugging tool that creates/restores btrfs metadump
image.
Signed-off-by: Yan Zheng <zheng.yan@oracle.com>
Gui Hecheng
Adam Buchbinder
Rakesh Pandit
Wang Shilong
Anand Jain
Josef Bacik
Zach Brown
Ross Kirk
Chris West (Faux)
Eric Sandeen
Qu Wenruo
Liu Bo
Chris Mason
Danny Kukawka
Chris Mason
Wang Cong
yanhai zhu
Yan Zheng