We have this check in the kernel but not in userspace, which makes fsck
fail when we wouldn't have a problem in the kernel. This was meant to
catch this case because it really isn't good, unfortunately it will
require a design change to fix in the kernel so in the meantime add this
check so we can be sure our tests only catch real problems. Thanks,
Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
We hold a transaction open for the entirety of fixing extent refs. This works
out ok most of the time but we can be tight on space and run out of space when
fixing things. To get around this just push down the transaction starting dance
into the functions that actually fix things. This keeps us from ending up with
ENOSPC because we pinned everything and allows the code to be a bit simpler.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fb.com>
Hitting enospc problems with a really corrupt fs uncovered the fact that we
match any flag in a block group when creating space info's. This is a problem
if we have a raid level set, we'll end up with only one space info that covers
metadata and data because they share a raid level. We don't want this, we want
to separate out the data and metadata space infos, so mask off the raid level
and only use the main flags. Thanks,
Signed-off-by: Josef Bacik <jbacik@fb.com>
Allow read_tree_block() and read_node_slot() to return error pointer.
This should help caller to get more specified error number.
For existing callers, change (!eb) judgmentt to
(!extent_buffer_uptodate(eb)) to keep the compatibility, and for caller
missing the check, use PTR_ERR(eb) if possible.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
This change adds code to detect and fix the issue introduced in the kernel
release 3.17, where creation of read-only snapshots lead to a corrupted
filesystem if they were created at a moment when the source subvolume/snapshot
had orphan items. The issue was that the on-disk root items became incorrect,
referring to the pre orphan cleanup root node instead of the post orphan
cleanup root node.
A test filesystem can be generated with the test case recently submitted for
xfstests/fstests, which is essencially the following (bash script):
workout()
{
ops=$1
procs=$2
num_snapshots=$3
_scratch_mkfs >> $seqres.full 2>&1
_scratch_mount
snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
run_check $FSSTRESS_PROG -p $procs \
-x "$snapshot_cmd" -X $num_snapshots -d $SCRATCH_MNT -n $ops
}
ops=10000
procs=4
snapshots=500
workout $ops $procs $snapshots
Example of btrfsck's (btrfs check) behaviour against such filesystem:
$ btrfsck /dev/loop0
root item for root 311, current bytenr 44630016, current gen 60, current level 1, new bytenr 44957696, new gen 61, new level 1
root item for root 1480, current bytenr 1003569152, current gen 1271, current level 1, new bytenr 1004175360, new gen 1272, new level 1
root item for root 1509, current bytenr 1037434880, current gen 1300, current level 1, new bytenr 1038467072, new gen 1301, new level 1
root item for root 1562, current bytenr 33636352, current gen 1354, current level 1, new bytenr 34455552, new gen 1355, new level 1
root item for root 3094, current bytenr 1011712000, current gen 2935, current level 1, new bytenr 1008484352, new gen 2936, new level 1
root item for root 3716, current bytenr 80805888, current gen 3578, current level 1, new bytenr 73515008, new gen 3579, new level 1
root item for root 4085, current bytenr 714031104, current gen 3958, current level 1, new bytenr 716816384, new gen 3959, new level 1
Found 7 roots with an outdated root item.
Please run a filesystem check with the option --repair to fix them.
$ echo $?
1
$ btrfsck --repair /dev/loop0
enabling repair mode
fixing root item for root 311, current bytenr 44630016, current gen 60, current level 1, new bytenr 44957696, new gen 61, new level 1
fixing root item for root 1480, current bytenr 1003569152, current gen 1271, current level 1, new bytenr 1004175360, new gen 1272, new level 1
fixing root item for root 1509, current bytenr 1037434880, current gen 1300, current level 1, new bytenr 1038467072, new gen 1301, new level 1
fixing root item for root 1562, current bytenr 33636352, current gen 1354, current level 1, new bytenr 34455552, new gen 1355, new level 1
fixing root item for root 3094, current bytenr 1011712000, current gen 2935, current level 1, new bytenr 1008484352, new gen 2936, new level 1
fixing root item for root 3716, current bytenr 80805888, current gen 3578, current level 1, new bytenr 73515008, new gen 3579, new level 1
fixing root item for root 4085, current bytenr 714031104, current gen 3958, current level 1, new bytenr 716816384, new gen 3959, new level 1
Fixed 7 roots.
Checking filesystem on /dev/loop0
UUID: 2186e9b9-c977-4a35-9c7b-69c6609d4620
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 618537000 bytes used err is 0
total csum bytes: 130824
total tree bytes: 601620480
total fs tree bytes: 580288512
total extent tree bytes: 18464768
btree space waste bytes: 136939144
file data blocks allocated: 34150318080
referenced 27815415808
Btrfs v3.17-rc3-2-gbbe1dd8
$ echo $?
0
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
The three flags of @btrfs_path:
btrfs_path {
unsigned int keep_locks:1;
unsigned int skip_locking:1;
unsigned int leave_spinning:1;
}
have little meaning, because the userspace @btrfs_search_slot()
is free of locking and no other routines will decide their behavior
on these. So just remove them.
Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Fix following build warnings on 32bit platform:
...
utils.c:1708:3: warning: left shift count >= width of
type [enabled by default]
if (x << i & (1UL << 63))
^
qgroup-verify.c:393:9: warning: cast to pointer from integer
of different size [-Wint-to-pointer-cast]
return (struct tree_block *)unode->aux;
^
qgroup-verify.c:407:38: warning: cast from pointer to integer
of different size [-Wpointer-to-int-cast]
if (ulist_add(tree_blocks, bytenr, (unsigned long long)block, 0) >= 0)
^
cmds-restore.c:120:4: warning: format %lu expects argument of type
long unsigned int, but argument 3 has type size_t [-Wformat=]
fprintf(stderr, "bad compress length %lu\n", in_len);
...
BTW, this patch also switches other castings with new helpers.
Signed-off-by: Wang Shilong <wangshilong1991@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Previously, --init-extent-tree works just because btrfs_lookup_extent_info()
blindly return 0, and this make it work if there are not any *FULL BACKREF*
mode in broken filesystem.
It is just a coincidence that --init-extent-tree option works, let's
do it in the right way firstly.
For now, we have not supported to rebuild extent tree if there are
any *FULL BACKREF* mode which means if there are snapshots with broken
filesystem, avoid using --init-extent-tree option now.
Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <clm@fb.com>
mkfs -r wasn't creating chunks properly, making it very difficult to
allocate space for anything except tiny filesystems.
This changes it around to use more of the generic infrastructure, and
to do actual logical->physical block number translation.
It also allocates space to the files in smaller extents (max 1MB), which
keeps the allocator from trying to allocate an extent bigger than a
single chunk.
It doesn't quite support multi-device mkfs -r yet, but is much closer.
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In files copied from the kernel, mark many functions as static,
and remove any resulting dead code.
Some functions are left unmarked if they aren't static in the
kernel tree.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Port of commit b3b4aa7 to userspace.
parameter tree root it's not used since commit
5f39d397dfbe140a14edecd4e73c34ce23c4f9ee ("Btrfs: Create extent_buffer
interface for large blocksizes")
This gets userspace a tad closer to kernelspace by removing
this unused parameter that was all over the codebase...
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Also remove unused path in extent-tree.c:finish_current_insert().
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Before repeating the search in extent-tree.c:lookup_inline_extent_backref(),
release the current path.
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In btrfs_set_block_flags() we want to check if the slot
in the leaf points to the first item in the leaf - if it
doesn't check if the previous item in the leaf is an extent
item. By removing this extra slot decrement we are indeed
checking the item right before the slot, and not the second
item before.
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
If we did a tree search with the goal to find a metadata item
but the search failed with return value 1, we attempt to see
if in the same leaf there's a corresponding extent item, and if
there's one, just use it instead of doing another tree search
for this extent item. The check in the leaf was wrong because
it was seeking for a metadata item instead of an extent item.
This optimization was also being triggered incorrectly, as it
was evaluating path->slots which always evaluates to true. The
goal was to see if the leaf level slot was greater than zero
(i.e. not the first item in the leaf).
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Reviewed-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Several function return values were being completely
ignored.
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
extent-tree.c: In function 'btrfs_free_block_groups':
extent-tree.c:3190:12: warning: cast to pointer from integer of
different size [-Wint-to-pointer-cast]
Signed-off-by: Wang Shilong <wangsl-fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
extent_ref_type() contains inconsequential differences between
kernelspace and userspace, and has since the initial commits
to each. Just make userspace look like kernelspace.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
div_factor has been implemented for two times, cleanup it.
And I move them into a independent file named math.h because they are
common math functions.
[Eric Sandeen: port kernel commit 3fed40c to userspace]
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Remove some commented-out & #if 0'd code:
* close_blocks()
* btrfs_drop_snapshot()
* btrfs_realloc_node()
* btrfs_find_dead_roots()
There are still some #if 0'd functions in there, but I'm hedging
on those for now, they have been copied to cmds-check.c and I want
to see if they can be brough back into ctree.c eventually.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
Add chunk-recover program to check or rebuild chunk tree when the system
chunk array or chunk tree is broken.
Due to the importance of the system chunk array and chunk tree, if one of
them is broken, the whole btrfs will be broken even other data are OK.
But we have some hint(fsid, checksum...) to salvage the old metadata.
So this function will first scan the whole file system and collect the
needed data(chunk/block group/dev extent), and check for the references
between them. If the references are OK, the chunk tree can be rebuilt and
luckily the file system will be mountable.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: Miao Xie <miaox@cn.fujitsu.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The allocator looks for these hints when moving on to another block group which
will make it reset which block group it looks at, when we've already searched
that block group and didn't find any space to allocate, we need to fix this by
just letting the allocator make the determination if the block group is good
enough. This also fixes a problem where if we couldn't find space in the block
group we were given we'd just error out instead of moving on to the next block
group. Previously I couldn't fix some file systems that were relatively full,
but with this patch I can now run fsck on them with no allocation errors.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In some cases the extent tree can just be so gone there is no point in trying to
figure out how to put it back together. So add a --init-extent-tree mode which
will zero out the extent tree and then re-add extents for all of the blocks we
find. This will also undo any balance that was going on at the time of the
crash, this is needed because the reloc tree seems to confuse fsck at the
moment. With this patch I can put back together a users file system that was
completely gone. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
The tree log bug I introduced could create inconsistent file extent entries in
the file system tree and in some worst cases even create multiple extent entries
for the same entry. To fix this we need to do a few things
1) Keep track of extent items that overlap and then pick the one that covers the
largest area and delete the rest of the items.
2) Keep track of file extent items that land in extent items but don't match
disk_bytenr/disk_num_bytes exactly. Once we find these we need to figure out
who is the right ref and then fix all of the other refs to agree.
Each of these cases require a complete rescan of all of the extents, so
unfortunately if you hit this particular problem the fsck is going to take quite
a while since it will likely rescan all the trees 2 or 3 times. With this patch
the broken file system a user sent me is fixed and a broken file system that was
created by my reproducer is also fixed. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
In trying to track down a weird tree log problem I wanted to make sure that the
free space cache was actually valid, which we currently have no way of doing.
So this patch adds a bunch of support for the free space cache code and then a
checker to fsck. Basically we go through and if we can actually load the free
space cache then we will walk the extent tree and verify that the free space
cache exactly matches what is in the extent tree. Hopefully this will always be
correct, the only time it wouldn't is if the extent tree is corrupt or we have
some sort of awful bug in the free space cache. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
This fixes up the progs to properly deal with skinny metadata. This adds the -x
option to mkfs and btrfstune for enabling the skinny metadata option. This also
makes changes to fsck so it can properly deal with the skinny metadata entries.
Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
I've been working on btrfs-image and I kept seeing these leaks pop up on
valgrind so I'm just fixing them. We don't properly cleanup the device cache,
the chunk tree mapping cache, or the space infos on close. With this patch
valgrind doesn't complain about any memory leaks running btrfs-image. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
Allocate fs_info::super_copy dynamically of full BTRFS_SUPER_INFO_SIZE
and use it directly for saving superblock to disk.
This fixes incorrect superblock checksum after mkfs.
Signed-off-by: David Sterba <dsterba@suse.cz>
David Woodhouse originally contributed this code, and Chris Mason
changed it around to reflect the current design goals for raid56.
The original code expected all metadata and data writes to be full
stripes. This meant metadata block size == stripe size, and had a few
other restrictions.
This version allows metadata blocks smaller than the stripe size. It
implements both raid5 and raid6, although it does not have code to
rebuild from parity if one of the drives is missing or incorrect.
Signed-off-by: David Woodhouse <David.Woodhouse@intel.com>
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
This was a bug from long time ago that never actually got fixed. We start
with bytenr 0 when looping through all of the block groups, but
btrfs_lookup_block_group will bail out since it couldn't find a block group
with 0 as the bytenr. Btrfs_lookup_first_block_group will be nice and
adjust the start up to the right value, so this way we reset all the block
groups properly and not screw up the users block group accounting. Thanks,
Signed-off-by: Josef Bacik <jbacik@fusionio.com>
We need to align earlier to make sure we're getting things
properly setup against the raid56 stripes.
Signed-off-by: Chris Mason <chris.mason@fusionio.com>
This is mostly disabled, but it is step one in handling
corrupted block groups in the extent allocation tree.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
During btrfsck --repair, we make an index of extents that have incorrect
reference counts. Once we've collect the whole index, we go through
and modify the extent allocation tree to reflect the correct results.
Changing the extent allocation tree may free blocks, and so it may
end up removing a block that had a missing reference structure. The
fsck code may then circle back around and add the reference back.
The result is an extent that isn't actually used, but is recorded in the
extent allocation tree.
This commit adds a hook called as extents are freed. The hook searches
the index of incorrect references and updates it to reflect the freeing
of the extent.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
The block group accounting is fixed after we check the extent back
references. This makes sure the accounting is fixed unless we
were not able to repair the backrefs.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
The code that corrects the count of bytes used in each block group
was only marking block groups dirty when they contained extents. This
fixes things to dirty all the block groups, so any empty block groups
are written with their correct (zero) count.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
This also includes a new --repair btrfsck option. For now it can
only fix errors in the extent allocation tree.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
When creating a mixed fs with mkfs, an extra metadata chunk got allocated.
This is because btrfs_reserve_extent calls do_chunk_alloc for METADATA,
which in turn wasn't able to find the proper space_info, as __find_space_info
did a hard compare of the flags. It is now sufficient for the space_info to
include the proper flag. This reflects the change done to the kernel code
to support mixed chunks.
Also for a subsequent chunk allocation (which should not be hit in the mkfs
case), the chunk is now created with the flags from the space_info instead
of the requested flags. A better solution would be to pull the full changeset
for the mixed case from the kernel into the user mode (or, even better, share
the code)
The additional chunk probably confused block_rsv calculation, which in turn
led to severeal ENOSPC Oopses.
Signed-off-by: Arne Jansen <sensille@gmx.net>
Signed-off-by: Hugo Mills <hugo@carfax.org.uk>
When building on ppc64 I hit a number of warnings in printf:
btrfs-map-logical.c:69: error: format ‘%Lu’ expects type ‘long long
unsigned int’, but argument 4 has type ‘u64’
Fix them.
Signed-off-by: Anton Blanchard <anton@samba.org>
Signed-off-by: Hugo Mills <hugo@carfax.org.uk>
gcc-4.6 (as shipped in Fedora) turns on -Wunused-but-set-variable by
default, which breaks the build when combined with -Wall, e.g.:
debug-tree.c: In function ‘print_extent_leaf’:
debug-tree.c:45:13: error: variable ‘last_len’ set but not used [-Werror=unused-but-set-variable]
debug-tree.c:44:13: error: variable ‘last’ set but not used [-Werror=unused-but-set-variable]
debug-tree.c:41:21: error: variable ‘item’ set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
This patch fixes the errors by removing the unused variables.
Signed-off-by: Chris Ball <cjb@laptop.org>
Signed-off-by: Hugo Mills <hugo@carfax.org.uk>
When raid is setup with mkfs, it is supposed to cow the initial filesystem
it creates up to the desired raid level. RAID10 was not in the list
of RAID levels it checked for, so the initial FS created for RAID10
actually only lived on the first disk.
This works well enough because all the roots get quickly cowed during the
first mount. The exception is the data relocation tree, which only gets
cowed when we do a balance.
Signed-off-by: Chris Mason <chris.mason@oracle.com>
lookup_inline_extent_backref only checks for duplicate backref for data extent.
It assumes backrefs for tree block never conflict. This patch makes
lookup_inline_extent_backref check duplicate backrefs for both data and tree
block, so that we can detect potential bug earlier.
Signed-off-by: Yan Zheng <zheng.yan@oracle.com>
Signed-off-by: Chris Mason <chris.mason@oracle.com>
Write dirty block groups may make some block groups dirty.
This patch make btrfs_write_dirty_block_groups properly
handle the recursion.
Signed-off-by: Yan Zheng <zheng.yan@oracle.com>
Signed-off-by: Chris Mason <chris.mason@oracle.com>
This commit introduces a new kind of back reference for btrfs metadata.
Once a filesystem has been mounted with this commit, IT WILL NO LONGER
BE MOUNTABLE BY OLDER KERNELS.
The new back ref provides information about pointer's key, level and in which
tree the pointer lives. This information allow us to find the pointer by
searching the tree. The shortcoming of the new back ref is that it only works
for pointers in tree blocks referenced by their owner trees.
This is mostly a problem for snapshots, where resolving one of these fuzzy back
references would be O(number_of_snapshots) and quite slow. The solution used
here is to use the fuzzy back references in the common case where a given tree
block is only referenced by one root, and use the full back references when
multiple roots have a reference
There are still some warnings of the form:
format '%llu' expects type 'long long unsigned int' but argument has type 'u64'
In conjunction with -Werror, this is causing some build failures.
Now they're properly casted, avoiding compiler warnings.
Signed-off-by: Luca Bruno <lucab@debian.org>
Signed-off-by: Chris Mason <chris.mason@oracle.com>
This patch makes btrfsck check more things, including
directory items, file extents, checksumming, inode link
counts etc.
The code for these checks is similar to the code verifies
extent back references. The main difference is that
shared tree blocks are treated specially. The partial
checking results(unresolved references and/or errors)
of shared sub-trees are cached. This avoids scanning
the shared blocks several times. Thank you,
Signed-off-by: Yan Zheng <zheng.yan@oracle.com>
Josef Bacik
Qu Wenruo
Filipe Manana
Gui Hecheng
Wang Shilong
Wang Shilong
Chris Mason
Eric Sandeen
Filipe David Borba Manana
Wang Shilong
Miao Xie
Josef Bacik
David Sterba
David Woodhouse
Chris Mason
Josef Bacik
Arne Jansen
Anton Blanchard
Chris Ball
Yan Zheng
Luca Bruno