Commit Graph

415 Commits

Author SHA1 Message Date
Josef Bacik
05f4b61206 btrfs-progs: update btrfs_bin_search to match the kernel definition
This was updated to include a first_slot argument, update it to match
the kernel definition to make it easier to sync ctree.c.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-10-03 01:11:55 +02:00
Josef Bacik
ea58c1d9c2 btrfs-progs: stop using add_root_to_dirty_list in check
This is used to make sure the root is updated in the tree_root when we
re-init the root, however this function is static in the kernel and
doesn't need to be exported for any reason.  Simply update the root item
and then update it in the tree_root instead of adding it to the dirty
list.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-10-03 01:11:55 +02:00
Boris Burkov
14ac1a6051 btrfs-progs: mkfs: add support for squota
Add the ability to enable simple quotas from mkfs with '-O squota'

There is some complication around handling enable_gen while still
counting the root node of an fs. To handle this, employ a hack of doing
a no-op write on the root node to bump its generation up above that of
the qgroup enable generation, which results in counting it properly.

Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-10-03 01:11:55 +02:00
Boris Burkov
5bd97022f3 btrfs-progs: check: add support for squota
Add simple quotas checks to btrfs check.

Like the kernel feature, these checks bypass most of the backref walking
in the qgroups check. Instead, they enforce the invariant behind the
design of simple quotas by scanning the extent tree and determining the
owner of each extent:
Data: reading the owner ref inline item
Metadata: reading the tree block and reading its btrfs_header's owner

This gives us the expected count from squotas which we check against the
on-disk state of the qgroup items.

Signed-off-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-10-03 01:11:55 +02:00
Mark Harmstone
d8317a445c btrfs-progs: check: add check for metadata level mismatch
For a skinny metadata item in the extent tree, the key offset represents
the level of the tree it points to. This adds a check that these values
match, as otherwise it can cause a volume to go readonly when deleting a
large number of inodes.

See https://github.com/maharmstone/ntfs2btrfs/issues/51

Pull-request: #623
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-09-12 23:18:34 +02:00
David Sterba
2b35741a39 btrfs-progs: fix more typos found by codespell
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba
b06fe85011 btrfs-progs: check: convert int to bool in a few helpers
Signed-off-by: David Sterba <dsterba@suse.com>
2023-07-27 14:45:29 +02:00
Anand Jain
d46a0ef6a0 btrfs-progs: rename struct open_ctree_flags to open_ctree_args
The struct open_ctree_flags currently holds arguments for
open_ctree_fs_info(), it can be confusing when mixed with a local variable
named open_ctree_flags as below in the function cmd_inspect_dump_tree().

  cmd_inspect_dump_tree()
  ::
  struct open_ctree_flags ocf = { 0 };
  ::
  unsigned open_ctree_flags;

So rename struct open_ctree_flags to struct open_ctree_args.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-07-26 15:00:47 +02:00
Sam James
3ce3c8638c btrfs-progs: check: add a specific warning in --help for repair
Make it more explicit for the argument.

Pull-request: #635
Signed-off-by: Sam James <sam@gentoo.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-06-09 11:54:11 +02:00
Qu Wenruo
08a3bd7694 btrfs-progs: tune: add the ability to generate new data checksums
This patch would modify btrfs_csum_file_block() to handle csum type
other than the one used in the current fs.

The new data checksum would use a different objectid (-13) to
distinguish with the existing one (-10).
This needs to change tree-checker to skip the item size checks,
since new csum can be larger than the original csum.

After this stage, the resulted csum tree would look like this:

	item 0 key (CSUM_CHANGE EXTENT_CSUM 13631488) itemoff 8091 itemsize 8192
		range start 13631488 end 22020096 length 8388608
	item 1 key (EXTENT_CSUM EXTENT_CSUM 13631488) itemoff 7067 itemsize 1024
		range start 13631488 end 14680064 length 1048576

Note the itemsize is 8 times the original one, as the original csum is
CRC32, while target csum is SHA256, which is 8 times the size.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:32 +02:00
Josef Bacik
e150c843ea btrfs-progs: sync tree-checker.[ch] from kernel
This syncs tree-checker.c from the kernel.  The main modification was to
add a open ctree flag to skip the deeper leaf checks, and plumbing this
through tree-checker.c.  We need this for things like fsck or
btrfs-image that need to work with slightly corrupted file systems, and
these checks simply make us unable to look at the corrupted blocks.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
180bd5edd1 btrfs-progs: change btrfs_check_chunk_valid to match the kernel version
In btrfs-progs we check the actual leaf pointers as well as the chunk
itself in btrfs_check_chunk_valid.  However in the kernel the leaf stuff
is handled separately as part of the read, and then we have the chunk
checker itself.  Change the btrfs-progs version to match the in-kernel
version temporarily so it makes syncing the in-kernel code easier.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
361e4bea13 btrfs-progs: rename btrfs_check_* to __btrfs_check_*
These helpers are called __btrfs_check_* in the kernel as they return
the special enum to indicate what part of the leaf/node failed.  Rename
the uses in btrfs-progs to match the kernel naming convention to make it
easier to sync that code.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
e176d5eab6 btrfs-progs: add an atomic parameter to btrfs_buffer_uptodate
We have this extra parameter in the kernel to indicate if we are atomic
and thus can't lock the io_tree when checking the transid for an extent
buffer.  This isn't necessary in btrfs-progs, but to allow for easier
syncing of ctree.c add this argument to our copy of btrfs_buffer_uptodate.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
b3477244f9 btrfs-progs: update read_tree_block to match the kernel definition
The in-kernel version of read_tree_block adds some extra sanity checks
to make sure we don't return blocks that don't match what we expect.
This includes the owning root, the level, and the expected first key.
We don't actually do these checks in btrfs-progs, however kernel code
we're going to sync will expect this calling convention, so update it to
match the in-kernel code and then update all the callers.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
888a2b6a0c btrfs-progs: remove root argument from free_extent and inc_extent_ref
Neither of these actually need the root argument, we provide all the
information for the ref through the arguments we pass through.  Remove
the root argument from both of them.  These needed to be done in the
same patch because of the __btrfs_mod_ref helper which will pick one or
the other function for processing reference updates.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik
5780714b58 btrfs-progs: add btrfs_locking_nest to btrfs_alloc_tree_block
This is how btrfs_alloc_tree_block is defined in the kernel, so when we
go to sync this code in it'll be easier if we're already setup to accept
this argument.  Since we're in progs we don't care about nesting so just
use BTRFS_NORMAL_NESTING everywhere, as we sync in the kernel code it'll
get updated to whatever is appropriate.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik
006baaecdd btrfs-progs: rename btrfs_alloc_free_block to btrfs_alloc_tree_block
This is in keeping with what the function actually does, and is named
this way in the kernel.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik
8e427ada49 btrfs-progs: copy btrfs_root::state from kernel
We changed from members in the root for all the different flags to a
bit based flag system.  In order to make syncing the kernel code into
btrfs-progs easier go ahead and sync in the bits we use and update all
the users of the old ->track_dirty and ->ref_cows to use the state bits.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik
4a9a8f2a8a btrfs-progs: sync extent-io-tree.[ch] and misc.h from the kernel
This is a bit larger than the previous syncs, because we use
extent_io_tree's everywhere.  There's a lot of stuff added to
kerncompat.h, and then I went through and cleaned up all the API
changes, which were

- extent_io_tree_init takes an fs_info and an owner now.
- extent_io_tree_cleanup is now extent_io_tree_release.
- set_extent_dirty takes a gfpmask.
- clear_extent_dirty takes a cached_state.
- find_first_extent_bit takes a cached_state.

The diffstat looks insane for this, but keep in mind extent-io-tree.c
and extent-io-tree.h are ~2000 loc just by themselves.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik
f8efe9f724 btrfs-progs: sync file-item.h into progs
This patch syncs file-item.h into btrfs-progs.  This carries with it an
API change for btrfs_del_csums, which takes a root argument in the
kernel, so all callsites have been updated accordingly.

I didn't sync file-item.c because it carries with it a bunch of bio
related helpers which are difficult to adapt to the kernel.
Additionally there's a few helpers in the local copy of file-item.c that
aren't in the kernel that are required for different tools.

This requires more cleanups in both the kernel and progs in order to
sync file-item.c, so for now just do file-item.h in order to pull things
out of ctree.h.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik
c979ffd787 btrfs-progs: sync accessors.[ch] from the kernel
This syncs accessors.[ch] from the kernel.  For the most part
accessors.h will remain the same, there's just some helpers that need to
be adjusted for eb->data instead of eb->pages.  Additionally accessors.c
needed to be completely updated to deal with this as well.

This is a set of files where we will likely only sync the header going
forward, and leave the C file in place as it needs to be specific to
btrfs-progs.

This forced a few "unrelated" changes

- Using btrfs_dir_item_ftype() instead of btrfs_dir_item_type().  This
  is due to the encryption changes, and was simpler to just do in this
  patch.
- Adjusting some of the print tree code to use the actual helpers and
  not the btrfs-progs ones.

A local definition of static_assert is used to avoid compilation
failures on older gcc (< 9) where the 2nd parameter is mandatory.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
a754fe29d9 btrfs-progs: sync uapi/btrfs.h into btrfs-progs
We want to keep this file locally as we want to be uptodate with
upstream, so we can build btrfs-progs regardless of which kernel is
currently installed.  Sync this with the upstream version and put it in
kernel-shared/uapi to maintain some semblance of where this file comes
from.

There are some changes that need to be synced back to kernel. A local
definition of static_assert is used to avoid compilation problems on gcc
(< 9) due to mandatory 2nd parameter.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
b0a4eab561 btrfs-progs: remove parent_key arg from btrfs_check_* helpers
Now that this is unused by these helpers and only used by the repair
related code we can remove this argument from the main helpers.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
90e687d90c btrfs-progs: add a btrfs check helper for checking blocks
btrfs check wants to be able to record corrupted extents if it finds any
bad blocks.  This has been done directly inside of the
btrfs_check_leaf/btrfs_check_node helpers, but these are going to be
sync'ed from the kernel in the future.  Add another helper and move the
corrupt block handling into this helper and keep it inside of the check
code.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
d36571ed9a btrfs-progs: remove fs_info argument from btrfs_check_* helpers
This can be pulled out of the extent buffer that is passed in, drop the
fs_info argument from the function.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
bee10e7fc9 btrfs-progs: rename the qgroup structs to match the kernel
Now that the libbtrfs stuff has it's own local copy of ctree.h and
ioctl.h, let's rename these qgroup struct members to match the kernel
names, this way it'll make it easier to sync the kernel code into
btrfs-progs.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik
bf0f3db765 btrfs-progs: introduce UASSERT() for purely userspace code
While syncing messages.[ch] I had to back out the ASSERT() code in
kerncompat.h, which means we now rely on the kernel code for ASSERT().
In order to maintain some semblance of separation introduce UASSERT()
and use that in all the purely userspace code.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Qu Wenruo
d4f4d7b76e btrfs-progs: tune: add --convert-to-free-space-tree option
From the very beginning of free-space-tree feature, we allow mount
option "space_cache=v2" to convert the filesystem to the new feature.

But this is not the proper practice for new features (no matter if it's
incompat or compat_ro).

This is already making the clear_cache/space_cache mount option more
complex.

Thus this patch introduces the proper way to enable free-space-tree, and
I hope one day we can deprecate the "space_cache=" mount option.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:27 +02:00
Qu Wenruo
6fc223f0a0 btrfs-progs: make check/clear-cache.c to be separate from check/main.c
Currently check/clear-cache.c still uses a lot of global variables like
gfs_info and g_task_ctx, which are only implemented in check/main.c.

Since we have separated clear-cache code into its own c and header files,
we should not utilize those global variables.

Or it would lead to compiling errors for any tools which only has
check/clear-cache.o linked:

  /usr/bin/ld: check/clear-cache.o: in function `clear_free_space_cache':
  /home/adam/btrfs/btrfs-progs/check/clear-cache.c:46: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:56: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:67: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:73: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:84: undefined reference to `gfs_info'
  /usr/bin/ld: check/clear-cache.o:/home/adam/btrfs/btrfs-progs/check/clear-cache.c:85: more undefined references to `gfs_info' follow
  /usr/bin/ld: check/clear-cache.o: in function `check_space_cache':
  /home/adam/btrfs/btrfs-progs/check/clear-cache.c:357: undefined reference to `g_task_ctx'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:357: undefined reference to `g_task_ctx'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:358: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:365: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:373: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:374: undefined reference to `gfs_info'
  /usr/bin/ld: /home/adam/btrfs/btrfs-progs/check/clear-cache.c:382: undefined reference to `gfs_info'
  /usr/bin/ld: check/clear-cache.o:/home/adam/btrfs/btrfs-progs/check/clear-cache.c:383: more undefined references to `gfs_info' follow
  collect2: error: ld returned 1 exit status

This provides the basis for later clear cache usage out of check realm.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:27 +02:00
David Sterba
cdb82bf762 btrfs-progs: check: document --force + --repair
The combination of --force and --repair was mistakenly documented as not
working on a mounted filesystem. Update help text, documentation and the
warning itself.

Issue: #503
Signed-off-by: David Sterba <dsterba@suse.com>
2023-04-25 16:59:42 +02:00
David Sterba
4b8d51ded9 btrfs-progs: check: convert help text to option formatter
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-28 20:11:23 +01:00
Qu Wenruo
f61b90aff9 btrfs-progs: make usage call properly return an exit value
[BUG]
Currently cli/009 test case failed with different exit number:

  ====== RUN CHECK /home/adam/btrfs-progs/btrfstune --help
  usage: btrfstune [options] device
  [...]
  failed: /home/adam/btrfs-progs/btrfstune --help
  test failed for case 009-btrfstune

[CAUSE]
In tune/main.c, we have the following call on usage():

  static void print_usage(int ret)
  {
	usage(&tune_cmd);
	exit(ret);
  }

However usage() itself would always call exit(1):

  void usage(const struct cmd_struct *cmd)
  {
	usage_command_usagestr(cmd->usagestr, NULL, 0, true, true);
	exit(1);
  }

This makes prevents any caller of usage() to modify its exit number.

[FIX]
Add a new argument @error for print_usage(), so we can properly return 0
for -h/--help usage.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-28 20:11:23 +01:00
Qu Wenruo
04930385fd btrfs-progs: check: enhance the error output for backref mismatch
[PROBLEM]
Btrfs check original mode output is not that reader friendly already, it
even includes pointer output:

  backref 15353727729664 parent 1140559929556992 not referenced back 0xc9133d70
  tree backref 15353727729664 parent 14660022714368 not found in extent tree
  incorrect global backref count on 15353727729664 found 3 wanted 2
  backpointer mismatch on [15353727729664 16384]

In above case, the "0xc9133d70" is completely useless, as it's a pointer
for the tree_backref structure.

And the term "backref" is quite abused in above case.

[ENHANCEMENT]
To enhance the situation, let's use some output format from lowmem mode
instead.

Now above example will be changed to:

  tree extent[15353727729664, 16384] parent 1140559929556992 has no tree block found
  tree extent[15353727729664, 16384] parent 14660022714368 has no backref item in extent tree
  incorrect global backref count on 15353727729664 found 3 wanted 2
  backpointer mismatch on [15353727729664 16384]

And some example for data backrefs:

  data extent[12845056, 1048576] bytenr mimsmatch, extent item bytenr 12845056 file item bytenr 0
  data extent[12845056, 1048576] referencer count mismatch (root 5 owner 257 offset 0) wanted 1 have 0

  data extent[14233600, 12288] referencer count mismatch (parent 42139648) wanted 0 have 1
  data extent[14233600, 12288] referencer count mismatch (root 5 owner 307 offset 0) wanted 0 have 1
  data extent[14233600, 12288] referencer count mismatch (parent 30507008) wanted 0 have 1

Furthermore, the original function print_tree_backref_error() is a mess
already, here we clean it up by exacting all the error output into a
dedicated helper, print_backref_error(), so the function itself only
need to find out errors.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-01-25 16:19:59 +01:00
David Sterba
e78fe2d92e Revert "btrfs-progs: rename qgroup items to match the kernel naming scheme"
This reverts commit 03451430de.
(It's not 1:1, there are some additional trivial fixups in cmds/qgroup.c)

This breaks a lot of 3rd party tools that depend on it as Neal reports:

* btrfs-assistant
* buildah
* cri-o
* podman
* skopeo
* containerd
* moby/docker
* snapper
* source-to-image

Link: https://lore.kernel.org/linux-btrfs/CAEg-Je8L7jieKdoWoZBuBZ6RdXwvwrx04AB0fOZF1fr5Pb-o1g@mail.gmail.com/
Reported-by: Neal Gompa <ngompa@fedoraproject.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-01-03 13:10:54 +01:00
Josef Bacik
405b5cadd3 btrfs-progs: replace btrfs_leaf_data with btrfs_item_nr_offset
We're using btrfs_item_nr_offset(leaf, 0) to get the start of the leaf
data in the kernel, we don't have btrfs_leaf_data.  Replace all
occurrences of btrfs_leaf_data() with btrfs_item_nr_offset(leaf, 0) in
order to make syncing accessors.[ch] easier.  ctree.c will be synced
later, so this is simply an intermediate step.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-30 19:14:29 +01:00
Josef Bacik
788a71c16a btrfs-progs: sync compression.h from the kernel
This patch copies in compression.h from the kernel.  This is relatively
straightforward, we just have to drop the compression types definition
from ctree.h, and update the image to use BTRFS_NR_COMPRESS_TYPES
instead of BTRFS_COMPRESS_LAST, and add a few things to kerncompat.h to
make everything build smoothly.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-30 19:14:29 +01:00
Josef Bacik
fac1fae3ef btrfs-progs: rename extent buffer flags to EXTENT_BUFFER_*
We have been overloading the extent_state flags for use on the extent
buffers as well.  When we sync extent-io-tree.[ch] this will become
impossible, so rename these flags to EXTENT_BUFFER_* and use those
definitions instead of the extent_state definitions.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-28 18:57:44 +01:00
Josef Bacik
2b9fba7974 btrfs-progs: rename btrfs_item_end to btrfs_item_data_end
This matches what we did in the kernel, btrfs_item_data_end is more
inline with what the helper does, which is give us the offset of the end
of the data portion of the item, not the offset of the end of the item
itself.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-28 18:57:43 +01:00
Josef Bacik
53f8b8fd01 btrfs-progs: make btrfs_qgroup_level helper match the kernel
We return __u16 in the kernel, as this is actually the size of
btrfs_qgroup_level.  Adjust the existing helpers and update all the
callers to deal with the new size appropriately.  This will make syncing
the kernel code cleaner.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-28 18:57:43 +01:00
Josef Bacik
03451430de btrfs-progs: rename qgroup items to match the kernel naming scheme
We're going to sync the kernel source into btrfs-progs, and in the
kernel we have all these qgroup fields named with short names instead of
the full name, so rename

referenced -> rfer
compressed -> cmpr
exclusive -> excl

to match the kernel and update all the users, this will make the sync
cleaner.

ioctl.h is a public header but there are no users of the
btrfs_qgroup_limit structure.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-28 18:57:43 +01:00
David Sterba
44efde7fa0 btrfs-progs: unify naming of qgroup subvolid helpers
Kernel function name is btrfs_qgroup_subvolid so rename it in progs. The
libbtrfs can't API be changed without versioning so at least add the new
helper.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-26 09:36:44 +02:00
David Sterba
ccb2d4aa45 btrfs-progs: device-utils: rename btrfs_device_size
There's a group of helpers to read device size, the btrfs_device_size
should be one of them. Rename it and so minor cleanup.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
a827bb2db8 btrfs-progs: use template for transaction commit error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
8fcafae04a btrfs-progs: use template for transaction start error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
c2be0e2ce0 btrfs-progs: use template for out of memory error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
b5cf79f107 btrfs-progs: check: rename global repair option
Make the repair mode more visible with prefix opt_ for option and also
add 'check' for context.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
2267708bfe btrfs-progs: move repair.c from common/ to check/
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
e012d1fedb btrfs-progs: check: factor out code for clearing caches
Move helpers related to clearing space and inode cache from main.co to a
separate file.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
f7224bdd12 btrfs-progs: check: move global variables to common headers
Add declarations for global fs_info and task context so they can be
accessed from any .c file once the main.c will be split. Add prefix "g_"
for the task.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
feef6aaaf6 btrfs-progs: kernel-lib: remove radix-tree
The radix-tree is not used in userspace code. In kernel it's for
tracking unpersisted and in-memory structures and has been replaced by
the xarray.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:07 +02:00
David Sterba
0e38e1c4f2 btrfs-progs: use error helper for messages in non-kernel code
Lots of code still uses fprintf(stderr, "...") that should be the
error() helper. The kernel-shared code is left out of the conversion for
now.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:07 +02:00
David Sterba
b5aac254c7 btrfs-progs: common: update include lists, part 1
The tool IWYU (include what you use) suggests to remove and add some
includes. This is only partial to avoid accidental build breakage, the
includes are entangled and will have to be cleaned in the future again.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:07 +02:00
David Sterba
6edd4b2121 btrfs-progs: factor string helpers out of utils.c
Utils is the catch-all file, we can now separate some string utility
functions.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:13 +02:00
David Sterba
bd94a00de0 btrfs-progs: check: update include lists
The tool IWYU (include what you use) suggests to remove and add some
includes.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:12 +02:00
David Sterba
8802a81eb8 btrfs-progs: check: reorder includes
The preferred order:
- system headers
- standard headers
- libraries
- kernel library
- kernel shared
- common headers
- other tools
- own headers

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:12 +02:00
Qu Wenruo
1430b41427 btrfs-progs: separate block group tree from extent tree v2
Block group tree feature is completely a standalone feature, and it has
been over 5 years before the initial introduction to solve the long
mount time.

I don't really want to waste another 5 years waiting for a feature which
may or may not work, but definitely not properly reviewed for its
preparation patches.

So this patch will separate the block group tree feature into a
standalone compat RO feature.

There is a catch, in mkfs create_block_group_tree(), current
tree-checker only accepts block group item with valid chunk_objectid,
but the existing code from extent-tree-v2 didn't properly initialize it.

This patch will also fix above mentioned problem so kernel can mount it
correctly.

Now mkfs/fsck should be able to handle the fs with block group tree.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-09-12 18:25:32 +02:00
Su Yue
c7d5638824 btrfs-progs: free extent buffer after repairing wrong transid eb
In read_tree_block, extent buffer EXTENT_BAD_TRANSID flagged will
be added into fs_info->recow_ebs with an increment of its refs.

The corresponding free_extent_buffer should be called after we
fix transid error by cowing extent buffer then remove them from
fs_info->recow_ebs.

Otherwise, extent buffers will be leaked as fsck-tests/002 reports:
===================================================================
====== RUN CHECK /root/btrfs-progs/btrfs check --repair --force ./default_case.img.restored
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
parent transid verify failed on 29360128 wanted 9 found 755944791
Ignoring transid failure
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
extent buffer leak: start 29360128 len 4096
enabling repair mode
===================================================================

Fixes: c64485544b ("Btrfs-progs: keep track of transid failures and fix them if possible")
Signed-off-by: Su Yue <glass@fydeos.io>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-09-12 15:31:23 +02:00
Qu Wenruo
0a348d3c81 btrfs-progs: check: verify the underlying block device size is valid
[BUG]
There is a bug report that, one btrfs got its underlying device shrunk
accidentally.

Fortunately the user has no data at the truncated range.  However kernel
will reject such filesystem, while btrfs-check reports nothing wrong
with it.

This can be easily reproduced by:

  # truncate -s 1G test.img
  # mkfs.btrfs test.img
  # truncate -s 996M test.img
  # btrfs check test.img
  Opening filesystem to check...
  Checking filesystem on test.img
  UUID: dbf0a16d-f158-4383-9025-29d7f4c43f17
  [1/7] checking root items
  [2/7] checking extents
  [3/7] checking free space tree
  [4/7] checking fs roots
  [5/7] checking only csums items (without verifying data)
  [6/7] checking root refs
  [7/7] checking quota groups skipped (not enabled on this FS)
  found 16527360 bytes used, no error found
                             ^^^^^^^^^^^^^^
  total csum bytes: 13836
  total tree bytes: 2359296
  total fs tree bytes: 2162688
  total extent tree bytes: 65536
  btree space waste bytes: 503569
  file data blocks allocated: 14168064
   referenced 14168064

[CAUSE]
Btrfs check really only checks the metadata cross references, not really
bothering if the underlying device has correct size.  Thus we completely
ignored such size mismatch.

[FIX]
For both regular and lowmem mode, add extra check against the underlying
block device size.
If the block device size is smaller than its total_bytes, gives a error
message and error out.

Now the check looks like this for both modes:

  ...
  [2/7] checking extents
  ERROR: block device size is smaller than total_bytes in device item, has 1046478848 expect >= 1073741824
  ERROR: errors found in extent allocation tree or chunk allocation
  [3/7] checking free space tree
  ...
  found 16527360 bytes used, error(s) found

Issue: #504
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-09-12 15:31:15 +02:00
Qu Wenruo
650282a04c btrfs-progs: fix eb leak caused by missing btrfs_release_path() call.
[BUG]
Commit 06b6ad5e01 ("btrfs-progs: check: check for invalid free space
tree entries") makes btrfs check to report eb leakage even on newly
created btrfs:

 # mkfs.btrfs -f test.img
 # btrfs check test.img
  Opening filesystem to check...
  Checking filesystem on test.img
  UUID: 13c26b6a-3b2c-49b3-94c7-80bcfa4e494b
  [1/7] checking root items
  [2/7] checking extents
  [3/7] checking free space tree
  [4/7] checking fs roots
  [5/7] checking only csums items (without verifying data)
  [6/7] checking root refs
  [7/7] checking quota groups skipped (not enabled on this FS)
  found 147456 bytes used, no error found
  total csum bytes: 0
  total tree bytes: 147456
  total fs tree bytes: 32768
  total extent tree bytes: 16384
  btree space waste bytes: 140595
  file data blocks allocated: 0
   referenced 0
  extent buffer leak: start 30572544 len 16384 <<< Extent buffer leakage

[CAUSE]

The patch in mailinglist uses a dynamically allocated path while the
committed one has been converted to on-stack path, which is preferred.
However, the cleanup was not done properly.  We only release the path
inside the while loop, no at out label.  This means, if we hit error or
even just exhausted free space tree as expected, we will leak the path
to free space tree root.

Thus leading to the above leak report.

[FIX]
Fix the bug by calling btrfs_release_path() at out: label too.

This should make the code behave the same as the patch submitted to the
mailing list.

Fixes: 06b6ad5e01 ("btrfs-progs: check: check for invalid free space tree entries")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-09-10 22:50:47 +02:00
Josef Bacik
06b6ad5e01 btrfs-progs: check: check for invalid free space tree entries
While testing some changes to how we reclaim block groups I started
hitting failures with my TEST_DEV.  This occurred because I had a bug
and failed to properly remove a block groups free space tree entries.
However this wasn't caught in testing when it happened because
btrfs check only checks that the free space cache for the existing block
groups is valid, it doesn't check for free space entries that don't have
a corresponding block group.

Fix this by checking for free space entries that don't have a
corresponding block group.  Additionally add a test image to validate
this fix.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-08-16 15:18:12 +02:00
David Sterba
a33af50c52 btrfs-progs: add constant for initial getopt values
Add constant for initial value to avoid unexpected clashes with user
defined getopt values and shift the common size getopt values.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-08-16 15:18:11 +02:00
Qu Wenruo
9bded24a46 btrfs-progs: do not use btrfs_commit_transaction() just to update super blocks
There are several call sites utilizing btrfs_commit_transaction() just
to update members in super blocks, without any metadata update.

This can be problematic for some simple call sites, like zero_log_tree()
or check_and_repair_super_num_devs().

If we have big problems preventing the fs to be mounted in the first
place, and need to clear the log or super block size, but by some other
problems in extent tree, we're unable to allocate new blocks.

Then we fall into a deadlock that, we need to mount (even
ro,rescue=all) to collect extra info, but btrfs-progs can not do any
super block updates.

Fix the problem by allowing the following super blocks only operations
to be done without using btrfs_commit_transaction():

- btrfs_fix_super_size()
- check_and_repair_super_num_devs()
- zero_log_tree().

There are some exceptions in btrfstune.c, related to the csum type
conversion and seed flags.

In those btrfstune cases, we in fact wants to proper error report in
btrfs_commit_transaction(), as those operations are not mount critical,
and any early error can be helpful to expose any problems in the fs.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-05-20 15:54:16 +02:00
Qu Wenruo
90850c26fa btrfs-progs: remove unused header check/btrfsck.h
It should have been deleted, as CHANGES mentioned this in v5.14, but
obvious it's not.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-05-17 21:12:19 +02:00
Qu Wenruo
23c6d24668 btrfs-progs: check: lowmem, fix path leak when dev extents are invalid
[BUG]
When testing my new RAID56J code, there is a bug causing dev extents
overlapping.

Although both modes can detect the problem, lowmem has leaked some
extent buffers:

  $ btrfs check --mode=lowmem /dev/test/scratch1
  Opening filesystem to check...
  Checking filesystem on /dev/test/scratch1
  UUID: 65775ce9-bb9d-4f61-a210-beea52eef090
  [1/7] checking root items
  [2/7] checking extents
  ERROR: dev extent devid 1 offset 1095761920 len 1073741824 overlap with previous dev extent end 1096810496
  ERROR: dev extent devid 2 offset 1351614464 len 1073741824 overlap with previous dev extent end 1352663040
  ERROR: dev extent devid 3 offset 1351614464 len 1073741824 overlap with previous dev extent end 1352663040
  ERROR: errors found in extent allocation tree or chunk allocation
  [3/7] checking free space tree
  [4/7] checking fs roots
  [5/7] checking only csums items (without verifying data)
  [6/7] checking root refs done with fs roots in lowmem mode, skipping
  [7/7] checking quota groups skipped (not enabled on this FS)
  found 3221372928 bytes used, error(s) found
  total csum bytes: 0
  total tree bytes: 147456
  total fs tree bytes: 32768
  total extent tree bytes: 16384
  btree space waste bytes: 136231
  file data blocks allocated: 3221225472
   referenced 3221225472
  extent buffer leak: start 30752768 len 16384
  extent buffer leak: start 30752768 len 16384
  extent buffer leak: start 30752768 len 16384

[CAUSE]
In the function check_dev_item(), we iterate through all the dev
extents, but when we found overlapping extents, we exit without
releasing the path, causing extent buffer leakage.

[FIX]
Just release the path before we exit the function.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-05-17 21:12:19 +02:00
Qu Wenruo
3ff9d35257 btrfs-progs: use read_data_from_disk() to replace read_extent_from_disk() and replace read_extent_data()
The function read_extent_from_disk() is only a wrapper to read tree
block.

And read_extent_data() is just a while loop to eliminate short read
caused by stripe boundary.

In fact, a lot of call sites of read_extent_data() are either reading
metadata (thus no possible short read) or doing extra loop by
themselves.

This patch will replace those two functions with read_data_from_disk(),
making it the only entrance for data/metadata read.
And update read_data_from_disk() to return the read bytes, so caller can
do a simple while loop.

For the few callers of read_extent_data(), open-code a small while loop
for them.

This will allow later RAID56 read repair using P/Q much easier.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-04-25 19:08:30 +02:00
Qu Wenruo
0dc8b8b6a4 btrfs-progs: check: fix wrong total bytes check for seed device
[BUG]
The following script can lead to false positive from btrfs check:

  mkfs.btrfs -f $dev1
  mount $dev1 $mnt
  btrfstune -S1 $dev1
  mount $dev1 $mnt
  btrfs dev add -f $dev2 $mnt
  umount $mnt

  # Now dev1 is seed, and dev2 is the rw fs.
  btrfs check $dev2
  ...
  [2/7] checking extents
  WARNING: minor unaligned/mismatch device size detected
  WARNING: recommended to use 'btrfs rescue fix-device-size' to fix it
  ...

This false positive only happens on $dev2, $dev1 is completely fine.

[CAUSE]
The warning is from is_super_size_valid(), in that function we verify
the super block total bytes (@super_bytes) is correct against the total
device bytes (@total_bytes).

However the when calculating @total_bytes, we only use devices in
current fs_devices, which only contains RW devices.

Thus all bytes from seed device are not taken into consideration, and
trigger the false positive.

[FIX]
Fix it by also iterating seed devices.

Since we're here, also output @total_bytes and @super_bytes when
outputting the warning message, to allow end users have a better idea on
what's going wrong.

Reviewed-by: Su Yue <l@damenly.su>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-04-25 18:59:44 +02:00
Mark Harmstone
ef194732d5 btrfs-progs: check: add check for overlong xattr names
While working on my Windows driver, I found that it was inadvertently
allowing users to create xattrs with names longer than 255 bytes, which
wasn't being picked up by btrfs-check.

If the Linux driver encounters a file with an invalid xattr like this,
it makes the whole directory it's in inaccessible. If it's the root
directory, it'll refuse to mount the filesystem entirely.

Pull-request: #456
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-04-25 18:52:49 +02:00
Qu Wenruo
b18ceb8585 btrfs-progs: check: add check and repair ability for super num devs mismatch
[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in
super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in
chunk tree.

[FIX]
Add such check and repair ability for btrfs-check.

The ability is mode independent.

Reported-by: Luca Béla Palkovics <luca.bela.palkovics@gmail.com>
Link: https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-24 00:36:13 +01:00
Qu Wenruo
fd500029eb btrfs-progs: check: fix two error messages used in qgroup verification
[BUG]
There is a weird error message when running btrfs check on a specific
image:

  [7/7] checking quota groups
  ERROR: out of memory
  ERROR: Loading qgroups from disk: -2
  ERROR: failed to check quota groups

[CAUSE]
The "Out of memory" one is in load_quota_info(), which is output in two
cases:

- No memory can be allocated for btrfs_qgroup_list
  AKA, real -ENOMEM.

- No qgroup can be found for either the child or the parent qgroup
  This returnes -ENOENT.

Obvious the image has hit -ENOENT case, but the error message is fixed
to ENOMEM case.

[FIX]
Fix it by using %m to output the real reason of failure.

Reported-by: Andrei Borzenkov <arvidjaar@gmail.com>
Link: https://forums.opensuse.org/showthread.php/567851-btrfs-fails-to-load-qgroups-from-disk-with-error-2-(out-of-memory)
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-23 18:18:31 +01:00
Josef Bacik
e8f7f034af btrfs-progs: check: don't do the root item check for extent tree v2
With the current set of changes we could probably do this check, but it
would involve changing the code quite a bit, and in the future we're not
going to track the metadata in the extent tree at all.  Since this check
was for a very old kernel just skip it for extent tree v2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:34 +01:00
Josef Bacik
c36d8ea235 btrfs-progs: check: handle the block group tree properly
We need to make sure we process the block group root, and mark its
blocks as used for the free space tree checking.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:12 +01:00
Josef Bacik
37d05e9b1e btrfs-progs: check: add helper to reinit the root based on a key
In the case of per-bg roots we may be missing the root items.  To
re-initialize them we want to add the root item as well as allocate the
empty block.  To achieve this extract out the reinit root logic to a
helper that just takes the root key and then does the appropriate work
to allocate an empty root and update the root item.  Fix the normal
reinit root helper to use this new helper.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:10 +01:00
Josef Bacik
f9b65a54c0 btrfs-progs: check: make free space tree validation extent tree v2 aware
The free space tree needs to be validated against all referenced blocks
in the file system, so use the btrfs_mark_used_blocks() helper to check
the free space tree and free space cache against.  This will do the
right thing for both extent tree v1 and extent tree v2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:08 +01:00
Josef Bacik
b4d714a47a btrfs-progs: qgroup-verify: scan extents based on block groups
When we switch to per-block group extent roots we'll need to scan each
individual extent root.  To make this easier in the future go ahead and
use the range of the block groups to scan the extents.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:06 +01:00
Josef Bacik
1bbfc9c9a6 btrfs-progs: check: add block group tree support
This makes the appropriate changes to enable the block group tree
checking for both lowmem and normal check modes.  This is relatively
straightforward, simply need to use the helper to get the right root for
dealing with block groups.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:07:04 +01:00
Josef Bacik
9f1f88c642 btrfs-progs: check-lowmem: use the btrfs_block_group_root helper
When we're messing with block group items use the
btrfs_block_group_root() helper to get the correct root to search, and
this will do the right thing based on the file system flags.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:06:57 +01:00
Josef Bacik
360103e610 btrfs-progs: mkfs: use the btrfs_block_group_root helper
Instead of accessing the extent root directory for modifying block
groups, use the helper which will do the correct thing based on the
flags of the file system.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 18:06:56 +01:00
Josef Bacik
4f184cc911 btrfs-progs: make all of the item/key_ptr offset helpers take an eb
When we change the size of the btrfs_header we're going to need to
change how these helpers calculate where to find the start of items or
block ptrs.  To prepare for that make these helpers take the
extent_buffer as an argument so we can do the appropriate math based on
the version type.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:14 +01:00
Josef Bacik
5dc3964aaa btrfs-progs: remove the _nr from the item helpers
Now that all callers are using the _nr variations we can simply rename
these helpers to btrfs_item_##member/btrfs_set_item_##member and change
the actual item SETGET funcs to raw_item_##member/set_raw_item_##member
and then change all callers to drop the _nr part.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik
f3be0ff01a btrfs-progs: rename btrfs_item_end_nr to btrfs_item_end
All callers use the btrfs_item_end_nr() variation, simply drop
btrfs_item_end() and make btrfs_item_end_nr() use the _nr() variations
of the item get helpers.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik
49539423fa btrfs-progs: change btrfs_file_extent_inline_item_len to take a slot
This matches how the kernel does it, simply pass in the slot and fix up
btrfs_file_extent_inline_item_len to use the btrfs_item_nr() helper and
the correct define.  Fixup all the callers to use the slot now instead
of passing in the btrfs_item.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik
04ffea07e4 btrfs-progs: add btrfs_set_item_*_nr() helpers
We have a lot of the following patterns

	item = btrfs_item_nr(nr);
	btrfs_set_item_*(eb, item, val);

	btrfs_set_item_*(eb, btrfs_item_nr(nr), val);

in a lot of places in our code.  Instead add _nr variations of these
helpers and convert all of the users to this new helper.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik
80ff3f2c8e btrfs-progs: btrfs_item_size_nr/btrfs_item_offset_nr everywhere
We have this pattern in a lot of places

	item = btrfs_item_nr(slot);
	btrfs_item_size(leaf, item);
	btrfs_item_offset(leaf, item);

when we could simply use

	btrfs_item_size_nr(leaf, slot);
	btrfs_item_offset_nr(leaf, slot);

Fix all callers of btrfs_item_size() and btrfs_item_offset() to use the
_nr variation of the helpers.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik
c9703d8726 btrfs-progs: handle orphan directories properly
When implementing the GC tree I started getting btrfsck errors when a
test rm -rf <directory> with files inside of it and immediately unmount,
leaving behind orphaned directory items that have GC items for them.

This made me realize that we don't actually handle this case currently
for our normal orphan path.  If we fail to clean everything up and leave
behind the orphan items we'll fail fsck.

Fix this by not processing any backrefs we find if we found an inode
item and its nlink is 0.  This allows us to pass the test case I've
provided to validate this patch.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-08 18:40:45 +01:00
Josef Bacik
65dde617b1 btrfs-progs: check: fix check_global_roots_uptodate
While running make test on other patches I noticed we are now
segfaulting on the fuzz tests.  This is because when I converted us to a
rb tree for the global roots we lost the ability to catch that there's
no extent root at all.  Before we'd populate a dummy
fs_info->extent_root with a not uptodate node, but now you simply don't
get an extent root in the rb_tree.  Fix the check_global_roots_uptodate
helper to count how many roots we find and make sure it matches the
number we expect.  If it doesn't then we can return -EIO.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-08 18:18:00 +01:00
Qu Wenruo
1a8f283c97 btrfs-progs: check: orig: reject bad metadata backref with invalid level
[BUG]
There is a bug report that kernel tree-checker rejected an invalid
metadata item:

 corrupt leaf: block=934474399744 slot=68 extent bytenr=425173254144 len=16384 invalid tree level, have 33554432 expect [0, 7]

But original mode btrfs-check reports nothing wrong.
(lowmem mode will just crash, and fixed in previous patch).

[CAUSE]
For original mode it doesn't really check tree level, thus didn't find
the problem.

[FIX]
I don't have a good idea to completely make original mode to verify the
level in backref and in the tree block (while lowmem does that).

But at least we can detect obviously corrupted level just like kernel.

Now original mode will detect such problem:

 ...
 [2/7] checking extents
 ERROR: tree block 30457856 has bad backref level, has 256 expect [0, 7]
 ref mismatch on [30457856 16384] extent item 0, found 1
 tree backref 30457856 root 5 not found in extent tree
 backpointer mismatch on [30457856 16384]
 ERROR: errors found in extent allocation tree or chunk allocation
 [3/7] checking free space tree
 ...

Reported-by: Stickstoff <stickstoff@posteo.de>
Link: https://lore.kernel.org/linux-btrfs/6ed4cd5a-7430-f326-4056-25ae7eb44416@posteo.de/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-02-01 18:42:01 +01:00
Qu Wenruo
bf21c82675 btrfs-progs: check: lowmem: fix crash when METADATA_ITEM has invalid level
[BUG]
When running lowmem mode with METADATA_ITEM which has invalid level, it
will crash with the following backtrace:

 (gdb) bt
 #0  0x0000555555616b0b in btrfs_header_bytenr (eb=0x4)
     at ./kernel-shared/ctree.h:2134
 #1  0x0000555555620c78 in check_tree_block_backref (root_id=5,
     bytenr=30457856, level=256) at check/mode-lowmem.c:3818
 #2  0x0000555555621f6c in check_extent_item (path=0x7fffffffd9c0)
     at check/mode-lowmem.c:4334
 #3  0x00005555556235a5 in check_leaf_items (root=0x555555688e10,
     path=0x7fffffffd9c0, nrefs=0x7fffffffda30, account_bytes=1)
     at check/mode-lowmem.c:4835
 #4  0x0000555555623c6d in walk_down_tree (root=0x555555688e10,
     path=0x7fffffffd9c0, level=0x7fffffffd984, nrefs=0x7fffffffda30,
     check_all=1) at check/mode-lowmem.c:4967
 #5  0x000055555562494f in check_btrfs_root (root=0x555555688e10, check_all=1)
     at check/mode-lowmem.c:5266
 #6  0x00005555556254ee in check_chunks_and_extents_lowmem ()
     at check/mode-lowmem.c:5556
 #7  0x00005555555f0b82 in do_check_chunks_and_extents () at check/main.c:9114
 #8  0x00005555555f50ea in cmd_check (cmd=0x55555567c640 <cmd_struct_check>,
     argc=3, argv=0x7fffffffdec0) at check/main.c:10892
 #9  0x000055555556b2b1 in cmd_execute (argv=0x7fffffffdec0, argc=3,
     cmd=0x55555567c640 <cmd_struct_check>) at cmds/commands.h:125

[CAUSE]
For function check_extent_item() it will go through inline extent items
and then check their backrefs.

But for METADATA_ITEM, it doesn't really validate key.offset, which is
u64 and can contain value way larger than BTRFS_MAX_LEVEL (mostly caused
by bit flip).

In that case, if we have a larger value like 256 in key.offset, then
later check_tree_block_backref() will use 256 as level, and overflow
path->nodes[level] and crash.

[FIX]
Just verify the level, no matter if it's from btrfs_tree_block_level()
(which is just u8), or it's from key.offset (which is u64).

To do the check properly and detect higher bits corruption, also change
the type of @level from u8 to u64.

Now lowmem mode can detect the problem properly:

 ...
 [2/7] checking extents
 ERROR: tree block 30457856 has bad backref level, has 256 expect [0, 7]
 ERROR: extent[30457856 16384] level mismatch, wanted: 0, have: 256
 ERROR: errors found in extent allocation tree or chunk allocation
 [3/7] checking free space tree
 ...

Reviewed-by: Su Yue <l@damenly.su>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-02-01 18:41:58 +01:00
Qu Wenruo
ec36c043ba btrfs-progs: check: handle csum generation properly for --init-csum-tree --init-extent-tree
When using `--init-csum-tree` with `--init-extent-tree`, csum tree
population will be done by iterating all file extent items.

This allow us to skip preallocated extents, but it still has the
following problems:

- Inodes with NODATASUM

- Hole file extents

- Written preallocated extents
  We will generate csum for the whole extent, while other part may still
  be unwritten.

Make it to follow the same behavior of recently introduced
fill_csum_for_file_extent(), so we can generate correct csum.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-02-01 18:41:16 +01:00
Qu Wenruo
9e349ff18c btrfs-progs: check: don't calculate csum for preallocated file extents
[BUG]
If a btrfs filesystem has preallocated file extents, 'btrfs check
--init-csum-tree' will create csum item for preallocated extents, and
cause error:

  # mkfs.btrfs -f test.img
  # mount test.img /mnt/btrfs
  # fallocate -l 32K /mnt/btrfs/file
  # umount /mnt/btrfs
  # btrfs check --init-csum-tree --force test.img
  ...
  [4/7] checking fs roots
  root 5 inode 257 errors 800, odd csum item
  ERROR: errors found in fs roots
  found 376832 bytes used, error(s) found

And the csum tree is not empty, containing csum for the preallocated
extent:

  $ btrfs ins dump-tree -t csum test.img
  btrfs-progs v5.15.1
  checksum tree key (CSUM_TREE ROOT_ITEM 0)
  leaf 30408704 items 1 free space 16226 generation 9 owner CSUM_TREE
  leaf 30408704 flags 0x1(WRITTEN) backref revision 1
  fs uuid ecc79835-5611-4609-b985-e4ccd6f15b54
  chunk uuid b1c75553-5b82-4aa6-bbbe-e7f50643b1a8
  	item 0 key (EXTENT_CSUM EXTENT_CSUM 13631488) itemoff 16251 itemsize 32
  		range start 13631488 end 13664256 length 32768

[CAUSE]
For `--init-csum-tree` alone, we will use extent tree to iterate each
data extent, and calculate csum for them.

But extent items alone can not tell us if the file extent belongs to a
NODATASUM inode, nor if it's preallocated.

Thus we create csums for those data extents, and cause the problem.

[FIX]
However the fix is not that simple, we can not just generate csum for
non-preallocated range.

As the following case we still need csum for the un-referred part:

  xfs_io -f -c "pwrite 0 8K" -c "sync" -c "punch 0 4K"

So here we have to go another direction by:

- Always generate csum for the whole data extent
  This is the same as the old code

- Iterate the file extents, and delete csum for preallocated range
  or NODATASUM inodes

Issue: #430
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-02-01 18:41:15 +01:00
Qu Wenruo
f8bc08cb75 btrfs-progs: check: move csum tree population into mode-common.[ch]
This part has no mode specific operations, just move them into
mode-common.[ch].

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-02-01 18:41:13 +01:00
Sidong Yang
7f94ccb20a btrfs-progs: check: change commit condition in fixup_extent_refs()
This patch fixes potential bugs in fixup_extent_refs(). If
btrfs_start_transaction() fails in some way and returns error ptr, It
goes to out logic. But old code checkes whether it is null and it calls
commit. This patch solves the problem with make that it calls only if
ret is no error.

Issue: #409
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:16:22 +01:00
Josef Bacik
e68e73318f btrfs-progs: check: fill csum root from all extent roots
We may have multiple extent roots, so cycle through all of the extent
roots and populate the csum tree based on the content of every extent
root we have in the file system.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:59:34 +01:00
Josef Bacik
bb575e3818 btrfs-progs: check: check all of the csum roots
Use the global roots tree to find all of the csum roots in the system
and check all of them as appropriate.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:59:20 +01:00
Josef Bacik
d26c0b4414 btrfs-progs: check: check the global roots for uptodate root nodes
Instead of checking the csum and extent tree individually, walk through
the global roots and validate them all.  This will work properly if we
have extent tree v1 or extent tree v2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:45 +01:00
Josef Bacik
bf0ce5cb86 btrfs-progs: check: make reinit work per found root item
Instead of looking for the first extent root or csum root in memory,
scan through the tree root and re-init any root items that match the
given objectid.  This will allow reinit to work with both extent tree v1
and extent tree v2 global roots when using the --reinit option.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:43 +01:00
Josef Bacik
db2ab47823 btrfs-progs: stop accessing ->extent_root directly
When we switch to multiple global trees we'll need to access the
appropriate extent root depending on the block group or possibly root.
To handle this, use a helper in most places and then the actual root in
places where it is required.  We will whittle down the direct accessors
with future patches, but this does the bulk of the preparatory work.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:56:54 +01:00
Josef Bacik
d67bf1b365 btrfs-progs: check: move btrfs_mark_used_tree_blocks to common
This is going to be used for the extent tree v2 stuff more commonly, so
move it out so that it is accessible from everywhere that we need it.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-22 21:45:37 +01:00
Josef Bacik
17f6374cfa btrfs-progs: check: abstract out the used marking helpers
We will walk all referenced tree blocks during check in order to avoid
writing over any referenced blocks during fsck.  However in the future
we're going to need to do this for things like fixing block group
accounting with extent tree v2.  This is because extent tree v2 will not
refer to all of the allocated blocks in the extent tree.  Refactor the
code some to allow us to send down an arbitrary extent_io_tree so we can
use this helper for any case where we need to figure out where all the
used space is on an extent tree v2 file system.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-22 21:45:37 +01:00
Josef Bacik
639b1fc2e7 btrfs-progs: stop accessing ->csum_root directly
With extent tree v2 we will have per-block group checksums, so add a
helper to access the csum root and rename the fs_info csum_root to
_csum_root to catch all the places that are accessing it directly.
Convert everybody to use the helper except for internal things.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-22 21:45:37 +01:00