Commit Graph

131 Commits

Author SHA1 Message Date
David Sterba a81a3d771b btrfs-progs: image: factor out the restore part from main.c
The remaining part of restore functionality starting from
restore_metadump() has been factored out. Same incremental set of
changes so the diff is not clean.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:25 +02:00
David Sterba add18714b8 btrfs-progs: image: factor out the create part from main.c
The functionality of create and restore is all in main.c, split the
create functionality first. This is not the cleaniest diff to do it, the
functions are entangled and the final result is from several compile and
edit cycles.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
David Sterba 00df7245f4 btrfs-progs: image: move defintions from main.c to header
Signed-off-by: David Sterba <dsterba@suse.com>
2023-08-28 17:24:24 +02:00
Anand Jain d46a0ef6a0 btrfs-progs: rename struct open_ctree_flags to open_ctree_args
The struct open_ctree_flags currently holds arguments for
open_ctree_fs_info(), it can be confusing when mixed with a local variable
named open_ctree_flags as below in the function cmd_inspect_dump_tree().

  cmd_inspect_dump_tree()
  ::
  struct open_ctree_flags ocf = { 0 };
  ::
  unsigned open_ctree_flags;

So rename struct open_ctree_flags to struct open_ctree_args.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-07-26 15:00:47 +02:00
David Sterba ed9339b403 btrfs-progs: image: convert int to bool in a few helpers
Signed-off-by: David Sterba <dsterba@suse.com>
2023-06-08 00:08:35 +02:00
Josef Bacik e150c843ea btrfs-progs: sync tree-checker.[ch] from kernel
This syncs tree-checker.c from the kernel.  The main modification was to
add a open ctree flag to skip the deeper leaf checks, and plumbing this
through tree-checker.c.  We need this for things like fsck or
btrfs-image that need to work with slightly corrupted file systems, and
these checks simply make us unable to look at the corrupted blocks.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik b3477244f9 btrfs-progs: update read_tree_block to match the kernel definition
The in-kernel version of read_tree_block adds some extra sanity checks
to make sure we don't return blocks that don't match what we expect.
This includes the owning root, the level, and the expected first key.
We don't actually do these checks in btrfs-progs, however kernel code
we're going to sync will expect this calling convention, so update it to
match the in-kernel code and then update all the callers.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:30 +02:00
Josef Bacik 4a9a8f2a8a btrfs-progs: sync extent-io-tree.[ch] and misc.h from the kernel
This is a bit larger than the previous syncs, because we use
extent_io_tree's everywhere.  There's a lot of stuff added to
kerncompat.h, and then I went through and cleaned up all the API
changes, which were

- extent_io_tree_init takes an fs_info and an owner now.
- extent_io_tree_cleanup is now extent_io_tree_release.
- set_extent_dirty takes a gfpmask.
- clear_extent_dirty takes a cached_state.
- find_first_extent_bit takes a cached_state.

The diffstat looks insane for this, but keep in mind extent-io-tree.c
and extent-io-tree.h are ~2000 loc just by themselves.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik f8efe9f724 btrfs-progs: sync file-item.h into progs
This patch syncs file-item.h into btrfs-progs.  This carries with it an
API change for btrfs_del_csums, which takes a root argument in the
kernel, so all callsites have been updated accordingly.

I didn't sync file-item.c because it carries with it a bunch of bio
related helpers which are difficult to adapt to the kernel.
Additionally there's a few helpers in the local copy of file-item.c that
aren't in the kernel that are required for different tools.

This requires more cleanups in both the kernel and progs in order to
sync file-item.c, so for now just do file-item.h in order to pull things
out of ctree.h.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
Josef Bacik a754fe29d9 btrfs-progs: sync uapi/btrfs.h into btrfs-progs
We want to keep this file locally as we want to be uptodate with
upstream, so we can build btrfs-progs regardless of which kernel is
currently installed.  Sync this with the upstream version and put it in
kernel-shared/uapi to maintain some semblance of where this file comes
from.

There are some changes that need to be synced back to kernel. A local
definition of static_assert is used to avoid compilation problems on gcc
(< 9) due to mandatory 2nd parameter.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
Josef Bacik bf0f3db765 btrfs-progs: introduce UASSERT() for purely userspace code
While syncing messages.[ch] I had to back out the ASSERT() code in
kerncompat.h, which means we now rely on the kernel code for ASSERT().
In order to maintain some semblance of separation introduce UASSERT()
and use that in all the purely userspace code.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:28 +02:00
psykose c9abbf6264 btrfs-progs: stop using legacy *64 interfaces
The *64 interfaces, such as fstat64, off64_t, etc, are legacy interfaces
created at a time when 64-bit file support was still new. They are
generally exposed when defining a macro named _LARGEFILE64_SOURCE, as
e.g. the glibc docs[0] say.

The modern way to utilise largefile support, is to continue to use the
regular interfaces (off_t, fstat, ..), and define _FILE_OFFSET_BITS=64.

We already use the autoconf macro AC_SYS_LARGEFILE[1] which arranges this
and sets this macro for us. Therefore, we can utilise the non-64 names
without fear of breaking on 32-bit systems.

This fixes the build against musl libc, ever since musl dropped the
*64 compat from interfaces by default[2] just for _GNU_SOURCE, unless
_LARGEFILE64_SOURCE is defined. However, there are plans for a future
removal of the whole *64 header API, and that workaround (adding another
define) might cease to exist.

So, rename all *64 API use to the regular non-suffixed names. For
consistency, rename the internal functions that were *64 named
(lstat64_path, ..) too.

This should have no regressions on any platform.

[0]: https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html#index-_005fLARGEFILE64_005fSOURCE
[1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/System-Services.html
[2]: 25e6fee27f

Pull-request: #615
Signed-off-by: psykose <alice@ayaya.dev>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-04-25 16:59:42 +02:00
David Sterba a7fa81f296 btrfs-progs: open code print_usage where applicable
After previous change to usage() that now has the return code, there's
no purpose of the print_usage() wrapper so it can be removed.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-28 20:11:23 +01:00
Qu Wenruo f61b90aff9 btrfs-progs: make usage call properly return an exit value
[BUG]
Currently cli/009 test case failed with different exit number:

  ====== RUN CHECK /home/adam/btrfs-progs/btrfstune --help
  usage: btrfstune [options] device
  [...]
  failed: /home/adam/btrfs-progs/btrfstune --help
  test failed for case 009-btrfstune

[CAUSE]
In tune/main.c, we have the following call on usage():

  static void print_usage(int ret)
  {
	usage(&tune_cmd);
	exit(ret);
  }

However usage() itself would always call exit(1):

  void usage(const struct cmd_struct *cmd)
  {
	usage_command_usagestr(cmd->usagestr, NULL, 0, true, true);
	exit(1);
  }

This makes prevents any caller of usage() to modify its exit number.

[FIX]
Add a new argument @error for print_usage(), so we can properly return 0
for -h/--help usage.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-28 20:11:23 +01:00
David Sterba 24ec095295 btrfs-progs: crypto: add common function for accelerated initialization
Prepare a single location that will detect or set accelerated versions
of hash algorithms. Right now it's the crc32c, blake2 and sha256 do
an if-else switch while crc32c sets a function pointer.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-28 19:49:31 +01:00
Qu Wenruo f914949b1a btrfs-progs: fix set but not used variables
[WARNING]
Clang 15.0.7 warns about several unused variables:

  kernel-shared/zoned.c:829:6: warning: variable 'num_sequential' set but not used [-Wunused-but-set-variable]
          u32 num_sequential = 0, num_conventional = 0;
              ^
  cmds/scrub.c:1174:6: warning: variable 'n_skip' set but not used [-Wunused-but-set-variable]
          int n_skip = 0;
              ^
  mkfs/main.c:493:6: warning: variable 'total_block_count' set but not used [-Wunused-but-set-variable]
          u64 total_block_count = 0;
              ^
  image/main.c:2246:6: warning: variable 'bytenr' set but not used [-Wunused-but-set-variable]
          u64 bytenr = 0;
              ^

[CAUSE]
Most of them are just straightforward set but not used variables.

The only exception is total_block_count, which has commented out code
relying on it.

[FIX]
Just remove those variables, and for @total_block_count, also remove the
comments.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-18 17:44:03 +01:00
Qu Wenruo c3a41e5d4c btrfs-progs: move a union with variable sized type to the end
[WARNING]
Clang 15.0.7 gives the following warning:

  image/main.c:95:2: warning: field '' with variable sized type 'union metadump_struct::(anonymous at image/main.c:95:2)' not at the end of a struct or class is a GNU extension [-Wgnu-variable-sized-type-not-at-end]
          union {
          ^

[CAUSE]
The union contains meta_cluster, which variable sized:

  struct meta_cluster {
  	struct meta_cluster_header header;
  	struct meta_cluster_item items[];
  } __attribute__ ((__packed__));

Thus clang gives above warning since it's a GNU extension.

[FIX]
Just move the union to the end of the structure.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-18 17:44:02 +01:00
Qu Wenruo 3a1d4aa089 btrfs-progs: fix fallthrough cases with proper attributes
[FALSE ALERT]
Unlike gcc, clang doesn't really understand the comments, thus it's
reportings tons of fall through related errors:

  cmds/reflink.c:124:3: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
                  case 'r':
                  ^
  cmds/reflink.c:124:3: note: insert '__attribute__((fallthrough));' to silence this warning
                  case 'r':
                  ^
                  __attribute__((fallthrough));
  cmds/reflink.c:124:3: note: insert 'break;' to avoid fall-through
                  case 'r':
                  ^
                  break;

[CAUSE]
Although gcc is fine with /* fallthrough */ comments, clang is not.

[FIX]
So just introduce a fallthrough macro to handle the situation properly,
and use that macro instead.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-18 17:44:02 +01:00
David Sterba 8affa4e116 btrfs-progs: image: use help and cmd_struct for printing help text
Unify the image help text so it uses the help framework. The cmd struct
is set up only partially.

Signed-off-by: David Sterba <dsterba@suse.com>
2023-02-18 17:44:02 +01:00
Josef Bacik 405b5cadd3 btrfs-progs: replace btrfs_leaf_data with btrfs_item_nr_offset
We're using btrfs_item_nr_offset(leaf, 0) to get the start of the leaf
data in the kernel, we don't have btrfs_leaf_data.  Replace all
occurrences of btrfs_leaf_data() with btrfs_item_nr_offset(leaf, 0) in
order to make syncing accessors.[ch] easier.  ctree.c will be synced
later, so this is simply an intermediate step.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-30 19:14:29 +01:00
Josef Bacik 27db5d147f btrfs-progs: image: rename BLOCK_* to IMAGE_BLOCK_* for metadump
When we sync the kernel we're going to pull in the fs.h dependency,
which defines BLOCK_SIZE/BLOCK_MASK.  Avoid this conflict by renaming
the image definitions with the IMAGE_ prefix.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-11-28 18:57:43 +01:00
David Sterba f5e07cc60a btrfs-progs: warn when an experimental functionality is used
Print warning when one of the following is requested by some command
line option:

- btrfstune -b: conversion to block-group-tree
- mkfs.btrfs --num-global-roots: extent-tree-v2
- btrfs-image -d: dump image with data

Issue: #523
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-20 16:39:11 +02:00
David Sterba ccb2d4aa45 btrfs-progs: device-utils: rename btrfs_device_size
There's a group of helpers to read device size, the btrfs_device_size
should be one of them. Rename it and so minor cleanup.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba a827bb2db8 btrfs-progs: use template for transaction commit error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba 8fcafae04a btrfs-progs: use template for transaction start error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba c2be0e2ce0 btrfs-progs: use template for out of memory error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba b73a29936a btrfs-progs: remove unnecessary casts for u64
The (unsigned long long) type casts can be dropped, printf understands
%llu and u64 and does not warn. In cases where the type is not u64 keep
the cast.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba 6edd4b2121 btrfs-progs: factor string helpers out of utils.c
Utils is the catch-all file, we can now separate some string utility
functions.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:13 +02:00
David Sterba 6d9b3835a6 btrfs-progs: image: update include lists
The tool IWYU (include what you use) suggests to remove and add some
includes.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:12 +02:00
David Sterba fe7a78f5df btrfs-progs: image: reorder includes
The preferred order:
- system headers
- standard headers
- libraries
- kernel library
- kernel shared
- common headers
- other tools
- own headers

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:11 +02:00
Qu Wenruo fc6925bfd3 btrfs-progs: avoid repeated data write for metadata
[BUG]
Shinichiro reported that "mkfs.btrfs -m DUP" is doing repeated write
into the device.
For non-zoned device this is not a big deal, but for zoned device this
is critical, as zoned device doesn't support overwrite at all.

[CAUSE]
The problem is related to write_and_map_eb() call, since commit
2a93728391 ("btrfs-progs: use write_data_to_disk() to replace
write_extent_to_disk()"), we call write_data_to_disk() for metadata
write back.

But the problem is, write_data_to_disk() will call btrfs_map_block()
with rw = WRITE.

By that btrfs_map_block() will always return all stripes, while in
write_data_to_disk() we also iterate through each mirror of the range.

This results above repeated writeback.

[FIX]
Fix this problem by completely remove @mirror argument
from write_data_to_disk().
With extra comments to explicitly show that function will write to
all mirrors.

Reported-by: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Fixes: 2a93728391 ("btrfs-progs: use write_data_to_disk() to replace write_extent_to_disk()")
Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-08-16 15:18:12 +02:00
Qu Wenruo 3ff9d35257 btrfs-progs: use read_data_from_disk() to replace read_extent_from_disk() and replace read_extent_data()
The function read_extent_from_disk() is only a wrapper to read tree
block.

And read_extent_data() is just a while loop to eliminate short read
caused by stripe boundary.

In fact, a lot of call sites of read_extent_data() are either reading
metadata (thus no possible short read) or doing extra loop by
themselves.

This patch will replace those two functions with read_data_from_disk(),
making it the only entrance for data/metadata read.
And update read_data_from_disk() to return the read bytes, so caller can
do a simple while loop.

For the few callers of read_extent_data(), open-code a small while loop
for them.

This will allow later RAID56 read repair using P/Q much easier.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-04-25 19:08:30 +02:00
Josef Bacik 4f184cc911 btrfs-progs: make all of the item/key_ptr offset helpers take an eb
When we change the size of the btrfs_header we're going to need to
change how these helpers calculate where to find the start of items or
block ptrs.  To prepare for that make these helpers take the
extent_buffer as an argument so we can do the appropriate math based on
the version type.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:14 +01:00
Josef Bacik 5dc3964aaa btrfs-progs: remove the _nr from the item helpers
Now that all callers are using the _nr variations we can simply rename
these helpers to btrfs_item_##member/btrfs_set_item_##member and change
the actual item SETGET funcs to raw_item_##member/set_raw_item_##member
and then change all callers to drop the _nr part.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik 49539423fa btrfs-progs: change btrfs_file_extent_inline_item_len to take a slot
This matches how the kernel does it, simply pass in the slot and fix up
btrfs_file_extent_inline_item_len to use the btrfs_item_nr() helper and
the correct define.  Fixup all the callers to use the slot now instead
of passing in the btrfs_item.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik 04ffea07e4 btrfs-progs: add btrfs_set_item_*_nr() helpers
We have a lot of the following patterns

	item = btrfs_item_nr(nr);
	btrfs_set_item_*(eb, item, val);

	btrfs_set_item_*(eb, btrfs_item_nr(nr), val);

in a lot of places in our code.  Instead add _nr variations of these
helpers and convert all of the users to this new helper.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik 80ff3f2c8e btrfs-progs: btrfs_item_size_nr/btrfs_item_offset_nr everywhere
We have this pattern in a lot of places

	item = btrfs_item_nr(slot);
	btrfs_item_size(leaf, item);
	btrfs_item_offset(leaf, item);

when we could simply use

	btrfs_item_size_nr(leaf, slot);
	btrfs_item_offset_nr(leaf, slot);

Fix all callers of btrfs_item_size() and btrfs_item_offset() to use the
_nr variation of the helpers.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-03-09 15:13:13 +01:00
Josef Bacik db2ab47823 btrfs-progs: stop accessing ->extent_root directly
When we switch to multiple global trees we'll need to access the
appropriate extent root depending on the block group or possibly root.
To handle this, use a helper in most places and then the actual root in
places where it is required.  We will whittle down the direct accessors
with future patches, but this does the bulk of the preparatory work.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:56:54 +01:00
Josef Bacik 29e56c7e05 btrfs-progs: image: keep track of seen blocks when walking trees
Extent tree v2 no longer tracks all allocated blocks on the file system,
so we'll have to default to walking trees to generate metadata images.
There's an annoying drawback with walking trees with btrfs-image where
we'll happily copy multiple blocks over and over again if there are
snapshots.  Fix this by keeping track of blocks we've seen and simply
skipping blocks that we've already queued up for copying.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-22 21:45:37 +01:00
Qu Wenruo 636b2e6027 btrfs-progs: remove temporary buffer for super block
There are a lot of call sites where we use the following code snippet:

	u8 super_block_data[BTRFS_SUPER_INFO_SIZE];
	struct btrfs_super_block *sb;
	u64 ret;

	sb = (struct btrfs_super_block *)super_block_data;

The reason for this is, structure btrfs_super_block was smaller than
BTRFS_SUPER_INFO_SIZE.

Thus for anything with csum involved, we have to use a proper 4K buffer.

Since the recent unification of sizeof(struct btrfs_super_block), we no
longer need such workaround, and can use struct btrfs_super_block
directly to do any operation.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-05 12:50:03 +01:00
Qu Wenruo 5bee5c99bf btrfs-progs: fix printf formats on 32bit x86
When compiling btrfs-progs on 32bit x86 using GCC 11.1.0, there are
several warnings:

  In file included from ./common/utils.h:30,
                   from check/main.c:36:
  check/main.c: In function 'run_next_block':
  ./common/messages.h:42:31: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'u32' {aka 'unsigned int'} [-Wformat=]
     42 |                 __btrfs_error((fmt), ##__VA_ARGS__);                    \
        |                               ^~~~~
  check/main.c:6496:33: note: in expansion of macro 'error'
   6496 |                                 error(
        |                                 ^~~~~

  In file included from ./common/utils.h:30,
                   from kernel-shared/volumes.c:32:
  kernel-shared/volumes.c: In function 'btrfs_check_chunk_valid':
  ./common/messages.h:42:31: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'u32' {aka 'unsigned int'} [-Wformat=]
     42 |                 __btrfs_error((fmt), ##__VA_ARGS__);                    \
        |                               ^~~~~
  kernel-shared/volumes.c:2052:17: note: in expansion of macro 'error'
   2052 |                 error("invalid chunk item size, have %u expect [%zu, %lu)",
        |                 ^~~~~

  image/main.c: In function 'search_for_chunk_blocks':
  ./common/messages.h:42:31: warning: format '%lu' expects argument of type 'long unsigned int', but argument 3 has type 'size_t' {aka 'unsigned int'} [-Wformat=]
     42 |                 __btrfs_error((fmt), ##__VA_ARGS__);                    \
        |                               ^~~~~
  image/main.c:2122:33: note: in expansion of macro 'error'
   2122 |                                 error(
        |                                 ^~~~~

There are two types of problems:

- __BTRFS_LEAF_DATA_SIZE()
  This macro has no type definition, making it behaves differently on
  different arches.

  Fix this by following kernel to use inline function to make its return
  value fixed to u32.

- size_t related output
  For x86_64 %lu is OK but not for x86.

  Fix this by using %zu.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-05 12:50:03 +01:00
Qu Wenruo 60651ad9da btrfs-progs: introduce OPEN_CTREE_ALLOW_TRANSID_MISMATCH flag
[BUG]
There is a report that, btrfstune can even work while the fs has transid
mismatch problems.

  $ btrfstune -f -u /dev/sdb1
  Current fsid: b2b5ae8d-4c49-45f0-b42e-46fe7dcfcb07
  New fsid: b2b5ae8d-4c49-45f0-b42e-46fe7dcfcb07
  Set superblock flag CHANGING_FSID
  Change fsid in extents
  parent transid verify failed on 792854528 wanted 20103 found 20091
  parent transid verify failed on 792854528 wanted 20103 found 20091
  parent transid verify failed on 792854528 wanted 20103 found 20091
  Ignoring transid failure
  parent transid verify failed on 792870912 wanted 20103 found 20091
  parent transid verify failed on 792870912 wanted 20103 found 20091
  parent transid verify failed on 792870912 wanted 20103 found 20091
  Ignoring transid failure
  parent transid verify failed on 792887296 wanted 20103 found 20091
  parent transid verify failed on 792887296 wanted 20103 found 20091
  parent transid verify failed on 792887296 wanted 20103 found 20091
  Ignoring transid failure
  ERROR: child eb corrupted: parent bytenr=38010880 item=69 parent level=1 child level=1
  ERROR: failed to change UUID of metadata: -5
  ERROR: btrfstune failed

This leaves a corrupted fs even more corrupted, and due to the extra
CHANGING_FSID flag, btrfs check will not even try to run on it:

  Opening filesystem to check...
  ERROR: Filesystem UUID change in progress
  ERROR: cannot open file system

[CAUSE]
Unlike kernel, btrfs-progs has a less strict check on transid mismatch.

In read_tree_block() we will fall back to use the tree block even its
transid mismatch if we can't find any better copy.

However not all commands in btrfs-progs needs this feature, only
btrfs-check (which may fix the problem) and btrfs-restore (it just tries
to ignore any problems) really utilize this feature.

[FIX]
Introduce a new open ctree flag, OPEN_CTREE_ALLOW_TRANSID_MISMATCH, to
be explicit about whether we really want to ignore transid error.

Currently only btrfs-check and btrfs-restore will utilize this new flag.

Also add btrfs-image to allow opening such fs with transid error.

Link: https://www.reddit.com/r/btrfs/comments/pivpqk/failure_during_btrfstune_u/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-09-20 12:17:29 +02:00
Qu Wenruo 3875c14a5a btrfs-progs: image: fix restored image size misalignment
[BUG]
There is a small device size misalignment between the super block device
size and the device extent size:

  total_bytes             10737418240 	<<<
  bytes_used              15097856
  dev_item.total_bytes    10737418240
  dev_item.bytes_used     1094713344

        item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
                devid 1 total_bytes 1095761920 bytes_used 1094713344
				    ^^^^^^^^^^

[CAUSE]
In fixup_device_size(), we only reset superblock device item size, which
will be overwritten in write_dev_supers() using btrfs_device::total_bytes.

And it doesn't touch btrfs_superblock::total_bytes either.

[FIX]
So fix the small mismatch by also resetting btrfs_device::total_bytes,
btrfs_device::bytes_used and btrfs_superblock::total_bytes.

Thankfully since commit 73dd4e3c87 ("btrfs-progs: image: Don't modify
the chunk and device tree if the source dump is single device") single
device dump won't have such problem, but it's still worth for
multi-device dump.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-08-25 15:38:53 +02:00
Qu Wenruo c4ff83cb5d btrfs-progs: image: reduce memory requirements for decompression
With recent change to enlarge max_pending_size to 256M for data dump,
the decompress code requires quite a lot of memory, up to 256M * 4.

The reason is we're using wrapped uncompress() function call, which
needs the buffer to be large enough to contain the decompressed data.

This patch will re-work the decompress work to use inflate() which can
resume it decompression so that we can use a much smaller buffer size.

Use 512K as buffer size.

Now the memory consumption for restore is reduced to

 cluster data size + 512K * nr_running_threads

Instead of the original one:

 cluster data size + 1G * nr_running_threads

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-08-25 15:38:53 +02:00
Qu Wenruo 0cc8a7bd4a btrfs-progs: image: introduce -d option to dump data
This new experimental data dump feature will dump the whole image, not
only the existing tree blocks but also all its data extents(*).

This feature will rely on the new dump format (_DUmP_v1), as it needs
extra large extent size limit, and older btrfs-image dump can't handle
such large item/cluster size.

Since we're dumping all extents including data extents, for the restored
image there is no need to use any extra super block flags to inform
kernel.
Kernel should just treat the restored image as any ordinary btrfs.

This new feature will be hidden behind the experimental features, that's
to say, if --enable-experimental is not enabled, although we still have
the option, it will not do anything but output an error message.

*: The data extents will be dumped as is, that's to say, even for
preallocated extent, its (meaningless) data will be read out and
dumpped.
This behavior will cause extra space usage for the image, but we can
skip all the complex partially shared preallocated extent check.

Issue: #394
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-08-25 15:38:53 +02:00
Qu Wenruo 6b03e0dc40 btrfs-progs: image: introduce framework for more dump versions
The original dump format only contains a magic member to verify the
format, this means if we want to introduce new on-disk format or change
certain size limit, we can only introduce new magic as version.

Introduce the framework to allow multiple magic numbers to co-exist for
further extensions.

Introduce the following members for each dump version.

- max_pending_size
  The threshold size of a cluster. It's not a hard limit but a soft
  one. One cluster can go larger than max_pending_size for one item, but
  next item would go to the next cluster.

- magic_cpu
  The magic number in CPU byte order.

- extra_sb_flags
  If the super block of this restore needs extra super block flags like
  BTRFS_SUPER_FLAG_METADUMP_V2.
  For incoming data dump feature, we don't need any extra super block
  flags.

This change also implies that all image dumps will use the same magic
for all clusters. No mixing is allowed, as we will use the first cluster
to determine the dump version.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-08-25 15:38:53 +02:00
Qu Wenruo e916d57466 btrfs-progs: image: enlarge output file if no tree modification is needed for restore
[BUG]
If restoring dumped image to a new file, under most cases kernel will
reject it since version 5.11:

 # mkfs.btrfs -f /dev/test/test
 # btrfs-image /dev/test/test /tmp/dump
 # btrfs-image -r /tmp/dump ~/test.img
 # mount ~/test.img /mnt/btrfs
 mount: /mnt/btrfs: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
 # dmesg -t | tail -n 7
 loop0: detected capacity change from 10592 to 0
 BTRFS info (device loop0): disk space caching is enabled
 BTRFS info (device loop0): has skinny extents
 BTRFS info (device loop0): flagging fs with big metadata feature
 BTRFS error (device loop0): device total_bytes should be at most 5423104 but found 10737418240
 BTRFS error (device loop0): failed to read chunk tree: -22
 BTRFS error (device loop0): open_ctree failed

[CAUSE]
When btrfs-image restores an image into a file, and the source image
contains only single device, then we don't need to modify the
chunk/device tree, as we can reuse the existing chunk/dev tree without
any problem.

This also means, for such restore, we also won't do any target file
enlarge. This behavior itself is fine, as at that time, kernel won't
check if the device is smaller than the device size recorded in device
tree.

But later kernel commit 3a160a933111 ("btrfs: drop never met disk total
bytes check in verify_one_dev_extent") introduces new check on device
size at mount time, rejecting any loop file which is smaller than the
original device size.

[FIX]
Do extra file enlarge for single device restore if the restored file is
smaller than the device size.

Reported-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Su Yue <l@damenly.su>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-05-06 16:41:47 +02:00
Qu Wenruo 9900c7a6bc btrfs-progs: image: remove the dead stat() call in metadump
In restore_metadump(), we call stat() but never use the result.  This
call site is left by some code refactoring, as the stat() call is now
moved into fixup_device_size().  We can safely remove the call.

Reviewed-by: Su Yue <l@damenly.su>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-05-06 16:41:47 +02:00
David Sterba 7fa07e2abb btrfs-progs: split open/close helpers from utils.c
There's a group of functions that are related to opening filesystem in
various modes, this can be moved to a separate file.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-05-06 16:41:47 +02:00
David Sterba 6bb6d1215d btrfs-progs: factor open_ctree parameters to a structure
Extending open_ctree with more parameters would be difficult, we'll need
to add more so factor out the parameters to a structure for easier
extension.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-03-24 22:20:19 +01:00