Commit Graph

4893 Commits

Author SHA1 Message Date
Long An
bb64c02b90 btrfs-progs: tests: fix path of btrfs-corrupt-block
There are two utilities that are required for the testsuite but not
installed on systems. The path in tests must be INTERNAL_BIN.

Fix this issue for below tests:
 * fsck-tests/013-extent-tree-rebuild
 * fsck-tests/037-freespacetree-repair
 * misc-tests/038-backup-root-corruption

Issue: #198
Signed-off-by: Long An <lan@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-03-03 19:58:53 +01:00
Nikolay Borisov
82ec6dbd4d btrfs-progs: tests: Extend metadata uuid testcase
This adds cooked images to exercise the case when a filesystem with
metadata uuid incompat flag is switched back to having fsid/metadata
uuid being equal.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-03-03 19:58:53 +01:00
Zygo Blaxell
3c5254b187 btrfs-progs: inspect-internal: document new logical-resolve options and kernel requirements
Document the new options requiring the V2 ioctl and the increased
default buffer size.

Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-03-03 19:58:53 +01:00
Zygo Blaxell
2cbddaba38 btrfs-progs: inspect: increase logical-resolve default buffer size to 64K
Filesystems with nontrivial snapshots or dedupe will easily overflow
a 4K buffer.  Bump the size up to the largest size supported by the
V1 ioctl.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-03-03 19:58:41 +01:00
Zygo Blaxell
ff3843b87d btrfs-progs: inspect: add support for LOGICAL_INO_V2 ioctl
Increase the maximum buffer size to SZ_16M.

Add an option (-o) to set the ..._IGNORE_OFFSET flag.

If the buffer size is greater than 64K or the IGNORE_OFFSET option
is used, call ioctl V2; otherwise, use ioctl V1 to be compatible with
older kernels.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 15:06:38 +01:00
Zygo Blaxell
6ccf24c7ef btrfs-progs: add LOGICAL_INO_V2 to ioctl.h
Update the args structure, add the flags constant and the ioctl magic
number.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 15:06:37 +01:00
Zygo Blaxell
e6ea9ed1f1 libbtrfsutil: add LOGICAL_INO_V2
Update the args structure, add the flags constant and the ioctl magic
number.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 15:06:37 +01:00
Zygo Blaxell
8987681299 btrfs-progs: ioctl-test: add LOGICAL_INO_V2
Same as LOGICAL_INO, except a different magic number.

Reviewed-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 15:06:37 +01:00
David Sterba
3fc2326d34
Btrfs progs v5.4.1
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:36:18 +01:00
David Sterba
3a8e6a78ff btrfs-progs: update CHANGES for 5.4.1
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:34:17 +01:00
A. Wilcox
2a064d77a7 btrfs-progs: docs: Don't erase XMLTO_EXTRA contents, fix docbook5 build
XMLTO_EXTRA is defined above, and then appended to if `asciidoctor` is
used to generate the manual pages.  The second definition was clearing
the value of this variable, which caused builds with asciidoctor to
fail with:

```
    [XMLTO]  btrfs-ioctl.3
xmlto: /usr/src/packages/user/btrfs-progs/src/btrfs-progs-v5.4/Documentation/btrfs-ioctl.xml does not validate (status 3)
xmlto: Fix document syntax or use --skip-validation option
validity error : no DTD found!
Document /usr/src/packages/user/btrfs-progs/src/btrfs-progs-v5.4/Documentation/btrfs-ioctl.xml does not validate
make[1]: *** [Makefile:114: btrfs-ioctl.3] Error 13
```

Fixes: 669f56177 ("btrfs-progs: docs: use docbook5 backend for asciidoctor")
Signed-off-by: A. Wilcox <AWilcox@Wilcox-Tech.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:28:52 +01:00
Qu Wenruo
d3f7315b3c btrfs-progs: tests: check: detect bad extent item generation
The new image has a bad data extent item generation:

        item 0 key (13631488 EXTENT_ITEM 4096) itemoff 16230 itemsize 53
                refs 1 gen 16384 flags DATA
                extent data backref root FS_TREE objectid 257 offset 0 count 1

Just like what older `btrfs check --init-extent-tree` could cause.

Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
b3621da146 btrfs-progs: check/original: Detect invalid extent generation
Much like what we have done in lowmem mode, also detect and report
invalid extent generation in original mode.

Unlike lowmem mode, we have extent_record::generation, which is the
higher number of generations in EXTENT_ITEM, EXTENT_DATA or tree block
header, so there is no need to check generations in different locations.

For repair, we still need to depend on --init-extent-tree, as directly
modifying extent items can easily cause conflicts with delayed refs,
thus it should be avoided.

Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
918ba847bc btrfs-progs: check/lowmem: Detect invalid EXTENT_ITEM and EXTENT_DATA generation
Since older `btrfs check --init-extent-tree` could cause invalid
EXTENT_ITEM generation for data extents, add such check to lowmem mode
check.

Also add such generation check to file extents too.

For the repair part, I don't have any good idea yet. So affected user
may depend on --init-extent-tree again.

Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
8d45dc270a btrfs-progs: check: Populate extent generation correctly for data extents
[BUG]
When using `btrfs check --init-extent-tree`, we will create incorrect
generation number for data extents in extent tree:

        item 10 key (13631488 EXTENT_ITEM 1048576) itemoff 15828 itemsize 53
                refs 1 gen 0 flags DATA
                extent data backref root FS_TREE objectid 257 offset 0 count 1

[CAUSE]
Since data extent generation is not as obvious as tree blocks, which has
header containing its generations, so for data extents, its
extent_record::generation is not really updated, resulting such 0
generation.

[FIX]
To get generation of a data extent, there are two sources we can rely:
- EXTENT_ITEM
  There is always a btrfs_extent_item::generation can be utilized.
  Although this is not possible for --init-extent-tree use case.

- EXTENT_DATA
  We have btrfs_file_extent_item::generation for regular and
  preallocated data extents.
  Since --init-extent-tree will go through subvolume trees, this would
  be the main source for extent data generation.

Then we only need to make add_data_backref() to accept @gen parameter,
and pass it down to extent_record structure.

And for the final extent item generation update, here we add extra
fallback values, if we can't find FILE_EXTENT items.
In that case, we just fall back to current transid.

With this modification, recreated data EXTENT_ITEM now has correct
generation number:

        item 10 key (13631488 EXTENT_ITEM 1048576) itemoff 15828 itemsize 53
                refs 1 gen 6 flags DATA
                extent data backref root FS_TREE objectid 257 offset 0 count 1

Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
9a85b7db02 btrfs-progs: check: Initialize extent_record::generation member
[BUG]
When using `btrfs check --init-extent-tree`, there is a pretty high
chance that the result fs can't pass tree-checker:

  BTRFS critical (device dm-3): corrupt leaf: block=5390336 slot=149 extent bytenr=20115456 len=4096 invalid generation, have 16384 expect (0, 360]
  BTRFS error (device dm-3): block=5390336 read time tree block corruption detected
  BTRFS error (device dm-3): failed to read block groups: -5
  BTRFS error (device dm-3): open_ctree failed

[CAUSE]
The result fs has a pretty screwed up EXTENT_ITEMs for data extents:

        item 148 key (20111360 EXTENT_ITEM 4096) itemoff 8777 itemsize 53
                refs 1 gen 0 flags DATA
                extent data backref root FS_TREE objectid 841 offset 0 count 1
        item 149 key (20115456 EXTENT_ITEM 4096) itemoff 8724 itemsize 53
                refs 1 gen 16384 flags DATA
                extent data backref root FS_TREE objectid 906 offset 0 count 1

Kernel tree-checker will accept 0 generation, but that 16384 generation
is definitely going to trigger the alarm.

Looking into the code, it's add_extent_rec_nolookup() allocating a new
extent_rec, but not copying all members from parameter @tmpl, resulting
generation not properly initialized.

[FIX]
Just copy tmpl->generation in add_extent_rec_nolookup(). And since all
call sites have set all members of @tmpl to 0 before
add_extent_rec_nolookup(), we shouldn't get garbage values.

For the 0 generation problem, it will be solved in another patch.

Issue: #225 (Not the initial report, but extent tree rebuild result)
Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Marcos Paulo de Souza
6038a5686c btrfs-progs: qgroup: Check for ENOTCONN error on create/assign/limit
When kernel returns ENOTCONN after the user tries to create a qgroup on
a subvolume without quota enabled, present a meaningful message to the
user. Kernels before 5.5 return EINVAL for that.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Su Yue
f0c607a137 btrfs-progs: misc-tests/038: fix wrong field filtered under root directory
Ran misc-tests/038 in /root/btrfs-progs:

  make test-misc TEST=038\*
      [TEST]   misc-tests.sh
      [TEST/misc]   038-backup-root-corruption
  ./test.sh: line 33: [: bytenr=65536,: integer expression expected
  Backup slot 2 is not in use
  test failed for case 038-backup-root-corruption
  make: *** [Makefile:401: test-misc] Error 1

It's caused by the wrong line filtered by
$(dump_super | grep root | head -n1 | awk '{print $2}').

The command $(dump-super | grep root) outputs:

  superblock: bytenr=65536, device=/root/btrfs-progs/tests/test.img
  root                    30605312
  chunk_root_generation   5
  root_level              0
  chunk_root              22036480
  chunk_root_level        0
  log_root                0
  log_root_transid        0
  log_root_level          0
  root_dir                6
  backup_roots[4]:

Here
"superblock: bytenr=65536, device=/root/btrfs-progs/tests/test.img" is
selected but not the line "root                    30605312".

Restricting the awk rule can fix it.

Fixes: 78a3831d46 ("btrfs-progs: tests: Test backup root retention logic")
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
132ccd98a8 btrfs-progs: disk-io: Remove duplicated ASSERT() call
There are two ASSERT() with completely the same check introduced in
commit f7717d8cdb ("btrfs-progs: Remove fsid/metdata_uuid fields
from fs_info").

Just remove it.

Fixes: f7717d8cdb ("btrfs-progs: Remove fsid/metdata_uuid fields from fs_info")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
e073b6e14f btrfs-progs: fix superblock range exclusion check
[BUG]
For certain btrfs images, a BUG_ON() can be triggered at open_ctree()
time:

  Opening filesystem to check...
  extent_io.c:158: insert_state: BUG_ON `end < start` triggered, value 1
  btrfs(+0x2de57)[0x560c4d7cfe57]
  btrfs(+0x2e210)[0x560c4d7d0210]
  btrfs(set_extent_bits+0x254)[0x560c4d7d0854]
  btrfs(exclude_super_stripes+0xbf)[0x560c4d7c65ff]
  btrfs(btrfs_read_block_groups+0x29d)[0x560c4d7c698d]
  btrfs(btrfs_setup_all_roots+0x3f3)[0x560c4d7c0b23]
  btrfs(+0x1ef53)[0x560c4d7c0f53]
  btrfs(open_ctree_fs_info+0x90)[0x560c4d7c11a0]
  btrfs(+0x6d3f9)[0x560c4d80f3f9]
  btrfs(main+0x94)[0x560c4d7b60c4]
  /usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7fd189773ee3]
  btrfs(_start+0x2e)[0x560c4d7b635e]

[CAUSE]
This is caused by passing @len == 0 to add_excluded_extent(), which
means one reverse mapped range is just out of the block group range,
normally means a by-one error.

[FIX]
Fix the boundary check on the reserve mapped range against block group
range.  If a reverse mapped super block starts at the end of the block
group, it doesn't cover so we don't need to bother the case.

Issue: #210
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
77b2af0a9a btrfs-progs: do proper error handling in btrfs_chunk_readonly()
[BUG]
For a fuzzed image, `btrfs check` both modes trigger BUG_ON():

  Opening filesystem to check...
  volumes.c:1795: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1
  btrfs(+0x2f712)[0x557beff3b712]
  btrfs(+0x32059)[0x557beff3e059]
  btrfs(btrfs_read_block_groups+0x282)[0x557beff30972]
  btrfs(btrfs_setup_all_roots+0x3f3)[0x557beff2ab23]
  btrfs(+0x1ef53)[0x557beff2af53]
  btrfs(open_ctree_fs_info+0x90)[0x557beff2b1a0]
  btrfs(+0x6d3f9)[0x557beff793f9]
  btrfs(main+0x94)[0x557beff200c4]
  /usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7f623ac97ee3]
  btrfs(_start+0x2e)[0x557beff2035e]

[CAUSE]
The fuzzed image has a bad extent tree:

        item 0 key (288230376165343232 BLOCK_GROUP_ITEM 8388608) itemoff 16259 itemsize 24
                block group used 0 chunk_objectid 256 flags DATA

There is no corresponding chunk for the block group.

In then we hit the BUG_ON(), which expects chunk mapping for
btrfs_chunk_readonly().

[FIX]
Remove that BUG_ON() with proper error handling, and make
btrfs_read_block_groups() handle the -ENOENT error from
read_one_block_group() to continue.

So one corrupted block group item won't screw up the remaining block
group items.

Issue: #209
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
6466dbfdab btrfs-progs: Add extra chunk item size check
[BUG]
For one fuzzed image, `btrfs check` both modes will trigger a BUG_ON():
  Opening filesystem to check...
  Checking filesystem on issue_208.raw
  UUID: 99e50868-0bda-4d89-b0e4-7e8560312ef9
  [1/7] checking root items
  [2/7] checking extents
  ctree.h:320: btrfs_chunk_item_size: BUG_ON `num_stripes == 0` triggered, value 1
  btrfs(+0x2f712)[0x55829afa6712]
  btrfs(+0x322e5)[0x55829afa92e5]
  btrfs(+0x6892a)[0x55829afdf92a]
  btrfs(+0x69099)[0x55829afe0099]
  btrfs(+0x69c68)[0x55829afe0c68]
  btrfs(+0x6dc27)[0x55829afe4c27]
  btrfs(main+0x94)[0x55829af8b0c4]
  /usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7f3edc715ee3]
  btrfs(_start+0x2e)[0x55829af8b35e]

[CAUSE]
The fuzzed image has an invalid chunk item in chunk tree:
  item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 13631488) itemoff 16105 itemsize 80
  invalid num_stripes: 0

Which triggers that BUG_ON().

[FIX]
Here we enhance the verification of btrfs_check_chunk_valid(), to check
the num_stripes and item size.

Issue: #208
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:10 +01:00
Qu Wenruo
6a06115861 btrfs-progs: disk-io: Verify the bytenr passed in is mapped for read_tree_block()
[BUG]
For a fuzzed image, `btrfs check` will segfault at open_ctree() stage:

  $ btrfs check --mode=lowmem issue_207.raw
  Opening filesystem to check...
  extent_io.c:665: free_extent_buffer_internal: BUG_ON `eb->refs < 0` triggered, value 1
  btrfs(+0x6bf67)[0x56431d278f67]
  btrfs(+0x6c16e)[0x56431d27916e]
  btrfs(alloc_extent_buffer+0x45)[0x56431d279db5]
  btrfs(read_tree_block+0x59)[0x56431d2848f9]
  btrfs(btrfs_setup_all_roots+0x29c)[0x56431d28535c]
  btrfs(+0x78903)[0x56431d285903]
  btrfs(open_ctree_fs_info+0x90)[0x56431d285b60]
  btrfs(+0x45a01)[0x56431d252a01]
  btrfs(main+0x94)[0x56431d2220c4]
  /usr/lib/libc.so.6(__libc_start_main+0xf3)[0x7f6e28519153]
  btrfs(_start+0x2e)[0x56431d22235e]

[CAUSE]
The fuzzed image has a strange log root bytenr:

  log_root                61440
  log_root_transid        0

In fact, the log_root seems to be fuzzed, as its transid is 0, which is
invalid.

Note that range [61440, 77824) covers the physical offset of the primary
super block.

The bug is caused by the following sequence:

1. cache for tree block [64K, 68K) is created by open_ctree()
   __open_ctree_fd()
   |- btrfs_setup_chunk_tree_and_device_map()
      |- btrfs_read_sys_array()
         |- sb = btrfs_find_create_tree_block()
         |- free_extent_buffer(sb)

   This created an extent buffer [64K, 68K) in fs_info->extent_cache, then
   reduce the refcount of that eb back to 0, but not freed yet.

2. Try to read that corrupted log root
   __open_ctree_fd()
   |- btrfs_setup_chunk_tree_and_device_map()
   |- btrfs_setup_all_roots()
      |- find_and_setup_log_root()
         |- read_tree_block()
            |- btrfs_find_create_tree_block()
               |- alloc_extent_buffer()

   The final alloc_extent_buffer() will try to free that cached eb
   [64K, 68K), since it doesn't match with current search.
   And since that cached eb is already released (refcount == 0), the
   extra free_extent_buffer() will cause above BUG_ON().

[FIX]
Here we fix it through a more comprehensive method, instead of simply
verifying log_root_transid, here we just don't pollute eb cache when
reading sys chunk array.

So that we won't have an eb cache [64K, 68K), and will error out at
logical mapping phase.

Issue: #207
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Qu Wenruo
3dcce48fd7 btrfs-progs: check/original: Do extra verification on file extent item
[BUG]
For certain fuzzed image, `btrfs check` will fail with the following
call trace:

  Checking filesystem on issue_213.raw
  UUID: 99e50868-0bda-4d89-b0e4-7e8560312ef9
  [1/7] checking root items
  [2/7] checking extents
  Program received signal SIGABRT, Aborted.
  0x00007ffff7c88f25 in raise () from /usr/lib/libc.so.6
  (gdb) bt
  #0  0x00007ffff7c88f25 in raise () from /usr/lib/libc.so.6
  #1  0x00007ffff7c72897 in abort () from /usr/lib/libc.so.6
  #2  0x00005555555abc3e in run_next_block (...) at check/main.c:6398
  #3  0x00005555555b0f36 in deal_root_from_list (...) at check/main.c:8408
  #4  0x00005555555b1a3d in check_chunks_and_extents (fs_info=0x5555556a1e30) at check/main.c:8690
  #5  0x00005555555b1e3e in do_check_chunks_and_extents (fs_info=0x5555556a1e30) a
  #6  0x00005555555b5710 in cmd_check (cmd=0x555555696920 <cmd_struct_check>, argc
  #7  0x0000555555568dc7 in cmd_execute (cmd=0x555555696920 <cmd_struct_check>, ar
  #8  0x0000555555569713 in main (argc=2, argv=0x7fffffffde70) at btrfs.c:386

[CAUSE]
This fuzzed images has a corrupted EXTENT_DATA item in data reloc tree:
        item 1 key (256 EXTENT_DATA 256) itemoff 16111 itemsize 12
                generation 0 type 2 (prealloc)
                prealloc data disk byte 16777216 nr 0
                prealloc data offset 0 nr 0

There are several problems with the item:
- Bad item size
  12 is too small.
- Bad key offset
  offset of EXTENT_DATA type key represents file offset, which should
  always be aligned to sector size (4K in this particular case).

[FIX]
Do extra item size and key offset check for original mode, and remove
the abort() call in run_next_block().

And to show off how robust lowmem mode is, lowmem can handle it without
any hiccup.

With this fix, original mode can detect the problem properly:
  Checking filesystem on issue_213.raw
  UUID: 99e50868-0bda-4d89-b0e4-7e8560312ef9
  [1/7] checking root items
  [2/7] checking extents
  ERROR: invalid file extent item size, have 12 expect (21, 16283]
  ERROR: errors found in extent allocation tree or chunk allocation
  [3/7] checking free space cache
  [4/7] checking fs roots
  root 18446744073709551607 root dir 256 error
  root 18446744073709551607 inode 256 errors 62, no orphan item, odd file extent, bad file extent
  ERROR: errors found in fs roots
  found 131072 bytes used, error(s) found
  total csum bytes: 0
  total tree bytes: 131072
  total fs tree bytes: 32768
  total extent tree bytes: 16384
  btree space waste bytes: 124774
  file data blocks allocated: 0
   referenced 0

Issue: #213
Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Qu Wenruo
121b9a2e4c btrfs-progs: tests: Add --force for repair command
Since commit e388bf38 ("btrfs-progs: check: warn users about the
possible dangers of --repair") `btrfs check --repair` will wait 10
seconds before really repair the fs.

This hugely slow down the fsck tests. Add --force for check_image()

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Qu Wenruo
dca55d6450 btrfs-progs: Sync the comment for btrfs_file_extent_item
The comment about data checksum on disk_bytes is completely wrong.  Sync
it with fixed kernel comment to avoid confusion.

Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
David Sterba
012e51ae6e btrfs-progs: tests: fsck/013: use correct path for btrfs-corrupt-block
All utilities should be found in the $TOP directory, INTERNAL_BIN was
there by accident.

Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Long An
bbb3d8f6f0 btrfs-progs: tests: mkfs/011: Fix path for rootdir
Documentation folder path is wrong on exported testsutie. Fix this by
replace TOP with INTERNAL_BIN.

Signed-off-by: An Long <lan@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Marcos Paulo de Souza
4be1b09293 btrfs-progs: tests: mkfs/020: use correct device for mount
It seems to be a typo, since $dev1 was not set, and we are dealing with
TEST_DEV and not with loop devices. As this is now equivalent to the
common mount/umount helpers, use them.

Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
Marcos Paulo de Souza
8f382bc75e btrfs-progs: tests: mkfs/020: properly set checksum from parameter
The value of checksum was not set making mkfs fail.

Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2020-01-09 14:27:09 +01:00
David Sterba
d39ab9fafe
Btrfs progs v5.4
Signed-off-by: David Sterba <dsterba@suse.com>
2019-12-03 16:47:34 +01:00
David Sterba
152496dac7 btrfs-progs: update CHANGES for 5.4
Signed-off-by: David Sterba <dsterba@suse.com>
2019-12-03 16:47:01 +01:00
David Sterba
ca210687f8 btrfs-progs: fi usage: print ratio of used/total for each chunk type
This is based on idea from Stanislaw Gruszka to print the ratio,
originally suggested for the 'fi df', but we don't want to add new
things there and let people use 'fi us' instead. The new output fits
there and is printed by default without options:

Example output:

  $ btrfs fi us /mnt
  [...]

  Data,single: Size:339.00GiB, Used:172.05GiB (50.75%)

  Metadata,single: Size:7.00GiB, Used:3.41GiB (48.70%)

  System,single: Size:32.00MiB, Used:64.00KiB (0.20%)

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:52 +01:00
David Sterba
a855c63625 btrfs-progs: build: optional dependencies for html doc target
Shell examples can be colorified using the source-highlight utility and
used in the asciidoc sources like

[source,shell]
---
btrfs filesystem show
---

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:52 +01:00
Alexandru Ungureanu
329a5bba15 btrfs-progs: docs: add example for replacing an online drive
Added example for replacing an online drive in a healthy with a bigger
one.

Pull-request: #217
Author: Alexandru Ungureanu <khakcarot@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:52 +01:00
David Sterba
c56439735a btrfs-progs: build: document install targets
Enumerate what exactly gets installed.

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:52 +01:00
Fabrice Fontaine
9a03f6d41f btrfs-progs: build: install static library and headers in install-static
Currently, the install-static target only installs the statically
compiled tools. However, some programs outside of btrfs-progs (for
example docker) link with the btrfs libraries. If such programs want to
link statically against the btrfs-progs library, then this library
should be installed by "install-static". Indeed, "make install" cannot
be used if the support for shared library is not enabled.

Pull-request: #211
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[Gustavo: Rebase for btrfs-progs 4.7.2]
Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
[Vincent: Rebase for btrfs-progs 4.10]
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
[baruch: Rebase for btrfs-progs 4.16]
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
[Retrieved from:
https://git.buildroot.net/buildroot/tree/package/btrfs-progs/0001-Makefile-install-static-library-and-headers-in-insta.patch]
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:52 +01:00
Marcos Paulo de Souza
50ec684a20 btrfs-progs: docs: clarify the default list sort order of subvolumes
By default, "btrfs subvolume list" will show the subvolumes sorted by
subovlume id, in ascending order. Make this explicit in subvolume manual
page.

Pull-request: #220
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
Qu Wenruo
f01d2b8558 libbtrfsutil: Convert to designated initialization for SubvolumeIterator_type
[BUG]
When compiling btrfs-progs with libbtrfsutil on a python3.8 system, we
got the following warning:

  subvolume.c:636:2: warning: initialization of ‘long int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
    636 |  NULL,     /* tp_print */
        |  ^~~~
  subvolume.c:636:2: note: (near initialization for ‘SubvolumeIterator_type.tp_vectorcall_offset’)

[CAUSE]
C definition of PyTypeObject changed in python 3.8.
Now at the old tp_print, we have tp_vectorcall_offset.

So we got above warning.

[FIX]
C has designated initialization, which can assign values to each named
member, without hard coding to match the offset.
And all the other uninitialized values will be set to 0, so we can save
a lot of unneeded "= 0" or "= NULL" lines.

Just use that awesome feature to avoid any future breakage.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
Qu Wenruo
425c950cc6 libbtrfsutil: Convert to designated initialization for QgroupInherit_type
[BUG]
When compiling btrfs-progs with libbtrfsutil on a python3.8 system, we
got the following warning:

  qgroup.c:110:2: warning: initialization of ‘long int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
    110 |  NULL,     /* tp_print */
        |  ^~~~
  qgroup.c:110:2: note: (near initialization for ‘QgroupInherit_type.tp_vectorcall_offset’)

[CAUSE]
C definition of PyTypeObject changed in python 3.8.
Now at the old tp_print, we have tp_vectorcall_offset.

So we got above warning.

[FIX]
C has designated initialization, which can assign values to each named
member, without hard coding to match the offset.
And all the other uninitialized values will be set to 0, so we can save
a lot of unneeded "= 0" or "= NULL" lines.

Just use that awesome feature to avoid any future breakage.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
Qu Wenruo
a528cbeead libbtrfsutil: Convert to designated initialization for BtrfsUtilError_type
[BUG]
When compiling btrfs-progs with libbtrfsutil on a python3.8 system, we
got the following warning:

  error.c:169:2: warning: initialization of ‘long int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
    169 |  NULL,      /* tp_print */
        |  ^~~~
  error.c:169:2: note: (near initialization for ‘BtrfsUtilError_type.tp_vectorcall_offset’)

[CAUSE]
C definition of PyTypeObject changed in python 3.8.
Now at the old tp_print, we have tp_vectorcall_offset.

So we got above warning.

[FIX]
C has designated initialization, which can assign values to each named
member, without hard coding to match the offset.
Also, uninitialized values will be 0, so we can also save a lot of
unneeded "= 0" or "= NULL" lines.

Just use that awesome feature to avoid any future breakage.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
Qu Wenruo
36128dff44 btrfs-progs: check/lowmem: Fix a false alert on uninitialized value
[BUG]
When compiling the devel branch with commit fb8f05e40b458
("btrfs-progs: check: Make repair_imode_common() handle inodes in
subvolume trees"), the following warning will be reported:

  check/mode-common.c: In function ‘detect_imode’:
  check/mode-common.c|1071 col 23| warning: ‘imode’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  1071 |   *imode_ret = (imode | 0700);
       |                ~~~~~~~^~~~~~~

This only occurs for regular build. If compiled with D=1, the warning
just disappears.

[CAUSE]
Looks like a bug in gcc optimization.
The code will only set @imode_ret when @found is true.
And for every "found = true" assignment we have assigned @imode.
So this is just a false alert.

[FIX]
I hope I can fix the problem of GCC, but obviously I can't (at least for
now).

So let's assign an initial value 0 to @imode to suppress the false
alert.

Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
Su Yue
97fc76c0ac btrfs-progs: add comments of block group lookup functions
The progs side function btrfs_lookup_first_block_group() calls
find_first_extent_bit() to find block group which contains bytenr
or after the bytenr. This behavior differs from kernel code, so
add the comments.

Add the coments of btrfs_lookup_block_group() too, this one works
like kernel side.

Signed-off-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
David Sterba
899977cd18 btrfs-progs: docs: update mkfs blockgroup description
- add raid1c34
- add utilization to the overview table
- wording updates

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
David Sterba
95117cbd4a btrfs-progs: tests: add tests for checksums
Add separate tests for basic coverage of new checksum algorithms. It
comes in two parts to do a full mkfs test and also a condition mount
test.

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:51 +01:00
David Sterba
b475a46f4d btrfs-progs: tests: add raid1c34 to basic mkfs tests
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:50 +01:00
David Sterba
1f5094bb5c btrfs-progs: add support for raid1c3 and raid1c4
Add support for 3- and 4- copy variants of RAID1. This adds resiliency
against 2 or resp. 3 devices lost or damaged.

$ ./mkfs.btrfs -m raid1c4 -d raid1c3 /dev/sd[abcd]

Label:              (null)
UUID:               f1f988ab-6750-4bc2-957b-98a4ebe98631
Node size:          16384
Sector size:        4096
Filesystem size:    8.00GiB
Block group profiles:
  Data:             RAID1C3         273.06MiB
  Metadata:         RAID1C4         204.75MiB
  System:           RAID1C4           8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata, raid1c34
Number of devices:  4
Devices:
   ID        SIZE  PATH
    1     2.00GiB  /dev/sda
    2     2.00GiB  /dev/sdb
    3     2.00GiB  /dev/sdc
    4     2.00GiB  /dev/sdd

Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:09:50 +01:00
Nikolay Borisov
beef042d50 btrfs-progs: Remove convert param from btrfs_alloc_data_chunk
Convert is always set to true so there's no point in having it as a
function parameter or using it as a predicate inside
btrfs_alloc_data_chunk.  Remove it and all relevant code which would
have never been executed.  No semantics changes.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:07:21 +01:00
Nikolay Borisov
f28a5a1673 btrfs-progs: Remove type argument from btrfs_alloc_data_chunk
It's always set to BTRFS_BLOCK_GROUP_DATA so sink it into the function.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:07:21 +01:00
Nikolay Borisov
6718ab4d33 btrfs-progs: Initialize sub_stripes to 1 in btrfs_alloc_data_chunk
sub_stripe variables is by default initialized to 0 and it's overriden
only in case we have RAID10 mode. This leads to the following (minor)
artifacts on a freshly created filesystem:

item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid a020fc2f-b526-4800-9278-156f2f431fe9
			stripe 1 devid 1 offset 30408704
			dev_uuid 0f78aa72-4626-4057-a8f2-285f46b2c664

After balance resulting chunk item is:

item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 3251634176) itemoff 15863 itemsize 112
		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 3230662656
			dev_uuid a020fc2f-b526-4800-9278-156f2f431fe9
			stripe 1 devid 1 offset 3251634176
			dev_uuid 0f78aa72-4626-4057-a8f2-285f46b2c664

Kernel code usually initializes it to 1, since it takes the value from
the raid description table which has it set to 1 for all but RAID10 types.
In userspace it has to be statically initialized to 1 since we don't
have btrfs_bg_flags_to_raid_index. Eventually the kernel/userspace needs
to be merged but for now it wouldn't bring much value if this function
is copied.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-11-22 19:07:21 +01:00