Commit Graph

5802 Commits

Author SHA1 Message Date
David Sterba
e96b91078f btrfs-progs: wip raid10c34
- works: mkfs.btrfs -d raid10c3 -m raid10c4 ...
- works: dump-super
- works: dump-tree

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-19 17:44:27 +01:00
David Sterba
aa2f0c3f38 btrfs-progs: ci: add helpers to update base images
Add convenience build scripts for updating all CI images.

    $ images-base-update
    $ images-build-all

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-18 18:14:27 +01:00
David Sterba
f691dce41d btrfs-progs: ci: add Leap 15.4 for testing
Reuse 15.2 for 15.4.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-18 18:14:27 +01:00
David Sterba
175ba9aaeb btrfs-progs: ci: add run scripts for more targets
Add some scripts for convenience, so far there was only one for musl as
it usually breaks first, but we've had some problems on centos due to
old kernel headers and potential breakage when changing kerncpomat.h.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-18 18:14:27 +01:00
David Sterba
aaa660bb89 btrfs-progs: tests: copy fsstress.c from fstests
The fsstress tool is a useful file generator, pull it from fstests as
it's not packaged as a standalone tool anywhere and the LTP version is
out of date.

The file has been modified to build, some xfs-specific ioctls are not
supported.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-18 18:14:27 +01:00
David Sterba
84b3f47a9a btrfs-progs: tests: sync fssum.c from fstests
The support for xattr has been in the fstests version for some time,
copy it here as well.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-17 17:32:17 +01:00
David Sterba
9fcb0efd66 btrfs-progs: btrfstune: experimental, new option to switch csums
This is still work in progress but can survive some stress testing.
There are still some sanity checks missing, do not user this on valuable
data. To enables this, configure must be run with the experimental
features enabled.

  $ mkfs.btrfs --csum crc32c /dev/sdx
  $ <mount, fill with data, unmount>

  $ btrfstune --csum sha256

Will change the checksum to sha256.

Implementation:

- set bit on superblock when the checksums are being changed (similar to
  the uuid rewrite)
- metadata checksums are overwritten in place
- data checksums:
  - the checksum tree is completely deleted and no checksums are
    verified
  - data blocks are enumerated and all checksums generated (same as
    check --init-csum-tree)

To make it usable, it should be restartable and track the current
progress somehow. Also the previous data checksums should be verified
any time they're available.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-17 16:48:20 +01:00
Qu Wenruo
d425a9cc0a btrfs-progs: tests: add test case for init-csum-tree
This new test script will create a fs with the following situations:

- Preallocated extents (no data csum)
- Nodatasum inodes (no data csum)
- Partially written preallocated extents (no data csum for part of the
  extent)
- Regular data extents (with data csum)
- Regular data extents then hole punched (with data csum)
- Preallocated data, then written, then hole punched (with data csum)
- Compressed extents (with data csum)

And make sure after --init-csum-tree (with or without
--init-extent-tree) the result fs can still pass check.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-17 14:45:43 +01:00
Qu Wenruo
4ade97b4e4 btrfs-progs: check: handle csum generation properly for --init-csum-tree --init-extent-tree
When using `--init-csum-tree` with `--init-extent-tree`, csum tree
population will be done by iterating all file extent items.

This allow us to skip preallocated extents, but it still has the
following problems:

- Inodes with NODATASUM

- Hole file extents

- Written preallocated extents
  We will generate csum for the whole extent, while other part may still
  be unwritten.

Make it to follow the same behavior of recently introduced
fill_csum_for_file_extent(), so we can generate correct csum.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-14 17:38:47 +01:00
Qu Wenruo
60876d8550 btrfs-progs: check: don't calculate csum for preallocated file extents
[BUG]
If a btrfs filesystem has preallocated file extents, 'btrfs check
--init-csum-tree' will create csum item for preallocated extents, and
cause error:

  # mkfs.btrfs -f test.img
  # mount test.img /mnt/btrfs
  # fallocate -l 32K /mnt/btrfs/file
  # umount /mnt/btrfs
  # btrfs check --init-csum-tree --force test.img
  ...
  [4/7] checking fs roots
  root 5 inode 257 errors 800, odd csum item
  ERROR: errors found in fs roots
  found 376832 bytes used, error(s) found

And the csum tree is not empty, containing csum for the preallocated
extent:

  $ btrfs ins dump-tree -t csum test.img
  btrfs-progs v5.15.1
  checksum tree key (CSUM_TREE ROOT_ITEM 0)
  leaf 30408704 items 1 free space 16226 generation 9 owner CSUM_TREE
  leaf 30408704 flags 0x1(WRITTEN) backref revision 1
  fs uuid ecc79835-5611-4609-b985-e4ccd6f15b54
  chunk uuid b1c75553-5b82-4aa6-bbbe-e7f50643b1a8
  	item 0 key (EXTENT_CSUM EXTENT_CSUM 13631488) itemoff 16251 itemsize 32
  		range start 13631488 end 13664256 length 32768

[CAUSE]
For `--init-csum-tree` alone, we will use extent tree to iterate each
data extent, and calculate csum for them.

But extent items alone can not tell us if the file extent belongs to a
NODATASUM inode, nor if it's preallocated.

Thus we create csums for those data extents, and cause the problem.

[FIX]
However the fix is not that simple, we can not just generate csum for
non-preallocated range.

As the following case we still need csum for the un-referred part:

  xfs_io -f -c "pwrite 0 8K" -c "sync" -c "punch 0 4K"

So here we have to go another direction by:

- Always generate csum for the whole data extent
  This is the same as the old code

- Iterate the file extents, and delete csum for preallocated range
  or NODATASUM inodes

Issue: #430
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-14 17:35:07 +01:00
Qu Wenruo
ab880074a2 btrfs-progs: check: move csum tree population into mode-common.[ch]
This part has no mode specific operations, just move them into
mode-common.[ch].

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-14 17:33:37 +01:00
Qu Wenruo
90cdc23d01 btrfs-progs: backref: properly queue indirect refs
[BUG]
When calling iterate_extent_inodes() on data extents with indirect ref
(with inline or keyed EXTENT_DATA_REF_KEY), it will fail to execute the
call back function at all.

[CAUSE]
In function find_parent_nodes(), we only add the target tree block if a
backref has @parent populated.

For indirect backref like EXTENT_DATA_REF_KEY, we rely on
__resolve_indirect_ref() to get the parent leaves.

However __resolve_indirect_ref() only grabs backrefs from
&prefstate->pending_indirect_refs.

Meaning callers should queue any indirect backref to
pending_indirect_refs.

But unfortunately in __add_prelim_ref() and __add_missing_keys(), none
of them properly queue the indirect backrefs to pending_indirect_refs,
but directly to pending.

Making all indirect backrefs never got resolved, thus no callback
function executed

[FIX]
Fix __add_prelim_ref() and __add_missing_keys() to properly queue
indirect backrefs to the correct list.

Currently there is no such direct user in btrfs-progs, but later csum
tree re-initialization code will rely this to do proper csum
re-calculate (to avoid preallocated/nodatasum extents).

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-14 17:32:16 +01:00
David Sterba
d6ee42633b btrfs-progs: docs: set version from VERSION file
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 20:18:13 +01:00
David Sterba
8ecc3cd7af btrfs-progs: docs: generate section 5 to final name directly
There was a workaround for asciidoc/xmlto build because page btrfs.5 and
btrfs.8 used the same intermediate file. To avoid clash for the section
5 page it is part of the name, this is still needed, but for sphinx we
can use the final name as it will get the right suffix. This affects
only the generated manual pages.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 18:51:30 +01:00
David Sterba
dd016af0a2 btrfs-progs: docs: remove all converted asciidoc sources
Remove all docs that have a RST version. Remaining: btrfs-ioctl.asciidoc
and DocConventions.asciidoc.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 18:39:25 +01:00
David Sterba
fb3fa2493c btrfs-progs: docs: remove asciidoc build support
Remove asciidoc makefile support and config files.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 18:39:22 +01:00
David Sterba
541f92193a btrfs-progs: build: drop asciidoc detection, default to sphinx
Make sphinx the default build tool, remove asciidoc target selection.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 18:29:42 +01:00
David Sterba
d7f48d16a6 btrfs-progs: docs: add remaining targets for sphinx build
Update sphinx build so that by default it now builds the manual pages,
in quiet mode. 'make html' builds the html, other sphinx targets are
available, see 'make help'. Installation now works as well.

Note: sphinx is still conditional and must be selected by
ASCIIDOOC_TOOL=sphinx ./configure

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 18:18:24 +01:00
Adam Borowski
c1ae9f7a93 btrfs-progs: fix a bunch of typos
These have been detected by lintian and codespell.

Signed-off-by: Adam Borowski <kilobyte@angband.pl>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 17:25:23 +01:00
David Sterba
b0cfe12c4d btrfs-progs: kerncompat: add local definition for alignment macros
There's still problem left with compilation on musl and kernel < 5.11,
because __ALIGN_KERNEL is not defined anymore:

../bin/ld: kernel-shared/volumes.o: in function `create_chunk':
volumes.c:(.text+0x17f8): undefined reference to `__ALIGN_KERNEL'

Due to the entangled includes and unconditional definition of
__ALIGN_KERNEL, we can't use #ifdef in kerncompat.h to define it
eventually (as kerncompat.h is the first include). Instead add local
definitions of the macros and rename them to avoid name clashes.

Pull-request: #433
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-13 14:47:08 +01:00
David Sterba
8ad326b2f2
Btrfs progs v5.16
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-12 18:39:52 +01:00
David Sterba
02a7c23c8e btrfs-progs: update CHANGES for 5.16
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-12 18:29:33 +01:00
David Sterba
23bdd548d8 btrfs-progs: docs: split mount options
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-12 00:09:29 +01:00
David Sterba
864d260154 btrfs-progs: docs: split swapfile from section 5
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-11 23:56:48 +01:00
David Sterba
6e78548b35 btrfs-progs: update README
- link to readthedocs
- update IRC channel
- other minor updates

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-11 23:40:48 +01:00
Nikolay Borisov
f59a229d2c btrfs-progs: remove redundant fs uuid validation from make_btrfs
cfg->fs_uuid is either 0 or set to the value of the -U parameter
passed to mkfs.btrfs. However the value of the latter is already being
validated in the main mkfs function. Just remove the duplicated checks
in make_btrfs as they effectively can never be executed.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-11 18:02:46 +01:00
David Sterba
433321ee8f btrfs-progs: send: properly handle an ERR_PTR in find_good_parent
Pointer returned from get_parent needs additional handling otherwise
we could return an error and then try to free it. Reset the pointer when
the error occurs so the cleanup is always done on a valid pointer.

Issue: #423
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-11 17:36:15 +01:00
David Sterba
5004a7de0a btrfs-progs: docs: update template, more about subvolumes
The max width of the page is set to 800px, that renders of commands and
output in a small box requiring a scrollbar. Increase the limit to
1200px. Also update some subvoulme related text.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-11 15:46:33 +01:00
Fabrice Fontaine
a2511aaa85 btrfs-progs: include sys/sysinfo.h conditionally on musl
Make inclusion of sys/sysinfo.h conditional to avoid the following build
failure on musl:

In file included from .../i586-buildroot-linux-musl/sysroot/usr/include/linux/kernel.h:4,
                 from ./kerncompat.h:31,
                 from common/utils.c:42:
.../i586-buildroot-linux-musl/sysroot/usr/include/linux/sysinfo.h:7:8: error: redefinition of 'struct sysinfo'
    7 | struct sysinfo {
      |        ^~~~~~~
In file included from common/utils.c:27:
.../i586-buildroot-linux-musl/sysroot/usr/include/sys/sysinfo.h:10:8: note: originally defined here
   10 | struct sysinfo {
      |        ^~~~~~~

Fixes:
 - http://autobuild.buildroot.org/results/16f44fb9dea72a7079e8e5517e760dd79d2724cc

The 'struct sysinfo' is defined in linux/sysinfo.h and sys/sysinfo.h,
while both must not be included at the same time. Stop including
linux/kernel.h that sometimes unconditionally includes sys/sysinfo.h and
causes the double definition for some reason. As we now include
linux/const.h directly, there's no other effective change.

Pull-request: #433
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-10 19:35:28 +01:00
Goffredo Baroncelli
817483211a btrfs-progs: prop: allow autodetect_object_types() to handle link
The function autodetect_object_types() tries to detect the type of
btrfs object passed. If it is an "inode" type (e.g. file) this function
returns the type as "inode". If it is a block device, it return it as
"block device".
However it doesn't handle the case where the object passed is a link
to a block device (which could be a valid btrfs device). For example
LVM/DM creates link to block devices. In this case it should return
the type as "block device".

This patch replace the lstat() call with a stat().

Reported-by: Boris Burkov <boris@bur.io>
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Goffredo Baroncelli <kreijack@inwind.it>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-10 18:40:52 +01:00
David Sterba
79ef78f0e4 btrfs-progs: docs: more docs updates
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-10 16:20:34 +01:00
David Sterba
df91bfd5d5 btrfs-progs: docs: update some chapters
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-05 00:44:14 +01:00
Bruce Ashfield
b28f7bd9bb btrfs-progs: include linux/const.h to fix build with 5.12+ headers
btrfs-tools compile fails with mips, musl and 5.12+ headers.

The definition of __ALIGN_KERNEL has moved in 5.12+ kernels, so we
add an explicit include of const.h to pickup the macro:

  | make: *** [Makefile:595: mkfs.btrfs] Error 1
  | make: *** Waiting for unfinished jobs....
  | libbtrfs.a(volumes.o): in function `dev_extent_search_start':
  | /usr/src/debug/btrfs-tools/5.12.1-r0/git/kernel-shared/volumes.c:464: undefined reference to `__ALIGN_KERNEL'
  | collect2: error: ld returned 1 exit status

This is safe for older kernel's as well, since the header still
exists, and is valid to include.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
[remove invalid OE Upstream-status]
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-04 16:56:33 +01:00
Dāvis Mosāns
07320bd20f btrfs-progs: receive: properly check ERR return value in process_snapshot
When some error happens when trying to search for parent subvolume
then parent_subvol will contain errno so don't try to free that

Crash backtrace would look like:
0  process_snapshot at cmds/receive.c:358
    358		free(parent_subvol->path);
1  0x00005646898aaa67 in read_and_process_cmd at common/send-stream.c:348
2  btrfs_read_and_process_send_stream at common/send-stream.c:525
3  0x00005646898c9b8b in do_receive at cmds/receive.c:1113
4  cmd_receive at cmds/receive.c:1316
5  0x00005646898750b1 in cmd_execute at cmds/commands.h:125
6  main at btrfs.c:405

(gdb) p parent_subvol
$1 = (struct subvol_info *) 0xfffffffffffffffe

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Dāvis Mosāns <davispuh@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-01-03 15:55:35 +01:00
David Sterba
208aed2ed4 btrfs-progs: docs: add more chapters (part 3)
All main pages have some content and many typos have been fixed.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-12-17 15:35:10 +01:00
David Sterba
c6be84840f btrfs-progs: docs: add more chapters (part 2)
The feature pages share the contents with the manual page section 5 so
put the contents to separate files. Progress: 2/3.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-12-17 11:53:44 +01:00
David Sterba
b871bf49f3 btrfs-progs: docs: add more chapters
The feature pages share the contents with the manual page section 5 so
put the contents to separate files.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-12-17 11:53:44 +01:00
Sidong Yang
7f94ccb20a btrfs-progs: check: change commit condition in fixup_extent_refs()
This patch fixes potential bugs in fixup_extent_refs(). If
btrfs_start_transaction() fails in some way and returns error ptr, It
goes to out logic. But old code checkes whether it is null and it calls
commit. This patch solves the problem with make that it calls only if
ret is no error.

Issue: #409
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:16:22 +01:00
Josef Bacik
5e8a779f5c btrfs-progs: add on disk pointers to global tree ids
We are going to start creating multiple sets of global trees, which at
the moment are the free space tree, csum tree, and extent tree.
Generally we will assign these at block group creation time, but Dave
would like to be able to have them per-subvolume at some point, so
reserve a slot for that as well.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:09:47 +01:00
Josef Bacik
ec0eaae673 btrfs-progs: add definitions for the block group tree
Add the on disk definitions for the block group tree.  This will be part
of the super block so we need to add the appropriate helpers to the
super block, as well as adding it to the backup roots.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:08:39 +01:00
Josef Bacik
3337b7993b btrfs-progs: common: allow users to select extent-tree-v2 option
We want to enable developers to test the extent tree v2 features as they
are added, add the ability to mkfs an extent tree v2 fs if we have
experimental enabled.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:07:34 +01:00
Josef Bacik
9dbfb6c162 btrfs-progs: search all extent roots for marking used space
We could have multiple extent roots, so add a helper to mark all the
used space in the FS based on any extent roots we find, and then use
this extent io tree to fixup the block group accounting.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 19:00:12 +01:00
Josef Bacik
e68e73318f btrfs-progs: check: fill csum root from all extent roots
We may have multiple extent roots, so cycle through all of the extent
roots and populate the csum tree based on the content of every extent
root we have in the file system.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:59:34 +01:00
Josef Bacik
bb575e3818 btrfs-progs: check: check all of the csum roots
Use the global roots tree to find all of the csum roots in the system
and check all of them as appropriate.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:59:20 +01:00
Josef Bacik
d26c0b4414 btrfs-progs: check: check the global roots for uptodate root nodes
Instead of checking the csum and extent tree individually, walk through
the global roots and validate them all.  This will work properly if we
have extent tree v1 or extent tree v2.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:45 +01:00
Josef Bacik
bf0ce5cb86 btrfs-progs: check: make reinit work per found root item
Instead of looking for the first extent root or csum root in memory,
scan through the tree root and re-init any root items that match the
given objectid.  This will allow reinit to work with both extent tree v1
and extent tree v2 global roots when using the --reinit option.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:43 +01:00
Josef Bacik
b057607325 btrfs-progs: track csum, extent, and free space trees in a rb tree
We are going to have multiples of these trees with extent tree v2, so
add a rb tree to track them based on their root key value.  This works
for both v1 and v2, so we can remove the direct pointers to these roots
in our fs_info.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:25 +01:00
Josef Bacik
0b23744de5 btrfs-progs: stop accessing ->free_space_root directly
We're going to have multiple free space roots in the future, so access
it via a helper in most cases.  We will address the remaining direct
accesses in future patches.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:57:19 +01:00
Josef Bacik
db2ab47823 btrfs-progs: stop accessing ->extent_root directly
When we switch to multiple global trees we'll need to access the
appropriate extent root depending on the block group or possibly root.
To handle this, use a helper in most places and then the actual root in
places where it is required.  We will whittle down the direct accessors
with future patches, but this does the bulk of the preparatory work.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:56:54 +01:00
Sidong Yang
bc2382eff9 btrfs-progs: fi du: skip inaccessible files
Filesystem du command fails and exits when it access file that has
permission denied. But it can continue the command except the files.
This patch prints error message just like /bin/du does and it continues
if it can.

Issue: #421
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-30 18:40:18 +01:00