Userspace utilities to manage btrfs filesystems
Go to file
Qu Wenruo 2a8cec4b12 btrfs-progs: Exhaust delayed refs and dirty block groups to prevent delayed refs lost
[BUG]
Btrfs-progs sometimes fails to find certain extent backref when
committing transaction.
The most reliable way to reproduce it is fsck-test/013 on 64K page sized
system:

  [...]
  adding new data backref on 315859712 root 287 owner 292 offset 0 found 1
  btrfs unable to find ref byte nr 31850496 parent 0 root 2  owner 0 offset 0
  Failed to find [30867456, 168, 65536]

Also there are some github bug reports related to this problem.

[CAUSE]
Commit 909357e867 ("btrfs-progs: Wire up delayed refs") introduced
delayed refs in btrfs-progs.

However in that commit, delayed refs are not run at correct timing.
That commit calls btrfs_run_delayed_refs() before
btrfs_write_dirty_block_groups(), which needs to update
BLOCK_GROUP_ITEMs in extent tree, thus could cause new delayed refs.

This means each time we commit a transaction, we may screw up the extent
tree by dropping some pending delayed refs, like:

Transaction 711:
btrfs_commit_transaction()
|- btrfs_run_delayed_refs()
|  Now all delayed refs are written to extent tree
|
|- btrfs_write_dirty_block_groups()
|  Needs to update extent tree root
|  ADD_DELAYED_REF to 315859712.
|  Delayed refs are attached to current trans handle.
|
|- __commit_transaction()
|- write_ctree_super()
|- btrfs_finish_extent_commit()
|- kfree(trans)
   Now delayed ref for 315859712 are lost

Transaction 712:
Tree block 315859712 get dropped
btrfs_commit_transaction()
|- btrfs_run_delayed_refs()
   |- run_one_delayed_ref()
      |- __free_extent()
         As previous ADD_DELAYED_REF to 315859712 is lost, extent tree
         doesn't have any backref for 315859712, causing the bug

In fact, commit c31edf610c ("btrfs-progs: Fix false ENOSPC alert by
tracking used space correctly") detects the tree block leakage, but in
the reproducer we have too much noise, thus nobody notices the leakage
warning.

[FIX]
We can't just move btrfs_run_delayed_refs() after
btrfs_write_dirty_block_groups(), as during btrfs_run_delayed_refs(), we
can re-dirty block groups.
Thus we need to exhaust both delayed refs and dirty blocks.

This patch will call btrfs_write_dirty_block_groups() and
btrfs_run_delayed_refs() in a loop until both delayed refs and dirty
blocks are exhausted. Much like what we do in commit_cowonly_roots() in
kernel.

Also, to prevent such problem from happening again (and not to debug
such problem again), add extra check on delayed refs before freeing the
transaction handle.

Reported-by: Klemens Schölhorn <klemens@schoelhorn.eu>
Issue: #187
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-07-26 17:46:41 +02:00
Documentation btrfs-progs: dump-tree: add noscan option 2019-07-04 15:36:00 +02:00
check btrfs-progs: move rbtree-utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
ci btrfs-progs: ci: rename travis/ to ci/ 2019-07-04 02:06:34 +02:00
cmds btrfs-progs: dump-tree: add noscan option 2019-07-04 15:36:00 +02:00
common btrfs-progs: fix a printf format string fatal warning 2019-07-26 17:46:41 +02:00
convert btrfs-progs: build most common tools into one binary (busybox style) 2019-07-04 15:30:40 +02:00
image btrfs-progs: build: add stub makefile to image and mkfs 2019-07-04 15:36:01 +02:00
kernel-lib btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
kernel-shared btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
libbtrfsutil libbtrfsutil: subvolume: use helpers to access search header 2019-07-04 15:36:02 +02:00
m4 btrfs-progs: build: detect whether -std=gnu90 is supported 2018-06-07 16:37:35 +02:00
mkfs btrfs-progs: build: add stub makefile to image and mkfs 2019-07-04 15:36:01 +02:00
tests btrfs-progs: tests: make test-clean work without built binaries 2019-07-04 15:36:02 +02:00
.gitignore btrfs-progs: gitignore: Ignore hidden files 2019-03-05 13:27:27 +01:00
.travis.yml btrfs-progs: ci: rename travis/ to ci/ 2019-07-04 02:06:34 +02:00
64-btrfs-dm.rules
Android.mk btrfs-progs: move string-table.[ch] to common/ 2019-07-03 20:49:03 +02:00
CHANGES btrfs-progs: update CHANGES for v5.2 2019-07-05 18:20:22 +02:00
COPYING
INSTALL btrfs-progs: document the new all-in-one binary 2019-07-04 15:36:00 +02:00
Makefile btrfs-progs: build most common tools into one binary (busybox style) 2019-07-04 15:30:40 +02:00
Makefile.extrawarn btrfs-progs: build: disable -Waddress-of-packed-member by default 2019-06-14 15:09:53 +02:00
Makefile.inc.in btrfs-progs: build: add --disable-shared and --disable-static 2018-08-06 15:03:41 +02:00
README.md btrfs-progs: document the new all-in-one binary 2019-07-04 15:36:00 +02:00
VERSION Btrfs progs v5.2 2019-07-05 18:21:55 +02:00
androidcompat.h
autogen.sh
backref.c btrfs-progs: move internal.h to common/ 2019-07-03 20:49:03 +02:00
backref.h
bcp
btrfs-completion btrfs-progs: completion: let dump-tree/dump-super/inode-resolve accept any file 2018-10-23 14:48:38 +02:00
btrfs-corrupt-block.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs-crc.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs-debugfs
btrfs-find-root.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs-fragments.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
btrfs-list.c btrfs-progs: move rbtree-utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
btrfs-list.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs-map-logical.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs-sb-mod.c btrfs-progs: sb-mod: add remaining superblock integer items 2019-07-04 15:36:01 +02:00
btrfs-select-super.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfs.c btrfs-progs: build most common tools into one binary (busybox style) 2019-07-04 15:30:40 +02:00
btrfsck.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
btrfstune.c btrfs-progs: build most common tools into one binary (busybox style) 2019-07-04 15:30:40 +02:00
common-defs.h
configure.ac btrfs-progs: fix typos in user-visible strings 2018-11-26 18:24:49 +01:00
ctree.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
ctree.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
delayed-ref.c btrfs-progs: Handle error properly in btrfs_commit_transaction() 2019-05-13 15:54:47 +02:00
delayed-ref.h btrfs-progs: delayed-ref: Fix memory leak and use-after-free caused by wrong condition to free delayed ref/head. 2019-07-03 13:31:14 +02:00
dir-item.c btrfs-progs: check: lowmem: Add ability to repair dir item with mismatched hash 2019-01-15 18:42:13 +01:00
disk-io.c btrfs-progs: utils: split device scanning functions to own file 2019-07-04 02:06:34 +02:00
disk-io.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
extent-cache.c btrfs-progs: move rbtree-utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
extent-cache.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
extent-tree.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
extent_io.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
extent_io.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
file-item.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
file.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
find-root.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
find-root.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
free-space-cache.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
free-space-cache.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
free-space-tree.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
free-space-tree.h btrfs-progs: Pull free space tree related code from kernel 2018-10-25 16:11:39 +02:00
fsck.btrfs
hash.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
hasher.c
inode-item.c
inode-map.c btrfs-progs: move internal.h to common/ 2019-07-03 20:49:03 +02:00
inode.c btrfs-progs: move messages.[ch] to common/ 2019-07-03 20:49:03 +02:00
ioctl-test.c
ioctl.h btrfs-progs: device scan: add new option to forget one or all scanned devices 2019-03-01 13:40:21 +01:00
kerncompat.h btrfs-progs: kerncompat: define __always_inline conditionally 2019-07-04 15:36:01 +02:00
library-test.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
print-tree.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
print-tree.h btrfs-progs: print-tree: Use bool for @follow 2018-10-31 18:24:14 +01:00
props.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
props.h
qgroup-verify.c btrfs-progs: move rbtree-utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
qgroup-verify.h btrfs-progs: check: enhanced progress indicator 2018-08-06 15:03:23 +02:00
qgroup.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
qgroup.h btrfs-progs: qgroups: introduce btrfs_qgroup_query 2019-07-03 13:31:14 +02:00
quick-test.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
random-test.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
repair.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
repair.h
root-tree.c
send-dump.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
send-dump.h
send-stream.c btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
send-stream.h
send-utils.c btrfs-progs: send: add gcc9 workaround for root item reset 2019-07-04 02:06:34 +02:00
send-utils.h btrfs-progs: build: drop kernel-lib from -I and update paths 2019-07-03 20:49:04 +02:00
send.h
show-blocks
transaction.c btrfs-progs: Exhaust delayed refs and dirty block groups to prevent delayed refs lost 2019-07-26 17:46:41 +02:00
transaction.h btrfs-progs: Avoid nested chunk allocation call 2019-06-05 20:27:31 +02:00
utils-lib.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
uuid-tree.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
version.h.in
volumes.c btrfs-progs: move utils.[ch] to common/ 2019-07-03 20:49:04 +02:00
volumes.h btrfs-progs: sync btrfs_raid_attr from kernel 2019-07-03 13:31:16 +02:00

README.md

Btrfs-progs build status coverity status

Userspace utilities to manage btrfs filesystems. License: GPLv2.

Btrfs is a copy on write (COW) filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration.

This repository hosts following utilities:

  • btrfs — the main administration tool (manual page)
  • mkfs.btrfs — utility to create the filesystem (manual page)
  • all-in-one binary in the busybox style with mkfs.btrfs, btrfs-image and other tools built-in (standalone tools)

The C and python 3 bindings are provided by a LGPL library libbtrfsutil see libbtrfsutil/README.md for more.

See INSTALL for build instructions and tests/README.md for testing information.

Release cycle

The major version releases are time-based and follow the cycle of the linux kernel releases. The cycle usually takes 2 months. A minor version releases may happen in the meantime if there are bug fixes or minor useful improvements queued.

The release tags are signed with a GPG key ID F2B4 1200 C54E FB30 380C 1756 C565 D5F9 D76D 583B, release tarballs are hosted at kernel.org. See file CHANGES or changelogs on wiki.

Reporting bugs

There are several ways, each has its own specifics and audience that can give feedback or work on a fix. The following list is sorted in the order of preference:

  • github issue tracker
  • to the mailing list linux-btrfs@vger.kernel.org -- (not required to subscribe), beware that the mail might get overlooked in other traffic
  • IRC (irc.freenode.net #btrfs) -- good for discussions eg. if a bug is already known, but reports could miss developers' attention
  • bugzilla.kernel.org -- (requires registration), set the product to Filesystems and component Btrfs, please put 'btrfs-progs' into the subject so it's clear that it's not a kernel bug report

Development

The patch submissions, development or general discussions take place at linux-btrfs@vger.kernel.org mailinglist, subsciption is not required to post.

The GitHub pull requests will not be accepted directly, the preferred way is to send patches to the mailinglist instead. You can link to a branch in any git repository if the mails do not make it to the mailinglist or just for convenience (makes it easier to test).

The development model of btrfs-progs shares a lot with the kernel model. The github way is different in some ways. We, the upstream community, expect that the patches meet some criteria (often lacking in github contributions):

  • one logical change per patch: eg. not mixing bugfixes, cleanups, features etc., sometimes it's not clear and will be usually pointed out during reviews
  • proper subject line: eg. prefix with btrfs-progs: subpart, ... , descriptive yet not too long, see git log --oneline for some inspiration
  • proper changelog: the changelogs are often missing or lacking explanation why the change was made, or how is something broken, what are user-visible effects of the bug or the fix, how does an improvement help or the intended usecase
  • the Signed-off-by line: this documents who authored the change, you can read more about the The Developer's Certificate of Origin (chapter 11)
    • if you are not used to the signed-off style, your contributions won't be rejected just because of it's missing, the Author: tag will be added as a substitute in order to allow contributions without much bothering with formalities

Documentation updates

Documentation fixes or updates do not need much explanation so sticking to the code rules in the previous section is not necessary. GitHub pull requests are OK, patches could be sent to me directly and not required to be also in the mailinglist. Pointing out typos via IRC also works, although might get accidentally lost in the noise.

References