The kernel seems to order inline extent items in a particular way:
forward by sub-type, then reverse by hash. Having these out of order can
cause a volume to go readonly when deleting an inode.
See https://github.com/maharmstone/ntfs2btrfs/issues/51
With additional comments from the pull request:
- lookup_inline_extent_backref() is skipping the remaining backref item
if data/metadata item is smaller (either through the data hash, or
metadata parent/ref_root) than the target range
- the fix could be still missing in lowmem mode
- image could be created according this comment
https://github.com/maharmstone/ntfs2btrfs/issues/51#issuecomment-1500781204
- due to late merge, the squota newly added key
BTRFS_EXTENT_OWNER_REF_KEY was not part of the patch and the value of
'hash' needs to be verified
Pull-request: #622
Signed-off-by: Mark Harmstone <mark@harmstone.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There were some files left after running 'make clean-all'. Reorganize
the target commands and group them by type of files so it's easier to
see what's cleaned and where to add new files.
Signed-off-by: David Sterba <dsterba@suse.com>
Add new option -p to 'subvolume create' so it behaves like 'mkdir -p'
and create all missing path components before the subvolume.
Issue: #429
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We need to validate the device uuid the same way as the fsid:
$ ./mkfs.btrfs --device-uuid 18eabcf0-6766-4fbf-b366-71b4ae725b2- img
btrfs-progs v6.5.2
See https://btrfs.readthedocs.io for more information.
ERROR: could not parse device UUID: 18eabcf0-6766-4fbf-b366-71b4ae725b2-
Signed-off-by: David Sterba <dsterba@suse.com>
Print the device uuid in the summary in case it's specified on the
command line, not always as it would be confusing and is not usually
needed. Can be found in 'btrfs inspect-internal dump-super' as
device_item.uuid .
Signed-off-by: David Sterba <dsterba@suse.com>
Add option --device-uuid that will set the device uuid item in super
block.
This is useful for creating a filesystem with a specific device uuid,
namely for testing.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The commit ("btrfs-progs: allow duplicate fsid for single device
filesystems") lets the duplicate fsid used for a new mkfs document this.
Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The initial patch used -q for enabling simple quota but this is not
right, -q is reserved for --quiet and we want the long options first.
Signed-off-by: David Sterba <dsterba@suse.com>
The new test case would:
- Prepare two loopback devices
One is for storing the source directory (on a btrfs).
This is to ensure we can set xattr for the directory, as filesystems
like tmpfs (mostly utilized by mktemp) is not supporting xattr.
The other one is the real target fs where we call
"mkfs.btrfs --rootdir" on.
- Create the source directory with the following contents:
* rootdir inode attributs:
# mode (750)
# uid (1000)
# gid (1000)
# xattr (user.rootdir)
* one regular file, with attributes:
# xattr (user.foorbar)
- Execute "mkfs.btrfs --rootdir" and mount the new fs
- Verify the above attributes
The target fs should have the same attributes, especially for the
rootdir inode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When using "mkfs.btrfs" with "--rootdir" option, the top level inode
(rootdir) will not get the same xattr from the source dir:
mkdir -p source_dir/
touch source_dir/file
setfattr -n user.rootdir_xattr source_dir/
setfattr -n user.regular_xattr source_dir/file
mkfs.btrfs -f --rootdir source_dir $dev
mount $dev $mnt
getfattr $mnt
# Nothing <<<
getfattr $mnt/file
# file: $mnt/file
user.regular_xattr <<<
[CAUSE]
In function traverse_directory(), we only call add_xattr_item() for all
the child inodes, not really for the rootdir inode itself, leading to
the missing xattr items.
Not only xattr, in fact we also miss the uid/gid/timestamps/mode for the
rootdir inode.
[FIX]
Extract a dedicated function, copy_rootdir_inode(), to handle every
needed attributes for the rootdir inode, including:
- xattr
- uid
- gid
- mode
- timestamps
Issue: #688
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
For running scrubs, with v6.3 and newer btrfs-progs, it can report
incorrect "Total to scrub":
Scrub resumed: Mon Oct 9 11:28:33 2023
Status: running
Duration: 0:44:36
Time left: 0:00:00
ETA: Mon Oct 9 11:51:38 2023
Total to scrub: 625.49GiB
Bytes scrubbed: 625.49GiB (100.00%)
Rate: 239.35MiB/s
Error summary: no errors found
[CAUSE]
Commit c88ac0170b ("btrfs-progs: scrub: unify the output numbers for
"Total to scrub"") changed the output method for "Total to scrub", but
that value is only suitable for finished scrubs.
For running scrubs, if we use the currently scrubbed values, it would
lead to the above problem.
The real scrubbed bytes is only reliable for finished scrubs, not for
running/canceled/interrupted ones.
[FIX]
Change print_scrub_dev() to do extra checks, and only for finished
scrubs to use the scrubbed bytes.
Otherwise fall back to the device's bytes_used.
Issue: #690
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When running mkfs.btrfs with --rootdir on a block device, and the source
directory contains a sparse file, whose size is larger than the block
size, then mkfs.btrfs would fail:
# lsblk /dev/test/test
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
test-test 253:0 0 10G 0 lvm
# mkdir -p /tmp/output
# truncate -s 20G /tmp/output/file
# mkfs.btrfs -f --rootdir /tmp/output /dev/test/test
# sudo mkfs.btrfs -f /dev/test/scratch1 --rootdir /tmp/output/
btrfs-progs v6.3.3
See https://btrfs.readthedocs.io for more information.
ERROR: unable to zero the output file
[CAUSE]
Mkfs.btrfs would try to zero out the target file according to the total
size of the directory.
However the directory size is calculated using the file size, not the
real bytes taken by the file, thus for such sparse file with holes only,
it would still take 20G.
Then we would use that 20G size to zero out the target file, but if the
target file is a block device, we would fail as we can not enlarge a block
device.
[FIX]
When zeroing the file, we only enlarge it if the target is a regular
file.
Otherwise we warn about the size and continue.
Please note that, since "mkfs.btrfs --rootdir" doesn't handle sparse
file any differently from regular file, above case would still fail due
to ENOSPC, as will write zeros into the target file inside the fs.
Proper handling for sparse files would need a new series of patch to
address.
Issue: #653
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Currently, write_dev_supers() compares the superblock location vs the size
of the device to check if it can write the superblock. This is not correct
for a zoned device, whose superblock location is different than a regular
device.
Introduce check_sb_location() to check if the superblock zone exists for
the zoned case.
Running btrfs check can fail on a certain zoned device setup (e.g,
zone size = 128MB, device size = 16GB).
From generic/330:
yes | btrfs check --repair --force /dev/nullb1
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
ERROR: zoned: failed to read zone info of 4096 and 4097: Invalid argument
ERROR: failed to write super block for devid 1: write error: Input/output error
failed to write new super block err -5
failed to repair damaged filesystem, aborting
This happens because write_dev_supers() is comparing the original
superblock location vs the device size to check if it can write out a
superblock copy or not.
For the above example, since the first copy location (64MB) < device size
(16GB), it tries to write out the copy. But, the copy must be written into
zone 4096 (512G / zone size (128M) = 4096), which is out of the device.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce sb_bytenr_to_sb_zone(), which converts the original superblock
location to the zone number of superblock log writing.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The convert tests take a long time and are not necessary for quick test
rounds under the convenience target 'make test'. We still run the tests
in CI.
Signed-off-by: David Sterba <dsterba@suse.com>
The kernel patches for RST and squota are queued for 6.7, we need to be
able to test the features so it's not necessary to hide the mkfs support
under experimental build. The kernel may still need debug build to
enable mount.
Signed-off-by: David Sterba <dsterba@suse.com>
While we like to have the descriptive names also add short aliases that
we also use for reference in changelogs and documentation.
$ mkfs.btrfs -O list-all
Filesystem features available:
mixed-bg - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
quota - quota support (qgroups) (compat=3.4)
extref - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
raid56 - raid56 extended format (compat=3.9)
skinny-metadata - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
no-holes - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
fst - free-space-tree alias
free-space-tree - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
raid1c34 - RAID1 with 3 or 4 copies (compat=5.5)
zoned - support zoned devices (compat=5.12)
extent-tree-v2 - new extent tree format (compat=5.15)
bgt - block-group-tree alias
block-group-tree - block group tree to reduce mount time (compat=6.1)
rst - raid-stripe-tree alias
raid-stripe-tree - raid stripe tree (compat=6.7)
squota - squota support (simple accounting qgroups) (compat=6.7)
Signed-off-by: David Sterba <dsterba@suse.com>
The list 'mkfs -O list-all' should be sorted by version of kernel
support (compat) so it's clear what's new.
Signed-off-by: David Sterba <dsterba@suse.com>
Unlike kernel, in btrfs-progs btrfs_start_transaction() never checks if
there is enough metadata space.
This can lead to very dangerous situation where there is no metadata
space left at all, deadlocking future tree operations.
This patch introduces a very basic version of metadata/system free space
check by:
- Check if there is enough metadata/system space left
If there is enough, go as usual.
- If there is not enough space left, try allocating a new chunk
- Recheck if the new space can meet our demand
If not, return ERR_PTR(-ENOSPC).
Otherwise, allocate a new trans handle to the caller.
This is possible thanks to the simplified transaction model in
btrfs-progs:
- We don't allow joining a transaction
This means we don't need to handle complex cases like data ordered
extents, which need to reserve space first, then join the current
transaction and use the reserved blocks.
- We don't allow multiple transaction handles for one transaction
Since btrfs-progs is single threaded, we always start a transaction
and then commit it.
However there is a feature that must be an exception for the new
metadata/system free space check:
- btrfs check --init-extent-tree
As all the meta/system free space check is based on the space info,
which is loaded from block group items.
Thus when rebuilding extent tree, we can no longer have an accurate
view, thus we have to disable the feature for the whole execution if
we're rebuilding the extent tree.
For now, there is no regression exposed during the self tests, but I
really hope this can be an extra safety net to prevent causing ENOSPC
deadlock in btrfs-progs.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Create subvolumes and delete every N-th to generate output where we can
also see how the stale qgroups are placed (at the end).
Issue: #687
Signed-off-by: David Sterba <dsterba@suse.com>
This patch fixes a bug that could occur when comparing paths in showing
qgroups list. Old code doesn't check it and the bug occurs when there is
stale qgroup and its path is null. Check whether it is null and return
without comparing paths.
Issue: #687
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Sidong Yang <realwakka@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The commands fix-device-size, clear-uuid-tree and clear-ino-cache have
been missing in the completion file since the command implementation was
added.
Signed-off-by: David Sterba <dsterba@suse.com>
The option "--clear-ino-cache" is not really that suitable for "btrfs
check" group.
Let's move it to "btrfs rescue" group to fix those small hiccups, just
like the existing "btrfs rescue fix-device-size" command.
For now, "btrfs check --clear-ino-cache" would still work, with one
extra warning referring to "btrfs rescue clear-ino-cache".
This is mostly to reduce the surprise, and keep script users (I doubt if
there is any though) happy for now.
In the next or two releases, we would fully remove the support in "btrfs
check" group.
Another small change is, in the documents, we refer to the feature as
"inode map", which doesn't match with the mount option documents.
Since we're here, unify them to "inode cache" feature.
Issue: #669
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The clear-cache functionality is shared by several commands:
- btrfs check
For --clear-cache and --clear-ino-cache.
- btrfstune
Mostly for block-group-tree feature conversion.
- btrfs-convert
To enable the now default v2 space cache.
Thus it's no longer proper to keep clear-cache.[ch] under check/
directory, move them to common/ directory.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There are quite some variable shadowing in btrfs-progs, most of them are
just reusing some common names like tmp.
And those are quite safe and the shadowed one are even different type.
But there are some exceptions:
- @end in traverse_tree_blocks()
There is already an @end with the same type, but a different meaning
(the end of the current extent buffer passed in).
Just rename it to @child_end.
- @start in generate_new_data_csums_range()
Just rename it to @csum_start.
- @size of fixup_chunk_tree_block()
This one is particularly bad, we declare a local @size and initialize
it to -1, then before we really utilize the variable @size, we
immediately reset it to 0, then pass it to logical_to_physical().
Then there is a location to check if @size is -1, which will always be
true.
According to the code in logical_to_physical(), @size would be clamped
down by its original value, thus our local @size will always be 0.
This patch would rename the local @size to @found_size, and only set
it to -1.
The call site is only to pass something as logical_to_physical()
requires a non-NULL pointer.
We don't really need to bother the returned value.
- duplicated @ref declaration in run_delayed_tree_ref()
- duplicated @super_flags in change_meta_csums()
Just delete the duplicated one.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The current implementation would introduce variable shadowing due to
both max() and min() are using the same __x and __y.
This may not be a big deal, but since kernel is already handling it
properly using __UNIQUE_ID() macro, and has more checks, we can
cross-port the kernel version to btrfs-progs.
There are some dependency needed, they are all small enough thus can be
put into the helper.
- __PASTE()
- __UNIQUE_ID()
- BUILD_BUG_ON_ZERO()
- __is_constexpr()
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There is a bug report [1] that btrfstune failed to convert to block
group tree. The bug report shows some very weird call trace, mostly
fail at free space tree code.
One of the concern is the failed conversion may be caused by exhausted
metadata space. In that case, we're doing quite some metadata
operations (one transaction to convert 64 block groups in one go).
But for each transaction we have only reserved the metadata for one
block group conversion (1 block for extent tree and 1 block for block
group tree, this excludes free space and extent tree operations, as they
should go global reserve).
Although we're not 100% sure about the failed conversion, at least we
should handle the metadata reservation correctly, by properly reserve
the needed space for the batch, and reduce the batch size just in case
there isn't much metadata space left.
[1] https://lore.kernel.org/linux-btrfs/3c93d0b5-a8cb-ebe3-f8d6-76ea6340f23e@gmail.com/
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Currently for multi-word tree names, we only allow '_' to connect the
two words, like "block_group".
Meanwhile for mkfs features, we go '-' to connect two words, like
"block-group-tree".
This makes users to use different separators for different commands.
This patch would allow using both '-' and '_' for tree ids.
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The stride length has been removed from kernel code, remove it here as
well.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The length has been removed from kernel, remove it here as well.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This partially reverts commit 2726a83952,
the part where the test runner scripts call the log scanning. There are
still unfixed extent buffer leak reported by fuzz tests. Disable it
temporarily so CI can pass and do other pre-release checks. Scanning
will be enabled after release again.
Signed-off-by: David Sterba <dsterba@suse.com>
There are several places that we call btrfs_abort_transaction() in a
failure case, but never call btrfs_commit_transaction(). This leaks the
trans handle and the associated extent buffers and such. Fix all these
sites by making sure we call btrfs_commit_transaction() after we call
btrfs_abort_transaction() to make sure all the appropriate cleanup is
done. This gets rid of the leaked extent buffer errors.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
When adding the extent buffer leak detection I started getting failures
on some of the fuzz tests. This is because we don't clean up dirty
buffers for aborted transactions, we just leave them dirty and thus we
leak them. Fix this up by making btrfs_commit_transaction() on an
aborted transaction properly cleanup the dirty buffers that exist in the
system.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Update parts of struct btrfs_delayed_ref_head and updated where used,
add more prototypes. More still needs to be synced.
Signed-off-by: David Sterba <dsterba@suse.com>
