[BUG]
When rolling back a converted btrfs, the filename output is corrupted:
$ btrfs-convert -r ~/test.img
btrfs-convert from btrfs-progs v6.9.2
Open filesystem for rollback:
Label:
UUID: df54baf3-c91e-4956-96f9-99413a857576
Restoring from: ext2_saved0ƨy/image
^^^ Corruption
Rollback succeeded
[CAUSE]
The error is in how we handle the filename. In btrfs all our strings
are not '\0' terminated, but with explicit length.
But in C, most strings are '\0' terminated, so after reading a filename
from btrfs, we need to manually terminate the string.
However the code adding the terminating '\0' looks like this:
/* Get the filename length. */
name_len = btrfs_root_ref_name_len(path.nodes[0], root_ref_item);
/*
* This should not happen, but as an extra handling for possible
* corrupted btrfs.
*/
if (name_len > sizeof(dir_name))
name_len = sizeof(dir_name) - 1;
/* Got the real filename into our buffer. */
read_extent_buffer(path.nodes[0], dir_name, (unsigned long)(root_ref_item + 1), name_len);
/* Terminate the string. */
dir_name[sizeof(dir_name) - 1] = 0;
The problem is, the final termination is totally wrong, it always make
the last buffer char '\0', not using the @name_len we read before.
[FIX]
Use @name_len to terminate the string, as we have already updated it to
handle buffer overflow, it can handle both the regular and corrupted
case.
Fixes: dc29a5c51d ("btrfs-progs: convert: update default output")
Signed-off-by: Qu Wenruo <wqu@suse.com>
This test case checks:
- If a regular btrfs-image dump has the unsanitized filenames
- If a sanitized btrfs-image dump has filenames properly censored
Signed-off-by: Qu Wenruo <wqu@suse.com>
The new test case does:
- Make sure the build has error injection support
This is done by checking "btrfs --version" output.
- Inject error at the last commit transaction of new data csum
generation
- Resume the csum conversion and make sure it works
Signed-off-by: Qu Wenruo <wqu@suse.com>
[BUG]
There is a bug report that image dump taken by "btrfs-image -s" doesn't
really sanitize the filenames:
# truncates -s 1G source.raw
# mkfs.btrfs -f source.raw
# mount source.raw $mnt
# touch $mnt/top_secret_filename
# touch $mnt/secret_filename
# umount $mnt
# btrfs-image -s source.raw dump.img
# string dump.img | grep filename
top_secret_filename
secret_filename
top_secret_filename
secret_filename
top_secret_filename
[CAUSE]
Using above image to store the fs, and we got the following result in fs
tree:
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
generation 3 transid 7 size 68 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 2 flags 0x0(none)
item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
index 0 namelen 2 name: ..
item 2 key (256 DIR_ITEM 439756795) itemoff 16062 itemsize 49
location key (257 INODE_ITEM 0) type FILE
transid 7 data_len 0 name_len 19
name: top_secret_filename
item 3 key (256 DIR_ITEM 693462946) itemoff 16017 itemsize 45
location key (258 INODE_ITEM 0) type FILE
transid 7 data_len 0 name_len 15
name: secret_filename
item 4 key (256 DIR_INDEX 2) itemoff 15968 itemsize 49
location key (257 INODE_ITEM 0) type FILE
transid 7 data_len 0 name_len 19
name: top_secret_filename
item 5 key (256 DIR_INDEX 3) itemoff 15923 itemsize 45
location key (258 INODE_ITEM 0) type FILE
transid 7 data_len 0 name_len 15
name: secret_filename
item 6 key (257 INODE_ITEM 0) itemoff 15763 itemsize 160
generation 7 transid 7 size 0 nbytes 0
block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
sequence 1 flags 0x0(none)
item 7 key (257 INODE_REF 256) itemoff 15734 itemsize 29
index 2 namelen 19 name: top_secret_filename
item 8 key (258 INODE_ITEM 0) itemoff 15574 itemsize 160
generation 7 transid 7 size 0 nbytes 0
block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
sequence 1 flags 0x0(none)
item 9 key (258 INODE_REF 256) itemoff 15549 itemsize 25
index 3 namelen 15 name: 1���'�gc*&R
The result shows, only the last INODE_REF got sanitized, all the
remaining are not touched at all.
This is caused by how we sanitize the filenames:
copy_buffer()
|- memcpy(dst, src->data, src->len);
| This means we copy the whole eb into our buffer already.
|
|- zero_items()
|- sanitize_name()
|- eb = alloc_dummy_eb();
|- memcpy(eb->data, src->data, src->len);
| This means we generate a dummy eb with the same contents of
| the source eb.
|
|- sanitize_dir_item();
| We override the dir item of the given item (specified by the
| slot number) inside our dummy eb.
|
|- memcpy(dst, eb->data, eb->lem);
The last one copy the dummy eb into our buffer, with only the slot
corrupted.
But when the whole work flow hits the next slot, we only corrupt the
next slot, but still copy the whole dummy eb back to buffer.
This means the previous slot would be overwritten by the old unsanitized
data.
Resulting only the last slot is corrupted.
[FIX]
Fix the bug by only copying back the corrupted item to the buffer.
So that other slots won't be overwritten by unsanitized data.
Reported-by: Andrea Gelmini <andrea.gelmini@gmail.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
The filename sanitization is not recommended as it introduces mismatches
between DIR_ITEM and INODE_REF.
Even hash collision mode (double "-s" option) is not ensured to always
find a hash collision, and when fails to find one, a mismatch happens.
And when a mismatch happens, the kernel will not resolve the path
correctly since kernel uses the hash from DIR_ITEM to lookup the child
inode.
So add a warning into the "-s" option of btrfs-image.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Add exceptions that should not be reported as typos for a reason (names,
abbreviations, preferred other spelling).
Author: Yaroslav Halchenko <debian@onerussian.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Run spellchecking on devel branch on push or on a pull-request.
Author: Yaroslav Halchenko <debian@onerussian.com>
Signed-off-by: David Sterba <dsterba@suse.com>
To ensure that package indexes are up to date.
That should help to avoid recent failed CI runs, which failed to install
certain packages as local cache is out-of-date and remote mirrors no
longer provide that specific (and out-of-date) version of package:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev-dev_255.4-1ubuntu8.1_amd64.deb 404 Not Found [IP: 52.147.219.192 80]
Signed-off-by: Yaroslav Halchenko <debian@onerussian.com>
[ Minor modification on the commit message. ]
Signed-off-by: Qu Wenruo <wqu@suse.com>
[ Move cache update to a separate command. ]
Signed-off-by: David Sterba <dsterba@suse.com>
The function btrfs_mksubvol() is very different between btrfs-progs and
kernel, the former version is really just linking a subvolume to another
directory inode, but the kernel version is really to make a completely
new subvolume.
Instead of same-named function, introduce btrfs_link_subvolume() and use
it to replace the old btrfs_mksubvol().
This is done by:
- Introduce btrfs_link_subvolume()
Which does extra checks before doing any modification:
* Make sure the target inode is a directory
* Make sure no filename conflict
Then do the linkage:
* Add the dir_item/dir_index into the parent inode
* Add the forward and backward root refs into tree root
- Introduce link_image_subvolume() helper
Currently btrfs_mksubvol() has a dedicated convert filename retry
behavior, which is unnecessary and should be done by the convert code.
Now move the filename retry behavior into the helper.
- Remove btrfs_mksubvol()
Since there is only one caller utilizing btrfs_mksubvol(), and it's
now gone, we can remove the old btrfs_mksubvol().
Signed-off-by: Qu Wenruo <wqu@suse.com>
There are two different subvolume/data reloc tree creation routines:
- create_subvol() from convert/main.c
* calls btrfs_copy_root() to create an empty root
This is not safe, as it relies on the source root to be empty.
* calls btrfs_read_fs_root() to add it to the cache and trace it
properly
* calls btrfs_make_root_dir() to initialize the empty new root
- create_data_reloc_tree() from mkfs/main.c
* calls btrfs_create_tree() to create an empty root
* Manually add the root to fs_root cache
This is only safe for data reloc tree as it's never updated
inside btrfs-progs.
But not safe for other subvolume trees.
* manually setup the root dir
Both have their good and bad aspects, so here we introduce a new helper,
btrfs_make_subvolume():
- Calls btrfs_create_tree() to create an empty root
- Calls btrfs_read_fs_root() to setup the cache and tracking properly
- Calls btrfs_make_root_dir() to initialize the root dir
- Calls btrfs_update_root() to reflect the rootdir change
So this new helper can replace both create_subvol() and
create_data_reloc_tree().
Signed-off-by: Qu Wenruo <wqu@suse.com>
The list-chunk command is deemed to be reasonably complete so make it
visible in the default build. The output can be tweaked later.
Issue: #559
Signed-off-by: David Sterba <dsterba@suse.com>
The long options now allow to pass the unit mode in the usual way, drop
the local variable for raw byte values.
Signed-off-by: David Sterba <dsterba@suse.com>
Filenames can contain a newline (or other funny characters), this makes
the dump-tree output confusing, same for xattr names or values that can
binary data. Encode the special characters in the C-style ('\e' ->
"\e", or \NNN if there's no single letter representation). This is based
on the isprint() as it's espected either on a terminal or in a dump
file.
Issue: #350
Issue: #407
Signed-off-by: David Sterba <dsterba@suse.com>
The string escaping functionality is more generic and can be used in
other commands (e.g. in dump-tree). Move it to the string utils.
Signed-off-by: David Sterba <dsterba@suse.com>
Codespell discovered a typo in tests that's actually a bug in filename
that is supposed to be created and written to.
=== Do not change lines below ===
{
"chain": [],
"cmd": "git-sedi compex complex",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
Pull-request: #846
Author: Yaroslav Halchenko <debian@onerussian.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit [1] 164bc10d "btrfs-progs: add musl compatibility for printf
format %pV" added a logic to detect the presence of the glibc
<printf.h> header, and if present, to use the
register_printf_specifier() and register_printf_modifier() functions.
The original intent (as the commit log suggests), was to support the
musl libc, which does not provides this <printf.h> header.
When compiling with another libc, such as uClibc-ng, btrfs-progs fail
to build with error:
common/messages.c: In function 'print_va_format':
common/messages.c:51:19: error: 'const struct printf_info' has no member named 'user'
51 | if (!(info->user & va_modifier))
| ^~
common/messages.c: In function 'btrfs_no_printk':
common/messages.c:76:17: warning: implicit declaration of function 'register_printf_specifier'; did you mean 'register_printf_function'? [-Wimplicit-function-declaration]
76 | register_printf_specifier('V', print_va_format,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| register_printf_function
common/messages.c:78:31: warning: implicit declaration of function 'register_printf_modifier'; did you mean 'register_printf_function'? [-Wimplicit-function-declaration]
78 | va_modifier = register_printf_modifier(L"p");
| ^~~~~~~~~~~~~~~~~~~~~~~~
| register_printf_function
This is because uClibc-ng provides a <printf.h> header, but not the
register_printf_specifier() and register_printf_modifier() functions.
See [2]. It mainly includes register_printf_function(). uClibc-ng
emulates an older glibc behavior. Glibc added support for printf user
elements in commit [3] (first included in glibc-2.10, in 2009). Checking
only the <printf.h> is not sufficient.
This commit fixes this build issue by refining the detection logic of
the <printf.h> functions required by btrfs-progs.
[1] 164bc10dfc
[2] https://gogs.waldemar-brodkorb.de/oss/uclibc-ng/src/v1.0.49/include/printf.h
[3] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=9d26efa90c6dcbcd6b3e586c9927b6058ef4d529
Pull-request: #843
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit [1] 3a1d4aa089 "btrfs-progs: fix fallthrough cases with proper
attributes" introduced a macro "fallthrough" to better handle compiler
warnings of fallthrough situations.
This macro is defined using the "__has_attribute" built-in
function-like macro, which was introduced in GCC 5. See [2]. It then
test for the "__fallthrough__" attribute, which was introduced in
GCC 7. See [3].
When compiling with a gcc version which supports "__has_attribute" and
not the "__fallthrough__" attribute, compilation fails with error
message:
common/format-output.c: In function 'print_escaped':
common/format-output.c:78:4: error: 'fallthrough' undeclared (first use in this function)
fallthrough;
^
btrfs-progs claim to support gcc at minimal version 4.8 in [4].
This commit fixes this issue by adding the missing definition.
The definition of the unsupported case is duplicated, because testing
for "__has_attribute" and an attribute at the same time is not
portable. See the cpp "__has_attribute" documentation [5].
Note: the issue was found with Buildroot Linux [6], while testing with
the command "utils/test-pkg -a -p btrfs-progs".
[1] 3a1d4aa089
[2] https://gcc.gnu.org/gcc-5/changes.html
[3] https://gcc.gnu.org/gcc-7/changes.html
[4] https://github.com/kdave/btrfs-progs/tree/v6.9.2#build-compatibility
[5] https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005fattribute.html
[6] https://buildroot.org/
Pull-request: #842
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Similar to the manual page on terminal preview, also do the html output
that will be on the RTD page. Due to the way how it's displayed in the
CI action summary the CSS is missing and there are some visual
artifacts, e.g. in the option lists or special characters.
Issue: #824
Signed-off-by: David Sterba <dsterba@suse.com>
tree-stats currently displays only some global trees and fs-tree 5. Add
support to show the stats of a specified tree.
Issue: #268
Signed-off-by: Chung-Chiang Cheng <cccheng@synology.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Extend CI workflow of devel branch to generate manual page preview as it
would be rendered in a terminal. The output is in the workflow summary
page, for each file changed (if any).
Issue: #824
Signed-off-by: David Sterba <dsterba@suse.com>
This makes btrfs scrub status --si show the Rate in metric units as well.
Before:
Total to scrub: 877.65GB
Rate: 609.22MiB/s
After:
Total to scrub: 877.65GB
Rate: 638.81MB/s
Pull-request: #832
Author: Ivan Kozik <ivan@ludios.org>
Signed-off-by: David Sterba <dsterba@suse.com>
Gcc 13.3 reports:
In file included from image/main.c:41:
./image/metadump.h:47:1: warning: ‘extern’ is not at beginning of declaration [-Wold-style-declaration]
47 | const extern struct dump_version *current_version;
| ^~~~~
Signed-off-by: David Sterba <dsterba@suse.com>
There's a number of optionally built features, print the list on the 2nd
line of 'btrfs version'. The format is same as for e.g. systemd:
$ btrfs version
btrfs-progs v6.9.2
+EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
Signed-off-by: David Sterba <dsterba@suse.com>
The workflow defined in ci-build-test.yml builds a fresh docker image,
this could update packages that are not yet in my docker hub source image.
The purpose of this workflow is to test namely backward compatibility
(LTS distros) so there are no expected differences regarding package
versions.
There's another workflow with the same goal but based on different
docker images, that are updated on a few-weekly basis (updaed by
ci/images-* scripts). This is much faster and can be also enabled for
pull requests.
Remove the original workflow as it's not useful anymore.
Signed-off-by: David Sterba <dsterba@suse.com>
Inside delete_old_data_csums(), after calling btrfs_search_slot() there
is no error handling at all.
Fix it by doing a proper error detection and abort the current transaction.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance the sorting capabilities of 'inspect list-chunks' to allow
multiple keys. Drop the gaps, this works only for pstart and it's hard
to make it work with arbitrary sort keys.
Usage is printed by default, assuming this is an interesting info and
even if it slows down the output (due to extra lookups) it's more
convenient to print it rather than not.
The options related to usage and empty were removed.
Output changes:
- rename Number to PNumber, meaning physical number on the device
- print Devid, device number, can be also sort key
Examples:
btrfs inspect list-chunks /mnt
btrfs inspect list-chunks --sort length,usage
btrfs inspect list-chunks --sort lstart
Depending on the sort key order, the output can be wild, for that the
PNumber and LNumber give some hint where the chunks lie in their space.
Example output:
$ sudo ./btrfs inspect list-chunks --sort length,usage /
Devid PNumber Type/profile PStart Length PEnd LNumber LStart Usage%
----- ------- ----------------- --------- --------- --------- ------- --------- ------
1 7 Data/single 1.52GiB 16.00MiB 1.54GiB 69 191.68GiB 86.04
1 3 System/DUP 117.00MiB 32.00MiB 149.00MiB 40 140.17GiB 0.05
1 2 System/DUP 85.00MiB 32.00MiB 117.00MiB 39 140.17GiB 0.05
1 15 Data/single 8.04GiB 64.00MiB 8.10GiB 61 188.60GiB 94.46
1 1 Data/single 1.00MiB 84.00MiB 85.00MiB 68 191.60GiB 74.24
1 5 Metadata/DUP 341.00MiB 192.00MiB 533.00MiB 60 188.41GiB 82.58
1 4 Metadata/DUP 149.00MiB 192.00MiB 341.00MiB 59 188.41GiB 82.58
1 20 Metadata/DUP 9.29GiB 256.00MiB 9.54GiB 38 139.92GiB 57.76
1 19 Metadata/DUP 9.04GiB 256.00MiB 9.29GiB 37 139.92GiB 57.76
1 22 Metadata/DUP 9.79GiB 256.00MiB 10.04GiB 25 113.15GiB 57.93
1 21 Metadata/DUP 9.54GiB 256.00MiB 9.79GiB 24 113.15GiB 57.93
1 46 Metadata/DUP 29.29GiB 256.00MiB 29.54GiB 43 142.71GiB 62.38
Signed-off-by: David Sterba <dsterba@suse.com>
Add parsing of user defined sorting specification and some helpers.
Example usage:
sortdef = "key1,key2,key3"
do {
id = compare_parse_key_to_id(&comp, sortdef)
if (id < 0) return error;
compare_add_sort_id(&comp, id)
} while(id >= 0);
Signed-off-by: David Sterba <dsterba@suse.com>
There's more information available in sysfs
(/sys/fs/btrfs/FSID/allocation) that we can print in 'fi df'. This is
still meant for debugging or deeper analysis of the filesystem, the
values need to be correctly interpreted with respect to the profiles,
persistence and other conditonal features.
The extended output is not printed by default and for now is behind the
verbosity options:
$ btrfs -vv fi df /mnt
Data, single: total=47.06GiB, used=25.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.44GiB, used=961.20MiB
GlobalReserve, single: total=125.62MiB, used=0.00B
Data:
bg_reclaim_threshold 0%
bytes_may_use 8.00KiB
bytes_pinned 0.00B
bytes_readonly 64.00KiB
bytes_reserved 0.00B
bytes_used 25.32GiB
bytes_zone_unusable 0.00B
chunk_size 10.00GiB
disk_total 47.06GiB
disk_used 25.32GiB
total_bytes 47.06GiB
Metadata:
bg_reclaim_threshold 0%
bytes_may_use 126.62MiB
bytes_pinned 0.00B
bytes_readonly 0.00B
bytes_reserved 0.00B
bytes_used 961.20MiB
bytes_zone_unusable 0.00B
chunk_size 256.00MiB
disk_total 2.88GiB
disk_used 1.88GiB
total_bytes 1.44GiB
System:
bg_reclaim_threshold 0%
bytes_may_use 0.00B
bytes_pinned 0.00B
bytes_readonly 0.00B
bytes_reserved 0.00B
bytes_used 16.00KiB
bytes_zone_unusable 0.00B
chunk_size 32.00MiB
disk_total 64.00MiB
disk_used 32.00KiB
total_bytes 32.00MiB
Signed-off-by: David Sterba <dsterba@suse.com>
For an unprivileged subvolume iterator, append_to_search_stack() closes
cur_fd. On the first call to btrfs_util_subvolume_iterator_next(),
cur_fd is equal to the fd that was passed to
btrfs_util_create_subvolume_iterator_fd(). We're not supposed to close
that. We didn't notice it because it's more common to use it through
btrfs_util_create_subvolume_iterator(), which opens its own fd that
should be closed, and because the fd number is often reused internally
by the subvolume iterator.
pop_search_stack() already has a check to avoid closing the passed fd;
add the same check to append_to_search_stack(). Also add a regression
test.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch reverts the following 3 commits:
d0cc40d23a ("btrfs-progs: tests: add test case for ram_bytes detection and repair")
7313573c19 ("btrfs-progs: check: original, detect and repair ram_bytes mismatch")
97bf7a5969 ("btrfs-progs: check: lowmem, detect and repair mismatched ram_bytes")
The problem with the ram_bytes check is, kernel can handle it without
any problem, and the original objective for this is to detect such
problem as I immaturelly believe the problem is fixed.
But it turns out to be incorrect and this check is already causing
problems.
Fix it by doing a full revert for now.
Pull-request: #828
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 4db925911c ("btrfs-progs: use strncpy_null everywhere")
replaced strncpy with strncpy_null, the maximum xattr name length is 255
(current limit), the target buffer is large enough for the whole size so
make sure the last character is also copied.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 4db925911c ("btrfs-progs: use strncpy_null everywhere") did
not properly convert the subvolume name copying to strncpy_null() and
trimmed the last character.
Issue: #829
Signed-off-by: David Sterba <dsterba@suse.com>
