To ensure that package indexes are up to date.
That should help to avoid recent failed CI runs, which failed to install
certain packages as local cache is out-of-date and remote mirrors no
longer provide that specific (and out-of-date) version of package:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev-dev_255.4-1ubuntu8.1_amd64.deb 404 Not Found [IP: 52.147.219.192 80]
Signed-off-by: Yaroslav Halchenko <debian@onerussian.com>
[ Minor modification on the commit message. ]
Signed-off-by: Qu Wenruo <wqu@suse.com>
[ Move cache update to a separate command. ]
Signed-off-by: David Sterba <dsterba@suse.com>
The function btrfs_mksubvol() is very different between btrfs-progs and
kernel, the former version is really just linking a subvolume to another
directory inode, but the kernel version is really to make a completely
new subvolume.
Instead of same-named function, introduce btrfs_link_subvolume() and use
it to replace the old btrfs_mksubvol().
This is done by:
- Introduce btrfs_link_subvolume()
Which does extra checks before doing any modification:
* Make sure the target inode is a directory
* Make sure no filename conflict
Then do the linkage:
* Add the dir_item/dir_index into the parent inode
* Add the forward and backward root refs into tree root
- Introduce link_image_subvolume() helper
Currently btrfs_mksubvol() has a dedicated convert filename retry
behavior, which is unnecessary and should be done by the convert code.
Now move the filename retry behavior into the helper.
- Remove btrfs_mksubvol()
Since there is only one caller utilizing btrfs_mksubvol(), and it's
now gone, we can remove the old btrfs_mksubvol().
Signed-off-by: Qu Wenruo <wqu@suse.com>
There are two different subvolume/data reloc tree creation routines:
- create_subvol() from convert/main.c
* calls btrfs_copy_root() to create an empty root
This is not safe, as it relies on the source root to be empty.
* calls btrfs_read_fs_root() to add it to the cache and trace it
properly
* calls btrfs_make_root_dir() to initialize the empty new root
- create_data_reloc_tree() from mkfs/main.c
* calls btrfs_create_tree() to create an empty root
* Manually add the root to fs_root cache
This is only safe for data reloc tree as it's never updated
inside btrfs-progs.
But not safe for other subvolume trees.
* manually setup the root dir
Both have their good and bad aspects, so here we introduce a new helper,
btrfs_make_subvolume():
- Calls btrfs_create_tree() to create an empty root
- Calls btrfs_read_fs_root() to setup the cache and tracking properly
- Calls btrfs_make_root_dir() to initialize the root dir
- Calls btrfs_update_root() to reflect the rootdir change
So this new helper can replace both create_subvol() and
create_data_reloc_tree().
Signed-off-by: Qu Wenruo <wqu@suse.com>
The list-chunk command is deemed to be reasonably complete so make it
visible in the default build. The output can be tweaked later.
Issue: #559
Signed-off-by: David Sterba <dsterba@suse.com>
The long options now allow to pass the unit mode in the usual way, drop
the local variable for raw byte values.
Signed-off-by: David Sterba <dsterba@suse.com>
Filenames can contain a newline (or other funny characters), this makes
the dump-tree output confusing, same for xattr names or values that can
binary data. Encode the special characters in the C-style ('\e' ->
"\e", or \NNN if there's no single letter representation). This is based
on the isprint() as it's espected either on a terminal or in a dump
file.
Issue: #350
Issue: #407
Signed-off-by: David Sterba <dsterba@suse.com>
The string escaping functionality is more generic and can be used in
other commands (e.g. in dump-tree). Move it to the string utils.
Signed-off-by: David Sterba <dsterba@suse.com>
Codespell discovered a typo in tests that's actually a bug in filename
that is supposed to be created and written to.
=== Do not change lines below ===
{
"chain": [],
"cmd": "git-sedi compex complex",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
Pull-request: #846
Author: Yaroslav Halchenko <debian@onerussian.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit [1] 164bc10d "btrfs-progs: add musl compatibility for printf
format %pV" added a logic to detect the presence of the glibc
<printf.h> header, and if present, to use the
register_printf_specifier() and register_printf_modifier() functions.
The original intent (as the commit log suggests), was to support the
musl libc, which does not provides this <printf.h> header.
When compiling with another libc, such as uClibc-ng, btrfs-progs fail
to build with error:
common/messages.c: In function 'print_va_format':
common/messages.c:51:19: error: 'const struct printf_info' has no member named 'user'
51 | if (!(info->user & va_modifier))
| ^~
common/messages.c: In function 'btrfs_no_printk':
common/messages.c:76:17: warning: implicit declaration of function 'register_printf_specifier'; did you mean 'register_printf_function'? [-Wimplicit-function-declaration]
76 | register_printf_specifier('V', print_va_format,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| register_printf_function
common/messages.c:78:31: warning: implicit declaration of function 'register_printf_modifier'; did you mean 'register_printf_function'? [-Wimplicit-function-declaration]
78 | va_modifier = register_printf_modifier(L"p");
| ^~~~~~~~~~~~~~~~~~~~~~~~
| register_printf_function
This is because uClibc-ng provides a <printf.h> header, but not the
register_printf_specifier() and register_printf_modifier() functions.
See [2]. It mainly includes register_printf_function(). uClibc-ng
emulates an older glibc behavior. Glibc added support for printf user
elements in commit [3] (first included in glibc-2.10, in 2009). Checking
only the <printf.h> is not sufficient.
This commit fixes this build issue by refining the detection logic of
the <printf.h> functions required by btrfs-progs.
[1] 164bc10dfc
[2] https://gogs.waldemar-brodkorb.de/oss/uclibc-ng/src/v1.0.49/include/printf.h
[3] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=9d26efa90c6dcbcd6b3e586c9927b6058ef4d529
Pull-request: #843
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit [1] 3a1d4aa089 "btrfs-progs: fix fallthrough cases with proper
attributes" introduced a macro "fallthrough" to better handle compiler
warnings of fallthrough situations.
This macro is defined using the "__has_attribute" built-in
function-like macro, which was introduced in GCC 5. See [2]. It then
test for the "__fallthrough__" attribute, which was introduced in
GCC 7. See [3].
When compiling with a gcc version which supports "__has_attribute" and
not the "__fallthrough__" attribute, compilation fails with error
message:
common/format-output.c: In function 'print_escaped':
common/format-output.c:78:4: error: 'fallthrough' undeclared (first use in this function)
fallthrough;
^
btrfs-progs claim to support gcc at minimal version 4.8 in [4].
This commit fixes this issue by adding the missing definition.
The definition of the unsupported case is duplicated, because testing
for "__has_attribute" and an attribute at the same time is not
portable. See the cpp "__has_attribute" documentation [5].
Note: the issue was found with Buildroot Linux [6], while testing with
the command "utils/test-pkg -a -p btrfs-progs".
[1] 3a1d4aa089
[2] https://gcc.gnu.org/gcc-5/changes.html
[3] https://gcc.gnu.org/gcc-7/changes.html
[4] https://github.com/kdave/btrfs-progs/tree/v6.9.2#build-compatibility
[5] https://gcc.gnu.org/onlinedocs/cpp/_005f_005fhas_005fattribute.html
[6] https://buildroot.org/
Pull-request: #842
Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Similar to the manual page on terminal preview, also do the html output
that will be on the RTD page. Due to the way how it's displayed in the
CI action summary the CSS is missing and there are some visual
artifacts, e.g. in the option lists or special characters.
Issue: #824
Signed-off-by: David Sterba <dsterba@suse.com>
tree-stats currently displays only some global trees and fs-tree 5. Add
support to show the stats of a specified tree.
Issue: #268
Signed-off-by: Chung-Chiang Cheng <cccheng@synology.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Extend CI workflow of devel branch to generate manual page preview as it
would be rendered in a terminal. The output is in the workflow summary
page, for each file changed (if any).
Issue: #824
Signed-off-by: David Sterba <dsterba@suse.com>
This makes btrfs scrub status --si show the Rate in metric units as well.
Before:
Total to scrub: 877.65GB
Rate: 609.22MiB/s
After:
Total to scrub: 877.65GB
Rate: 638.81MB/s
Pull-request: #832
Author: Ivan Kozik <ivan@ludios.org>
Signed-off-by: David Sterba <dsterba@suse.com>
Gcc 13.3 reports:
In file included from image/main.c:41:
./image/metadump.h:47:1: warning: ‘extern’ is not at beginning of declaration [-Wold-style-declaration]
47 | const extern struct dump_version *current_version;
| ^~~~~
Signed-off-by: David Sterba <dsterba@suse.com>
There's a number of optionally built features, print the list on the 2nd
line of 'btrfs version'. The format is same as for e.g. systemd:
$ btrfs version
btrfs-progs v6.9.2
+EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
Signed-off-by: David Sterba <dsterba@suse.com>
The workflow defined in ci-build-test.yml builds a fresh docker image,
this could update packages that are not yet in my docker hub source image.
The purpose of this workflow is to test namely backward compatibility
(LTS distros) so there are no expected differences regarding package
versions.
There's another workflow with the same goal but based on different
docker images, that are updated on a few-weekly basis (updaed by
ci/images-* scripts). This is much faster and can be also enabled for
pull requests.
Remove the original workflow as it's not useful anymore.
Signed-off-by: David Sterba <dsterba@suse.com>
Inside delete_old_data_csums(), after calling btrfs_search_slot() there
is no error handling at all.
Fix it by doing a proper error detection and abort the current transaction.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance the sorting capabilities of 'inspect list-chunks' to allow
multiple keys. Drop the gaps, this works only for pstart and it's hard
to make it work with arbitrary sort keys.
Usage is printed by default, assuming this is an interesting info and
even if it slows down the output (due to extra lookups) it's more
convenient to print it rather than not.
The options related to usage and empty were removed.
Output changes:
- rename Number to PNumber, meaning physical number on the device
- print Devid, device number, can be also sort key
Examples:
btrfs inspect list-chunks /mnt
btrfs inspect list-chunks --sort length,usage
btrfs inspect list-chunks --sort lstart
Depending on the sort key order, the output can be wild, for that the
PNumber and LNumber give some hint where the chunks lie in their space.
Example output:
$ sudo ./btrfs inspect list-chunks --sort length,usage /
Devid PNumber Type/profile PStart Length PEnd LNumber LStart Usage%
----- ------- ----------------- --------- --------- --------- ------- --------- ------
1 7 Data/single 1.52GiB 16.00MiB 1.54GiB 69 191.68GiB 86.04
1 3 System/DUP 117.00MiB 32.00MiB 149.00MiB 40 140.17GiB 0.05
1 2 System/DUP 85.00MiB 32.00MiB 117.00MiB 39 140.17GiB 0.05
1 15 Data/single 8.04GiB 64.00MiB 8.10GiB 61 188.60GiB 94.46
1 1 Data/single 1.00MiB 84.00MiB 85.00MiB 68 191.60GiB 74.24
1 5 Metadata/DUP 341.00MiB 192.00MiB 533.00MiB 60 188.41GiB 82.58
1 4 Metadata/DUP 149.00MiB 192.00MiB 341.00MiB 59 188.41GiB 82.58
1 20 Metadata/DUP 9.29GiB 256.00MiB 9.54GiB 38 139.92GiB 57.76
1 19 Metadata/DUP 9.04GiB 256.00MiB 9.29GiB 37 139.92GiB 57.76
1 22 Metadata/DUP 9.79GiB 256.00MiB 10.04GiB 25 113.15GiB 57.93
1 21 Metadata/DUP 9.54GiB 256.00MiB 9.79GiB 24 113.15GiB 57.93
1 46 Metadata/DUP 29.29GiB 256.00MiB 29.54GiB 43 142.71GiB 62.38
Signed-off-by: David Sterba <dsterba@suse.com>
Add parsing of user defined sorting specification and some helpers.
Example usage:
sortdef = "key1,key2,key3"
do {
id = compare_parse_key_to_id(&comp, sortdef)
if (id < 0) return error;
compare_add_sort_id(&comp, id)
} while(id >= 0);
Signed-off-by: David Sterba <dsterba@suse.com>
There's more information available in sysfs
(/sys/fs/btrfs/FSID/allocation) that we can print in 'fi df'. This is
still meant for debugging or deeper analysis of the filesystem, the
values need to be correctly interpreted with respect to the profiles,
persistence and other conditonal features.
The extended output is not printed by default and for now is behind the
verbosity options:
$ btrfs -vv fi df /mnt
Data, single: total=47.06GiB, used=25.32GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.44GiB, used=961.20MiB
GlobalReserve, single: total=125.62MiB, used=0.00B
Data:
bg_reclaim_threshold 0%
bytes_may_use 8.00KiB
bytes_pinned 0.00B
bytes_readonly 64.00KiB
bytes_reserved 0.00B
bytes_used 25.32GiB
bytes_zone_unusable 0.00B
chunk_size 10.00GiB
disk_total 47.06GiB
disk_used 25.32GiB
total_bytes 47.06GiB
Metadata:
bg_reclaim_threshold 0%
bytes_may_use 126.62MiB
bytes_pinned 0.00B
bytes_readonly 0.00B
bytes_reserved 0.00B
bytes_used 961.20MiB
bytes_zone_unusable 0.00B
chunk_size 256.00MiB
disk_total 2.88GiB
disk_used 1.88GiB
total_bytes 1.44GiB
System:
bg_reclaim_threshold 0%
bytes_may_use 0.00B
bytes_pinned 0.00B
bytes_readonly 0.00B
bytes_reserved 0.00B
bytes_used 16.00KiB
bytes_zone_unusable 0.00B
chunk_size 32.00MiB
disk_total 64.00MiB
disk_used 32.00KiB
total_bytes 32.00MiB
Signed-off-by: David Sterba <dsterba@suse.com>
For an unprivileged subvolume iterator, append_to_search_stack() closes
cur_fd. On the first call to btrfs_util_subvolume_iterator_next(),
cur_fd is equal to the fd that was passed to
btrfs_util_create_subvolume_iterator_fd(). We're not supposed to close
that. We didn't notice it because it's more common to use it through
btrfs_util_create_subvolume_iterator(), which opens its own fd that
should be closed, and because the fd number is often reused internally
by the subvolume iterator.
pop_search_stack() already has a check to avoid closing the passed fd;
add the same check to append_to_search_stack(). Also add a regression
test.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch reverts the following 3 commits:
d0cc40d23a ("btrfs-progs: tests: add test case for ram_bytes detection and repair")
7313573c19 ("btrfs-progs: check: original, detect and repair ram_bytes mismatch")
97bf7a5969 ("btrfs-progs: check: lowmem, detect and repair mismatched ram_bytes")
The problem with the ram_bytes check is, kernel can handle it without
any problem, and the original objective for this is to detect such
problem as I immaturelly believe the problem is fixed.
But it turns out to be incorrect and this check is already causing
problems.
Fix it by doing a full revert for now.
Pull-request: #828
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 4db925911c ("btrfs-progs: use strncpy_null everywhere")
replaced strncpy with strncpy_null, the maximum xattr name length is 255
(current limit), the target buffer is large enough for the whole size so
make sure the last character is also copied.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 4db925911c ("btrfs-progs: use strncpy_null everywhere") did
not properly convert the subvolume name copying to strncpy_null() and
trimmed the last character.
Issue: #829
Signed-off-by: David Sterba <dsterba@suse.com>
Remove the not needed encoding and reserved fields in struct
raid_stripe_extent.
This saves 8 bytes per stripe extent.
Note: this is a format change and previously created filesystems with
raid-stripe-tree will not be accessible. Similar patch is needed in
kernel.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Enhance the rescue mount option group by:
- Add a simple explanation on each rescue option
- Add the new 'ignoremetacsums' option
Signed-off-by: Qu Wenruo <wqu@suse.com>
The CI lcov generation fails due to:
Processing ./common/device-utils.gcda
geninfo: ERROR: Unexpected negative count '-6' for /home/runner/work/btrfs-progs/btrfs-progs/common/device-utils.h:69.
Perhaps you need to compile with '-fprofile-update=atomic
(use "geninfo --ignore-errors negative ..." to bypass this error)
Use a safer way to gather the profile stats.
Signed-off-by: David Sterba <dsterba@suse.com>
There's a report on the CI after base ubuntu image update:
geninfo: WARNING:
/home/runner/work/btrfs-progs/btrfs-progs/common/device-scan.c:429:
unexecuted block on non-branch line with non-zero hit count. Use
"geninfo --rc geninfo_unexecuted_blocks=1 to set count to zero.
(use "geninfo --ignore-errors gcov,gcov ..." to suppress this warning)
Signed-off-by: David Sterba <dsterba@suse.com>
Free memory after errors in sum(), this is reported by gcc 13 on the CI.
This was reproduced by:
$ make D=asan TEST=019-receive-clones-on-mounted-subvol test-misc
Signed-off-by: David Sterba <dsterba@suse.com>
Test the following:
- Create a btrfs with crc32c csum
- Populate the filesystem
- Convert the filesystem to the following csums:
* xxhash
* blake2
* sha256
* crc32c
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
'btrfstune --csum' would always fail for a newly created btrfs:
# truncate -s 1G test.img
# ./mkfs.btrfs -f test.img
# ./btrsftune --csum xxhash test.img
WARNING: Experimental build with unstable or unfinished features
WARNING: Switching checksums is experimental, do not use for valuable data!
Proceed to switch checksums
ERROR: failed to start transaction: Unknown error -28
ERROR: failed to start transaction: No space left on device
ERROR: failed to generate new data csums: No space left on device
ERROR: btrfstune failed
[CAUSE]
After commit e79f18a4a7 ("btrfs-progs: introduce a basic metadata free
space reservation check"), btrfs_start_transaction() would check the
metadata space.
But at the time of introduction of csum conversion, the parameter for
btrfs_start_transaction() was incorrect.
The 2nd parameter is the *number* of items to be added (if we're
deleting items, just pass 1).
However commit 08a3bd7694 ("btrfs-progs: tune: add the ability to
generate new data checksums") is using the item size, not the number of
items to be added.
This means we're passing a number 8 * nodesize times larger than the
original size, no wonder we would error out with -ENOSPC.
[FIX]
Use proper calcuation to convert the new csum item size to number of
leaves needed, and double it just in case.
Pull-request: #820
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The separator of key=value is only one or more space character, the
'encoded_write' also uses ',' which is inconsistent with the rest.
Signed-off-by: David Sterba <dsterba@suse.com>
