Commit Graph

44 Commits

Author SHA1 Message Date
Josef Bacik
8069b8b8cd btrfs-progs: drop btrfs_init_path
This simply zero's out the path, and this is used everywhere we use a
stack path.  Drop this usage and simply init the path's to empty instead
of using a function to do the memset.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-10-03 01:11:56 +02:00
David Sterba
ae73e89f28 btrfs-progs: mkfs: more verbose output for --rootdir
Print the source directory for --rootdir and if --shrink is used. With
-vv then print the individual files as added:

  $ mkfs.btrfs --rootdir dir --shrink -vv img
  ...
  Rootdir from:       Documentation
  ADD: /btrfs-progs/Documentation/btrfs-check.rst
  ...
  ADD: /btrfs-progs/Documentation/btrfs-send.rst
    Shrink:           yes
  Label:              (null)
  UUID:               40d3a16f-02d8-40d7-824b-239cee528093
  ...

The 'Rootdir from' is printed before the files are added so there's now
message before the files are added which could take some time.

Issue: #627
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 22:17:33 +02:00
Qu Wenruo
08a3bd7694 btrfs-progs: tune: add the ability to generate new data checksums
This patch would modify btrfs_csum_file_block() to handle csum type
other than the one used in the current fs.

The new data checksum would use a different objectid (-13) to
distinguish with the existing one (-10).
This needs to change tree-checker to skip the item size checks,
since new csum can be larger than the original csum.

After this stage, the resulted csum tree would look like this:

	item 0 key (CSUM_CHANGE EXTENT_CSUM 13631488) itemoff 8091 itemsize 8192
		range start 13631488 end 22020096 length 8388608
	item 1 key (EXTENT_CSUM EXTENT_CSUM 13631488) itemoff 7067 itemsize 1024
		range start 13631488 end 14680064 length 1048576

Note the itemsize is 8 times the original one, as the original csum is
CRC32, while target csum is SHA256, which is 8 times the size.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:32 +02:00
Qu Wenruo
46364d3766 btrfs-progs: replace write_and_map_eb() by write_data_to_disk()
The function write_and_map_eb() is quite abused as a way to write any
generic buffer back to disk.

But we have a more suitable function already, write_data_to_disk().

This patch would remove the abused write_data_to_disk() calls, and
convert the only three valid call sites to write_data_to_disk() instead.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:31 +02:00
Josef Bacik
f8efe9f724 btrfs-progs: sync file-item.h into progs
This patch syncs file-item.h into btrfs-progs.  This carries with it an
API change for btrfs_del_csums, which takes a root argument in the
kernel, so all callsites have been updated accordingly.

I didn't sync file-item.c because it carries with it a bunch of bio
related helpers which are difficult to adapt to the kernel.
Additionally there's a few helpers in the local copy of file-item.c that
aren't in the kernel that are required for different tools.

This requires more cleanups in both the kernel and progs in order to
sync file-item.c, so for now just do file-item.h in order to pull things
out of ctree.h.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-05-26 18:02:29 +02:00
psykose
c9abbf6264 btrfs-progs: stop using legacy *64 interfaces
The *64 interfaces, such as fstat64, off64_t, etc, are legacy interfaces
created at a time when 64-bit file support was still new. They are
generally exposed when defining a macro named _LARGEFILE64_SOURCE, as
e.g. the glibc docs[0] say.

The modern way to utilise largefile support, is to continue to use the
regular interfaces (off_t, fstat, ..), and define _FILE_OFFSET_BITS=64.

We already use the autoconf macro AC_SYS_LARGEFILE[1] which arranges this
and sets this macro for us. Therefore, we can utilise the non-64 names
without fear of breaking on 32-bit systems.

This fixes the build against musl libc, ever since musl dropped the
*64 compat from interfaces by default[2] just for _GNU_SOURCE, unless
_LARGEFILE64_SOURCE is defined. However, there are plans for a future
removal of the whole *64 header API, and that workaround (adding another
define) might cease to exist.

So, rename all *64 API use to the regular non-suffixed names. For
consistency, rename the internal functions that were *64 named
(lstat64_path, ..) too.

This should have no regressions on any platform.

[0]: https://www.gnu.org/software/libc/manual/html_node/Feature-Test-Macros.html#index-_005fLARGEFILE64_005fSOURCE
[1]: https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/System-Services.html
[2]: 25e6fee27f

Pull-request: #615
Signed-off-by: psykose <alice@ayaya.dev>
Signed-off-by: David Sterba <dsterba@suse.com>
2023-04-25 16:59:42 +02:00
Qu Wenruo
2cdc8dddbf btrfs-progs: mkfs: offset inode numbers of the source filesystem
[BUG]
When running mkfs tests on a newly rebooted minimal system, it can cause
mkfs/009 to fail.

The reproduce steps requires /tmp to has minimal files in the first
place.

  # mkdir /tmp/rootdir
  # xfs_io -f -c "pwrite 0 16k" /tmp/rootdir
  # mkfs.btrfs --rootdir /tmp/rootdir -f $dev
  # btrfs check $dev
  Opening filesystem to check...
  Checking filesystem on /dev/test/scratch1
  UUID: 6821b3db-f056-4c18-b797-32679dcd4272
  [1/7] checking root items
  [2/7] checking extents
  data backref 13631488 root 5 owner 170 offset 0 num_refs 0 not found in extent tree
  incorrect local backref count on 13631488 root 5 owner 170 offset 0 found 1 wanted 0 back 0x55ff6cd72260
  backref 13631488 root 5 not referenced back 0x55ff6cd4c1f0
  incorrect global backref count on 13631488 found 2 wanted 1
  backpointer mismatch on [13631488 16384]
  ERROR: errors found in extent allocation tree or chunk allocation

[CAUSE]
The extent tree has the following weird item:

	item 0 key (13631488 EXTENT_ITEM 16384) itemoff 16250 itemsize 33
		refs 1 gen 0 flags DATA
		tree block backref root FS_TREE

This is an extent item for data, thus it should not have an inline tree
backref.

Then checking the fs tree:

	item 0 key (170 INODE_ITEM 0) itemoff 16123 itemsize 160
		generation 7 transid 0 size 16384 nbytes 16384
		block group 0 mode 100600 links 1 uid 1000 gid 1000 rdev 0
		sequence 0 flags 0x0(none)
		atime 1664866393.0 (2022-10-04 14:53:13)
		ctime 1664863510.0 (2022-10-04 14:05:10)
		mtime 1664863455.0 (2022-10-04 14:04:15)
		otime 0.0 (1970-01-01 08:00:00)

There is an inode item before the root dir inode.

And that inode number 170 is causing the problem.

In traverse_directory(), we use the inode number reported from stat()
directly as btrfs inode number, and pass it to
btrfs_record_file_extent(), which finally calls btrfs_inc_extent_ref(),
with above 170 passed as @owner parameter.

But inside btrfs_inc_extent_ref() we use that @owner value to determine
if it's a data backref.
Since we got a smaller than BTRFS_FIRST_FREE_OBJECTID, btrfs treats it
as tree block, and cause the above problem.

[FIX]
As a quick fix, always add BTRFS_FIRST_FREE_OBJECTID to all inode number
directly grabbed from stat().

And add an ASSERT() in __btrfs_record_file_extent() to catch unexpected
objectid.

This is not a perfect solution, as the resulted fs will has a huge gap
in its inodes:

	item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
	item 4 key (426 INODE_ITEM 0) itemoff 15883 itemsize 160

For a proper fix, we should allocate new btrfs inode numbers in a
sequential order, but that would be another series of patches.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
ea0b894967 btrfs-progs: mkfs: do proper error handling
Replace BUG_ON after transaction start failures, all the functions
already handle errors and return them to the caller. The other error
handling is for impossible conditions.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
a827bb2db8 btrfs-progs: use template for transaction commit error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
8fcafae04a btrfs-progs: use template for transaction start error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:10 +02:00
David Sterba
c2be0e2ce0 btrfs-progs: use template for out of memory error messages
Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
b73a29936a btrfs-progs: remove unnecessary casts for u64
The (unsigned long long) type casts can be dropped, printf understands
%llu and u64 and does not warn. In cases where the type is not u64 keep
the cast.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:08:09 +02:00
David Sterba
0f03da53cc btrfs-progs: mkfs: update include lists
The tool IWYU (include what you use) suggests to remove and add some
includes.

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:12 +02:00
David Sterba
61de520917 btrfs-progs: mkfs: reorder includes
The preferred order:
- system headers
- standard headers
- libraries
- kernel library
- kernel shared
- common headers
- other tools
- own headers

Signed-off-by: David Sterba <dsterba@suse.com>
2022-10-11 09:06:11 +02:00
Josef Bacik
08b63c0fc5 btrfs-progs: stop passing root to csum related functions
We are going to need to start looking up the csum root based on the
bytenr with extent tree v2.  To that end stop passing the root to the
csum related functions so that can be done in the helper functions
themselves.

There's an unrelated deletion of a function prototype that no longer
exists.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-11-22 21:45:37 +01:00
Naohiro Aota
40ab7530df btrfs-progs: set eb::fs_info properly everywhere
Several extent_buffer initializations miss fs_info initialization. This
is OK before the following patch ("btrfs-progs: use direct-io for zoned
device") as eb->fs_info is not always necessary. But, after that patch,
we will use fs_info to determine it is zoned or not and that causes
segfault in such cases.

Properly set fs_info when initializing extent_buffers to fix the issue.

Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2021-10-08 20:47:04 +02:00
David Sterba
522945efc8 btrfs-progs: remove unused prototypes from send-utils.h
The functions are part of path-utils.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-10-08 20:47:04 +02:00
David Sterba
c3ee6a8a09 btrfs-progs: unify GPL header comments
Add the GPL v2 header to files where it was missing and is not from an
external source, update to the most recent version with the address.

Signed-off-by: David Sterba <dsterba@suse.com>
2021-09-07 13:58:44 +02:00
David Sterba
51f15d393a btrfs-progs: build: remove incomplete android support
There is a support to build on android but it's incomplete and there's
little interest to fix it.

To reinstate we'll need:

* fix remaining issues from
  lore.kernel.org/linux-btrfs/20170802185111.187922-1-filipbystricky@google.com

* find CI host with Android support to verify build, either local eg. in
  docker or in a hosted environment

* switch the make-based build to 'soong' (source.android.com/setup/build)

Issue: #357
Signed-off-by: David Sterba <dsterba@suse.com>
2021-05-06 16:41:47 +02:00
David Sterba
0144bcb713 btrfs-progs: move volumes.c to kernel-shared/
Signed-off-by: David Sterba <dsterba@suse.com>
2020-08-31 17:01:06 +02:00
David Sterba
6069bc52a9 btrfs-progs: move transaction.c to kernel-shared/
Signed-off-by: David Sterba <dsterba@suse.com>
2020-08-31 17:01:06 +02:00
David Sterba
abb670f883 btrfs-progs: move ctree.c to kernel-shared/
Signed-off-by: David Sterba <dsterba@suse.com>
2020-08-31 17:01:05 +02:00
David Sterba
772f0da6df btrfs-progs: move disk-io.c to kernel-shared/
Signed-off-by: David Sterba <dsterba@suse.com>
2020-08-31 17:01:05 +02:00
David Sterba
cdaf906d68 btrfs-progs: move send-utils.c to common/
Signed-off-by: David Sterba <dsterba@suse.com>
2020-08-31 17:01:04 +02:00
Vladimir Panteleev
11e126b162 btrfs-progs: mkfs: fix xattr enumeration
Use the return value of listxattr instead of tokenizing.

The end of the extended attribute list is indicated by the return value,
not an empty list item (two consecutive NULs). Using strtok in this way
thus sometimes caused add_xattr_item to reuse stack data in xattr_list
from the previous invocation, thus querying attributes that are not
actually in the file's xattr list.

Issue: #194
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Vladimir Panteleev <git@vladimir.panteleev.md>
Signed-off-by: David Sterba <dsterba@suse.com>
2019-10-14 17:27:56 +02:00
David Sterba
c07960c8be btrfs-progs: move utils.[ch] to common/
Update include paths and remove some duplicates.

Signed-off-by: David Sterba <dsterba@suse.com>
2019-07-03 20:49:04 +02:00
David Sterba
d1efe50d0a btrfs-progs: move messages.[ch] to common/
Signed-off-by: David Sterba <dsterba@suse.com>
2019-07-03 20:49:03 +02:00
David Sterba
f63f29e9e9 btrfs-progs: move internal.h to common/
Create directory for all sources that can be used by anything that's not
rellated to a relevant kernel part, all common functions, helpers,
utilities that do not fit any other specific category.

The traditional location would be probably lib/ with all things that are
statically linked to the main binaries, but we have libbtrfs and
libbtrfsutil so this would be confusing.

Signed-off-by: David Sterba <dsterba@suse.com>
2019-07-03 20:49:03 +02:00
Josh Soref
b7439eeb3a btrfs-progs: mkfs: fix typo in "multipler" variables
Generated by https://github.com/jsoref/spelling

Issue: #154
Author: Josh Soref <jsoref@users.noreply.github.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-11-26 18:24:49 +01:00
David Sterba
e578b59bf6 btrfs-progs: convert strerror to implicit %m
Similar to the changes where strerror(errno) was converted, continue
with the remaining cases where the argument was stored in another
variable.

The savings in object size are about 4500 bytes:

 $ size btrfs.old btrfs.new
   text    data     bss     dec     hex filename
 805055   24248   19748  849051   cf49b btrfs.old
 804527   24248   19748  848523   cf28b btrfs.new

Signed-off-by: David Sterba <dsterba@suse.com>
2018-10-31 18:24:14 +01:00
Rosen Penev
251d32ea5c btrfs-progs: Eliminate remaining uses of strerror(errno)
%m allows a smaller filesize. Useful on embedded systems.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-10-25 16:11:40 +02:00
Yevgeny Popovych
9ff39d48aa btrfs-progs: mkfs: Fix typos in strings and comments
Signed-off-by: Yevgeny Popovych <yevgenyp@pointgrab.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-06-07 16:37:40 +02:00
Matthias Benkard
db84e5908d btrfs-progs: mkfs: traverse_directory: Reset error code on continue
In case add_inode_items returned -EEXIST, traverse_directory would
handle the condition and still continue under certain circumstances, but
it would not reset the error code, leading to spurious failure later.
This patch fixes that.

Pull-request: #124
Author: Matthias Benkard <matthias.benkard@egym.de>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-06-07 16:37:40 +02:00
Misono Tomohiro
eb9dd12d0e btrfs-progs: mkfs rootdir: use lgetxattr() not to follow a symbolic link
mkfs-test 016 "rootdir-bad-symbolic-link" fails when selinux is enabled.
This is because add_xattr_item() uses getxattr() and tries to follow a
bad symbolic link for selinux item, which causes ENOENT error.

The line above already uses llistxattr() for getting list of xattr in
order not to follow a symbolic link, so just use lgetxattr() too.

Signed-off-by: Tomohiro Misono <misono.tomohiro@jp.fujitsu.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-04-06 15:06:50 +02:00
Qu Wenruo
4e6e5804f2 btrfs-progs: mkfs/rootdir: Don't follow symbolic link when calcuating size
[BUG]
If we have a symbolic link in rootdir pointing to non-existing location,
mkfs.btrfs --rootdir will just fail:
------
$ mkfs.btrfs  -f --rootdir /tmp/rootdir/ /dev/data/btrfs
btrfs-progs v4.15.1
See http://btrfs.wiki.kernel.org for more information.

ERROR: ftw subdir walk of /tmp/rootdir/ failed: No such file or directory
------

[CAUSE]
Commit 599a0abed5 ("btrfs-progs: mkfs/rootdir: Use over-reserve method
to make size estimate easier") add extra ftw walk to estimate the
filesystem size.

Such default ftw walk will follow symbolic link and gives ENOENT error.

[FIX]
Use nftw() to specify FTW_PHYS so we won't follow symbolic link for size
calculation.

Issue: #109
Reported-by: Alexander Kanavin <alexander.kanavin@intel.com>
Fixes: 599a0abed5 ("btrfs-progs: mkfs/rootdir: Use over-reserve method to make size estimate easier")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-03-30 22:15:55 +02:00
Qu Wenruo
51667947e4 btrfs-progs: mkfs/rootdir: Fix inline extent creation check
Just like convert, we need extra check against sector size for creating
inline extent.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-03-30 22:15:54 +02:00
Qu Wenruo
78c5a90ebf btrfs-progs: mkfs/rootdir: Fix memory leak in traverse_directory()
The bug is exposed by mkfs test case 009, with D=asan.

We are leaking memory of parent_dir_entry->path() which ,except the
rootdir, is allocated by strdup().

Before fixing it, unifiy the allocation of parent_dir_entry() to heap
allocation.

Then fix it by adding "free(parent_dir_entry->path);" in
traverse_directory() and error handler.

Issue: #92
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-02-14 16:31:19 +01:00
Gu Jinxiang
c958110785 btrfs-progs: Sync code with kernel for BTRFS_MAX_INLINE_DATA_SIZE
Do a cleanup. Also make it consistent with kernel.  Use fs_info instead
of root for BTRFS_MAX_INLINE_DATA_SIZE, since maybe in some situation we
do not know root, but just know fs_info.

Change macro to inline function to be consistent with kernel.  And
change the function body to match kernel.

Signed-off-by: Gu Jinxiang <gujx@cn.fujitsu.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-02-02 16:01:57 +01:00
Qu Wenruo
1c9c5f7fb3 btrfs-progs: mkfs: Separate shrink from rootdir
Make --shrink a separate option for --rootdir, and change the default to
off.

The shrinking behaviour is not a commonly used feature but can be useful
for creating minimal pre-filled images, in one step, without requiring
to mount.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ update changelog and error messages ]
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:15:15 +01:00
Qu Wenruo
c28418daed btrfs-progs: mkfs/rootdir: Shrink fs for rootdir option
Use the new dev extent based shrink method for rootdir option. This
restores the original behaviour when --rootdir will create a minimal
filesystem size.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ update changelog ]
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:15:13 +01:00
Qu Wenruo
599a0abed5 btrfs-progs: mkfs/rootdir: Use over-reserve method to make size estimate easier
Use an easier method to calculate the estimate device size for
mkfs.btrfs --rootdir.

The new method will over-estimate, but should ensure we won't encounter
ENOSPC.

It relies on the following data:
1) number of inodes -- for metadata chunk size
2) rounded up data size of each regular inode -- for data chunk size

Total meta chunk size = round_up(nr_inode * (PATH_MAX * 3 + sectorsize),
min_chunk_size) * profile_multiplier

PATH_MAX is the maximum size possible for INODE_REF/DIR_INDEX/DIR_ITEM.
Sectorsize is the maximum size possible for inline extent.
min_chunk_size is 8M for SINGLE, and 32M for DUP, get from
btrfs_alloc_chunk().
profile_multiplier is 1 for Single, 2 for DUP.

Total data chunk size is much easier.
Total data chunk size = round_up(total_data_usage, min_chunk_size) *
profile_multiplier

Total_data_usage is the sum of *rounded up* size of each regular inode
use.
min_chunk_size is 8M for SINGLE, 64M for DUP, get from btrfS_alloc_chunk().
Same profile_multiplier for meta.

This over-estimate calculate is, of course inacurrate, but since we will
later shrink the fs to its real usage, it doesn't matter much now.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ update comments ]
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:15:11 +01:00
Qu Wenruo
0a7a48e1e8 btrfs-progs: mkfs/rootdir: Introduce function to get end position of last device extent
Useful for later 'mkfs.btrfs --rootdir' shrink support.

Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:11:36 +01:00
Qu Wenruo
8719161b4c btrfs-progs: mkfs: move source dir size calculation to its own files
Also rename the function from size_sourcedir() to mkfs_size_dir().

Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:11:33 +01:00
Qu Wenruo
075580471e btrfs-progs: mkfs: move image creation of rootdir to its own files
In fact, --rootdir option is getting more and more independent from
normal mkfs code.

So move image creation function, make_image() and its related code to
mkfs/rootdir.[ch], and rename the function to btrfs_mkfs_fill_dir().

Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
2018-01-08 18:11:24 +01:00