The accelerated crc32c needs to check for two CPU features, the crc32c
instructions is in SSE 4.2 and 'pclmulqdq' is a separate. There's still
old hardware used that does not have the PCLMUL instructions. Detect it
and make it the condition.
The pclmul is not supported on old compilers so also add a
configure-time detection and leave the SSE 4.2 only implementation as
the accelerated one if possible.
Issue: #676
Signed-off-by: David Sterba <dsterba@suse.com>
Copy faster implementation of crc32c from linux kernel as of 6.5-rc7
(x86_64, arch/x86/crypto/crc32c-pcl-intel-asm_64.S). This needs
assembler build support, so detect target architecture so
cross-compilation still works.
Add a special CPU flag so the old and new implementations can be
benchmarked and verified separately.
Sample benchmark:
CPU flags: 0x1ff
CPU features: SSE2 SSSE3 SSE41 SSE42 SHA AVX AVX2 CRC32C_PCL
Block size: 4096
Iterations: 1000000
Implementation: builtin
Units: CPU cycles
NULL-NOP: cycles: 77177218, cycles/i 77
NULL-MEMCPY: cycles: 226313072, cycles/i 226, 62133.395 MiB/s
CRC32C-ref: cycles: 24418596066, cycles/i 24418, 575.859 MiB/s
CRC32C-NI: cycles: 1188335920, cycles/i 1188, 11833.073 MiB/s
CRC32C-PCL: cycles: 463193456, cycles/i 463, 30358.037 MiB/s
XXHASH: cycles: 851606646, cycles/i 851, 16511.916 MiB/s
SHA256-ref: cycles: 74476234956, cycles/i 74476, 188.808 MiB/s
SHA256-NI: cycles: 34198637428, cycles/i 34198, 411.177 MiB/s
BLAKE2-ref: cycles: 14761411664, cycles/i 14761, 952.597 MiB/s
BLAKE2-SSE2: cycles: 18101896796, cycles/i 18101, 776.807 MiB/s
BLAKE2-SSE41: cycles: 12599091062, cycles/i 12599, 1116.087 MiB/s
BLAKE2-AVX2: cycles: 9668247506, cycles/i 9668, 1454.418 MiB/s
The new implementation is about 2.5x faster.
Note: there new version does not work on musl because of linkage
problems (relocations in .rodata), so it's still using the old
implementation.
Signed-off-by: David Sterba <dsterba@suse.com>
Move files from m4/ to config that is also used for build and we can
reduce the number of toplevel directories.
Signed-off-by: David Sterba <dsterba@suse.com>
If I run autoreconf -fi in btrfs-progs repo in VirtualBox, it fails with
possibly undefined macro AC_DEFINE. To fix this, I added m4 as the macro
directory in configure.ac as follows. Adding that line fixes the error
when running autoreconf.
Versions:
autoreconf 2.69
automake 1.15.1
btrfs-progs v6.3-fba31d63
openSUSE 15.5
VirtualBox 6.1.40
configure.ac:7: error: possibly undefined macro: AC_DEFINE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:63: error: possibly undefined macro: AC_MSG_ERROR
configure.ac:276: error: possibly undefined macro: AC_MSG_WARN
autoreconf: /usr/bin/autoconf failed with exit status: 1
Issue: #632
Signed-off-by: David Sterba <dsterba@suse.com>
Glibc provides an interface to extend the printf formats but this is not
standardized and does not work on musl. The code brought from kernel
uses %pV for varargs and also has own implementation of printk.
As a workaround for musl expand the pV value to a string and then
simply print it. The details are hidden behind macros:
- DECLARE_PV(vaf)
- PV_ASSIGN(vaf, format, args)
- PV_FMT in printf string
- PV_VAL in arguments
Signed-off-by: David Sterba <dsterba@suse.com>
The wiki has been archived so remove the links from manual page
footers. Also replace the wiki link by RTD site in configure and
libbtrfsutil.
Signed-off-by: David Sterba <dsterba@suse.com>
Change what hash-speedtest benchmarks according to the
--with-crypto=backend option. Until now it would run the same version
under different names inherited from the builting.
At configure time detect availability of all backends and define all
macros.
Signed-off-by: David Sterba <dsterba@suse.com>
The build fails with crypto backends other than builtin, the
initializers cannot be reached as they're ifdef-ed out. Move
hash_init_accel under the right condition and delete the
algorithm-specific initializers as they're used only by the hash test
and that can simply call hash_init_accel to set the implementation.
All the -m flags need to be detected at configure time and the flag used
for ifdef (HAVE_CFLAG_m*), not the actual feature defined by compiler as
the dispatcher function is not built with the -m flags.
The uname check for x86_64 must be dropped so on i386/i586 we can still
build accelerated version.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 0d5767e664 ("btrfs-progs: build: use -std=gnu11") was supposed
to set C level to gnu11 to match kernel but it was done only in the C=1
(checker) branch. The std used was still gnu90 that can be verified by
running with V=1.
Set the std option to gnu11 unconditionally and delete the m4 macros
that were detecting support on older gcc (4.5). The reference build on
lowest supported distro (Centos 7) compiles fine.
Signed-off-by: David Sterba <dsterba@suse.com>
Copy sha256-x86.c from https://github.com/noloader/SHA-Intrinsics, that
uses the compiler intrinsics to implement the update step with the
native x86_64 instructions.
To avoid dependencies of the reference code and the x86 version, check
runtime support only if the compiler also supports -msha.
Signed-off-by: David Sterba <dsterba@suse.com>
Process an enable_verity cmd by running the enable verity ioctl on the
file. Since enabling verity denies write access to the file, it is
important that we don't have any open write file descriptors.
This also revs the send stream format to version 3 with no format
changes besides the new commands and attributes. This version is not
finalized and commands may change, also this needs to be synchronized
with any kernel changes.
Note: the build is conditional on the header linux/fsverity.h
Signed-off-by: Boris Burkov <boris@bur.io>
Signed-off-by: David Sterba <dsterba@suse.com>
Use the autoconf archive macros for gcc builtin detection and add the
overflow from recently added from kernel.
New:
__builtin_add_overflow
__builtin_sub_overflow
__builtin_mul_overflow
Signed-off-by: David Sterba <dsterba@suse.com>
Now that LZO and ZSTD are optional for not just restore, rename the
build variables to a more generic name and update configure summary.
Signed-off-by: David Sterba <dsterba@suse.com>
LZO as a compression format is pretty archaic these days, there are
better algorithms in all metrics for compression and decompression, and
lzo hasn't had a new release since 2017.
Add an option to disable LZO (defaulting to enabled), and respect it in
cmds/restore.c.
NOTE: disabling support for LZO will make make it impossible to restore
data from filesystems where the compression has ever been used. It's not
recommended to build without the support in general.
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: David Sterba <dsterba@suse.com>
There was a workaround for asciidoc/xmlto build because page btrfs.5 and
btrfs.8 used the same intermediate file. To avoid clash for the section
5 page it is part of the name, this is still needed, but for sphinx we
can use the final name as it will get the right suffix. This affects
only the generated manual pages.
Signed-off-by: David Sterba <dsterba@suse.com>
Update sphinx build so that by default it now builds the manual pages,
in quiet mode. 'make html' builds the html, other sphinx targets are
available, see 'make help'. Installation now works as well.
Note: sphinx is still conditional and must be selected by
ASCIIDOOC_TOOL=sphinx ./configure
Signed-off-by: David Sterba <dsterba@suse.com>
The zoned support requires a header file and some structures for full
support. There are distros that have only part of that and the
autodetection at configure time does not handle that properly, assuming
the user requested the support.
Check if there was any of the --*able-zoned options and if not detect
the support level and fallback to no zoned eventually, not requiring
users to specify --disable-zoned as before.
Issue: #425
Signed-off-by: David Sterba <dsterba@suse.com>
Make it possible to select sphinx doc generator instead of asciidoc so
we don't have two makefiles for that. It's still a bit crude and does
not support installing the files.
The required package is python-Sphinx (or similar name), built by
'sphinx-build'.
Configure:
$ ASCIIDOC_TOOL=sphinx ./configure
...
doc generator: sphinx
...
Generate:
$ cd Documentation/
$ make man
$ make html
$ make info # yes we can have info pages too
There are several more targets provided by sphinx, run 'make' to list
them.
Signed-off-by: David Sterba <dsterba@suse.com>
Previously the build would be optional based on what is installed on the
system. Add an option to make this selectable by user, ie. to allow
build without libudev even if it is installed on the system.
For most users and distros libudev dependency should be ok, but we've
had requests to disable various features (embedded, systems with
non-systemd user space base system) so let's do the same.
Signed-off-by: David Sterba <dsterba@suse.com>
This is needed for future code which will make btrfs-progs' device
scanning logic a little smarter by filtering out path device in
multipath setups. libudev is added as an optional dependency since the
library doesn't have a static version so making it a hard dependency
means forfeiting static build support. To alleviate this a fallback code
will be added for the static build case which doesn't rely on libudev.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Running convert-tests.sh Reported that the 019-ext4-copy-timestamps test
failed:
...
mount -o loop -t ext4 btrfs-progs/tests/test.img btrfs-progs/tests/mnt
====== RUN CHECK touch btrfs-progs/tests/mnt/file
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 700h/1792d Inode: 13 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.999209679 +0800
Modify: 2021-08-24 22:10:21.999209679 +0800
Change: 2021-08-24 22:10:21.999209679 +0800
...
btrfs-progs/btrfs-convert btrfs-progs/tests/test.img
...
====== RUN CHECK mount -t btrfs -o loop btrfs-progs/tests/test.img btrfs-progs/tests/mnt
====== RUN CHECK stat btrfs-progs/tests/mnt/file
File: 'btrfs-progs/tests/mnt/file'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 2ch/44d Inode: 267 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: unconfined_u:object_r:unlabeled_t:s0
Access: 2021-08-24 22:10:21.000000000 +0800
Modify: 2021-08-24 22:10:21.000000000 +0800
Change: 2021-08-24 22:10:21.000000000 +0800
...
atime on converted inode does not match
test failed for case 019-ext4-copy-timestamps
Obviously, the log says that btrfs-convert does not support nanoseconds.
I looked at the source code and found that only if ext2_fs.h defines
EXT4_EPOCH_MASK btrfs-convert to support nanoseconds. But in e2fsprogs,
EXT4_EPOCH_MASK was introduced in v1.43, but in some older versions,
such as v1.40, e2fsprogs actually supports nanoseconds. It seems that if
struct ext2_inode_large contains the i_atime_extra member, ext4 is
supports nanoseconds, so I updated the logic to determine whether the
current ext4 file system supports nanosecond precision. In addition, I
imported some definitions to encode and decode tv_nsec (copied from
e2fsprogs source code).
Author: Li Zhang <zhanglikernel@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add --enable-experimental configure option that allows to merge unstable
features or partially implemented features. This is supposed to help
features that need time to settle, tweak output or formatting and would
require constant rebases and would have limited exposure to users that
could provide feedback.
If this is enabled, the following may change without notice:
- the whole feature may disappear in the future
- new command names could change or relocate to other subcommands
- parameter names
- output formatting
- json output
Signed-off-by: David Sterba <dsterba@suse.com>
The build now fails on older distros that have incomplete support for
zoned mode. Lack of blkzoned.h will automatically skip it. The
member blk_zone.capacity has been added in 5.9 and this would fail the
build, similarly BLKGETZONESZ.
Check each of them separately and fail the build unless --disable-zoned
is set. Build verified on Leap 15.2, Centos 7/8.
Signed-off-by: David Sterba <dsterba@suse.com>
If the kernel supports zoned block devices, the file
/usr/include/linux/blkzoned.h will be present. Check this and define
BTRFS_ZONED if the file is present.
If it present, enables ZONED feature, if not disable it.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Autoheader uses the AC_DEFINE macros (and a few others) to populate
the config.h.in file. The autotools documentation does not tell
what happens if AC_DEFINE is used twice for the same identifier.
This patch prevents using AC_DEFINE twice for
HAVE_OWN_FIEMAP_EXTENT_DEFINE, preserving the logic (using the
fact that an undefined identifier in a preprocessor directive is
taken as zero).
Signed-off-by: Pierre Labastie <pierre.labastie@neuf.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit b3df561fbf ("btrfs-progs: convert: copy extra timespec on
ext4") has introduced the ability to convert extended inode time
precision on ext4, but this breaks builds on older distros, where ext4
does not have the nsec time precision.
Commit c615287cc0 ("btrfs-progs: a bunch of typo fixes") tried to fix
that by testing the availability of the EXT4_EPOCH_MASK macro, but the
test is not complete.
This patch aims at fixing the macro test, and changes the
name of the associated HAVE_ macro, since the logic is reverted.
This fixes#353 when ext4 has nsec time precision. Note that the test
convert/019-ext4-copy-timestamps fails when ext4 does not have the nsec
time precision and needs to check for the support.
Issue: #353
Signed-off-by: Pierre Labastie <pierre.labastie@neuf.fr>
Signed-off-by: David Sterba <dsterba@suse.com>
Hard-coding the pkg-config executable might result in build errors
on system and cross environments that have prefixed toolchains. The
PKG_CONFIG variable already holds the proper one and is already used
in a few other places.
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Signed-off-by: Heiko Becker <heirecka@exherbo.org>
Signed-off-by: David Sterba <dsterba@suse.com>
In 5.10 the convert gained support for extended inode time precision,
but this is not available on older distros and breaks build. Add a
configure-time check for the EXT4_EPOCH_MASK macro and add a stub in
case it's not detected.
This means that the 64bit timestamps will not be transferred from the
original filesystem in such environment, at least a warning is printed.
Issue: #344
Signed-off-by: David Sterba <dsterba@suse.com>
This reverts commit 61ecaff036.
The libmount functionality is not used anymore, we can remove it
entirely.
Signed-off-by: David Sterba <dsterba@suse.com>
API provided by libmount allows to read various information from /proc
files about mount paths.
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add .in substitutions for main package version, split and compound
without the 'v' prefix. BTRFS_VERSION_PATCHLEVEL could be empty.
Signed-off-by: David Sterba <dsterba@suse.com>
Add pkg-config file for libbtrfsutil which can be used to set up
compilation and link flags.
The paths in pc file depend on prefix variable but ignore DESTDIR.
DESTDIR is used for packaging and it should not affect the paths in pc
file.
Issue: #285
Signed-off-by: Sheng Mao <shngmao@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Update the summary after configure finishes with versions of all
projects that are now hosted in this repository. Mark libbtrfs as
deprecated do discourage new use, formal deprecation will happen in the
future.
Signed-off-by: David Sterba <dsterba@suse.com>
https://github.com/smuellerDD/libkcapi allows user-space to access the
Linux kernel crypto API. Uses netlink interface and exports easy to use
APIs.
Signed-off-by: David Sterba <dsterba@suse.com>
For environments that require certified implementations of cryptographic
primitives allow to select a library providing them. The requirements
are SHA256 and BLAKE2 (with the 2b variant and 256 bit digest).
For now there are two: libgrcrypt and libsodium (openssl does not
provide the BLAKE2b-256). Accellerated versions are typically provided
and automatically selected.
Signed-off-by: David Sterba <dsterba@suse.com>
The manual pages are not compressed anymore and we can remove gzip from
build dependencies and build steps.
Signed-off-by: David Sterba <dsterba@suse.com>
The build system mentioned in the previous commit builds libraries in
both PIC and non-PIC mode. Shared libraries don't work in PIC mode, so
it expects a --disable-shared configure option, which most open source
libraries using autoconf have. Let's add it, too.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We have a build system internally which only needs to build and install
the libraries out of a repository, not any binaries. There's no easy way
to do this in btrfs-progs currently. Add --disable-programs to
./configure to support this.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: David Sterba <dsterba@suse.com>
GCC releases prior to 4.5.0 don't support -std=gnu90 so btrfs-progs won't
build at all on older distros. We can detect whether the compiler
supports -std=gnu90 and fall back to -std=gnu89 if it doesn't.
AX_CHECK_COMPILE_FLAG is the right way to do this, but it depends on
autoconf 2.64. AX_GCC_VERSION has been deprecated, so we'll use that
only for earlier autoconf versions so we can drop it when we drop
support for older autoconf releases.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 2e1932e6a3 (btrfs-progs: build: simplify version tracking)
started m4_chomp to strip the newlines from the version file. m4_chomp
was introduced in autoconf 2.64 but SLE11 ships with autoconf 2.63.
For purposes of just stripping the newline, m4_flatten is sufficient.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 324d4c1857 (btrfs-progs: convert: Add larger device support)
introduced new dependencies on the 64-bit API provided by e2fsprogs.
That API was introduced in v1.42 (along with bigalloc).
This patch maps the following to their equivalents in e2fsprogs < 1.42.
- ext2fs_get_block_bitmap_range2
- ext2fs_inode_data_blocks2
- ext2fs_read_ext_attr2
Since we need to detect and define EXT2_FLAG_64BITS for compatibilty
anyway, it makes sense to use that to detect the older e2fsprogs instead
of defining a new flag ourselves.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
AC_PATH_PROG won't fail even if it fails to find command path.
xmlto is required for document build and we should report error
if it doesn't exist at configure time.
Signed-off-by: Tomohiro Misono <misono.tomohiro@jp.fujitsu.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba
Nathan Mills
Boris Burkov
Ross Burton
Nikolay Borisov
Li Zhang
Naohiro Aota
Pierre Labastie
Heiko Becker
Marcos Paulo de Souza
Sheng Mao
Adam Borowski
Josh Soref
Omar Sandoval
Jeff Mahoney
Misono Tomohiro