moving files and directories from $pkgdir to $subpkgdir is a common
pattern, so make a helper function for this.
usage: amove FILESPEC...
FILESPEC is a list of files or patterns/globs that will be exanded by
the shell.
amove will clean up empty directories after moving the files/dirs.
Example usage:
amove 'usr/lib/lib*.a'
amove 'etc/*.d' # moves both etc/conf.d and etc/init.d
amove 'lib/*.so' 'usr/lib/*.so'
cd "$pkgdir"
find usr -name '*.h' | xargs amove
This is based on the work of Chloe Kudryavtsev:
https://github.com/alpinelinux/abuild/pull/92
- set PATH in the rootbld environment so ccache is actually used.
- drop the check for command -v ccache. ccache will be pulled in as
build dependency so we don't need to die if its missing.
- create ~/.ccache if missing rather than die. This directory will
normally be created by ccache itself, but we need to create it so we
can bind mount it incase of rootbld.
- don't die if ccache.conf is missing. ccache will create it.
avoid install ccache and bind mount ~/.ccache when USE_CCACHE is not
set.
This fixes bind mount error when ~/.ccache is missing and USE_CCACHE is
unset.
This introduces basic support for ccache, during packaging builds.
If you are building many packages, it is recommended to manually
increase the maximum size of the ccache cache. This is typically
achieved by modifying `~/.ccache/ccache.conf` and adjusting the
`max_size` setting.
Signed-off-by: Joseph Benden <joe@benden.us>
When you have `-dev` and install `-libs-static`, for example,
it helps to only need to add one to `makedepends` instead of both.
After a grep of the current aports, it turns out matching the
prefix of `subpkgname` will be more useful.
This functionality is no longer needed by the build servers and is broken as it
does not handle
* provides= tags
* automatic dependencies added by trace_apk_deps()
* inter-repository dependencies
* circular dependencies caused by the unit tests in check()
With a recent change in apk [1], virtual packages of the same name will
upgrade each other. Adjust abuild to this by not using the same virtual
package name for two types of dependencies.
This fixes the way crosscompilers are built in postmarketOS [2], which
is essentially the same as running this on Alpine's gcc aport:
$ cd aports/main/gcc
$ C_TARGET_ARCH=armhf CTARGET=armv6-alpine-linux-musleabihf \
BOOTSTRAP=nobuildbase CBUILDROOT=/ abuild -r
...
>>> gcc-armhf: Installing for host:
(1/24) Upgrading .makedepends-gcc-armhf (20190714.104731 -> 20190714.104741)
(2/24) Purging binutils-armhf (2.31.1-r2)
...
[1] apk-tools.git 37fbafcd928c466c82c892a7868d686d710e5d07
("add: make virtual packages upgradeable (ref #9957)")
[2] https://gitlab.com/postmarketOS/pmaports/blob/master/cross/gcc-armhf/APKBUILD
Fixes: https://gitlab.alpinelinux.org/alpine/apk-tools/issues/10649
Before this change an invalid licence= in an APKBUILD will pass `abuild
sanitycheck`. '/usr/share/spdx/license.lst' contains one licence per line.
`grep -x` will match partial lines whereas `grep -w` will only match whole
lines.
An simple demonstration is with 'GPL-3.0' which is not a valid SPDX licence
identifier. 'GPL-3.0-only' and 'GPL-3.0-or-later' are valid licences.
```
$ grep --help
BusyBox v1.30.1 (2019-04-26 06:26:16 UTC) multi-call binary.
Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...
Search for PATTERN in FILEs (or stdin)
✂
-w Match whole words only
-x Match whole lines only
✂
$ grep -w -F GPL-3.0 /usr/share/spdx/license.lst
GPL-3.0-only
GPL-3.0-or-later
$ grep -x -F GPL-3.0 /usr/share/spdx/license.lst
$
```
Replace text in usage description of fetch that claims to verify sources
with a suggestion to use 'abuild fetch verify', which will actually
verify them.
'abuild fetch' alone will not verify sources, as it only executes the
fetch() function.
Run the loop in a subshell via a pipe so we dont need a subshell for
each iteration.
Use `if ...; then` to make code slightly more readable.
Fix a whitespace before tab while at it.
- do not overwrite variables
srcdir is very important for abuild operation
- quoted various paths
- use a sub-shell to contain directory changing
Resolvesalpinelinux/abuild#58
Some projects might leave files which are not writable for the current
user. The cleanup process then fails and leaves files / directories
behind.
This can easily be fixed by making everything writable before removing
the files.
Add the option 'chmod-clean' which does just that.
Apparently there are many packages that does soemthing like:
subpackages="$pkgname-foo:_foo"
_foo() {
depends="$depends something-else"
}
and thus depend on the previous behavior. We need to revert and plan
this better.
This reverts commit 8fbbffd201.
Other subpackage such as -dev, -doc and even -openrc allow adjusting
depends of the subpackage through such a variable. This is, for
instance, useful to remove a dependency of the origin package from the
-libs subpackage.
While at it document it in APKBUILD(5).
In some cases, a simple rm -rf is not sufficent to clean srcdir.
One such case is the new go module system, that marks everything as
read-only - thus only letting root rm -rf it without a chmod.
There is a command intended to clean them - `go clean -modcache`.
However, for that to work, GOPATH must be defined and existent.
Running chmod for all srcdir cleanups makes no sense, nor does enforcing
root, or putting global overrides just for go.
This patch allows overriding what happens on `cleanup srcdir`, by
overriding cleanup_srcdir, and allows the use of default_cleanup_srcdir.
In our go example, it might be used as such:
cleanup_srcdir() {
go clean -modcache
default_cleanup_srcdir
}
Fix issue when two -dev packages provides same pkg-config wil but with
different versions. For example libressl-dev and openssl-dev both ships
libssl.pc and libcrypto.pc, which resulted in automatic provides of
pc:libssl and pc:libcrypto.
apk would end up picking libressl-dev over openssl-dev for packages that
had automatic pc:libssl depends (for example libssl2-dev), when
openssl-dev was the one that was used during build.
To fix this we add support for a pcprefix so we can set
pcprefix="libressl:" in libressl APKBUILD which makes libressl-dev
provide pc:libressl:libssl. This is similar to what we do with
sonameprefix.
We do not yet automatically detect when the prefixed variant should be
used so for now we will have to explicitly add libressl-dev.
ref #9959
packages should never depend on themselves which does not make sense.
This may happen if main package depends on a subpackage, then the
subpackages will inherit the global depends and the subpackage ends up
depend on itself.
Fix abuild to avoid this.
The 'Compressing data' step takes a significant amount of time when
packaging software with huge binaries, like Kubernetes. This can
certainly be shortened using multithreaded compression, like 'pigz'.
Only run check in fakeroot if options="checkroot" is set. This makes
options="!checkroot" the default.
I expect most checks work as non-root, and if a testsuite requires root,
it will likely fail in fakeroot too. Fakeroot has also shown lower
performance for parallel builds.
Fix case where 1) checksum is a single line and 2) there is a variable
under the checksum that should be kept. For example:
sha512sum="...."
keepthis="..."
Previously the `keepthis` variable would have been removed.
ref https://github.com/alpinelinux/abuild/pull/41
Abuild-fetch uses curl (fallback to wget) to download files. They are
saved with a ".part" extension first, so they can be resumed if
necessary. When the download is through, the ".part" extension gets
removed. However, when the server does not support resume of downloads
(e.g. GitHub's on the fly generated tarballs), then the ".part"
extension got removed anyway. Abuild aborts in that case. But when
running a third time, the distfile exists and it is assumed that this
is the full download.
Changes:
* abuild-fetch:
* Only remove the ".part" extension, when curl/wget exit with 0
* Pass the exit code from curl/wget as exit code of abuild-fetch
* Wherever abuild-fetch would return an exit code on its own, the
codes have been changed to be > 200 (so they don't collide with
curl's as of now 92 exit codes)
* Remove undocumented feature of downloading multiple source URLs at
a time. This doesn't match with the usage description, was not used
in abuild at all and it would have made it impossible to pass the
exit code.
* abuild:
* After downloading, when curl is installed and abuild-fetch has
33 as exit code (curl's HTTP range error), then delete the partfile
and try the download again.
scanelf may pick up tempfiles created by strip or setfattr since it runs
in spearate process and pipes the out to a subshell. This causes a race
and may lead to the while loop attempt to strip seomthing that no longer
exists.
We fix that by test if file exists before try manipulate it. We could
have written he file list to a temp file first, but this way we benefit
from multiple cores working in parallel.
`git describe` by default looks for tags, but `git clone` does not clone
tags by default which causes failures on travis currently.
Also redirect `git describe` errors to /dev/null while being here.
I've forgot to add a patch file to the source variable in an APKBUILD,
altought I did add it to the sums variable.
The error message made it
seem that I've forgot to add the file to the source directory, which led
me to check if my build system was missing the files for some reason.
Only after reading the `abuild.in` file did I understood what happened.
Hopefully this change makes the message clearer and more helpful.
licenses will be checked against the license.lst file provided by
the spdx-licenses-list package when installed except when explicitly
disabled by the !spdx options flag.
abuild, as packaged in Alpine Linux, does not depend on git. But when
you use it without git, it will print out errors like the following:
/usr/bin/abuild: line 2554: git: not found
With this commit, it saves the git_path in the beginning (just like
abuild_path). Later in the code it does not try to run git if that
variable is empty.
Notably `abuild rootbld` is already checking whether `abuild-rootbld`
is installed, and that subpackage of `abuild` does already depend on
`git`. So no additional check was added before using `git ` inside
`rootbld`.
Fixes#32
The force flag used to skip the following functions, without any
documentation in the help (-h) output:
* verify (checksum verification)
* initdcheck (check if the init scripts are openrc scripts)
* check_arch (check if the target architecture is in "arch=")
* check_libc (check if the target libc is masked in the options)
This was counter-intuitive and could even be dangerous (when one relies
on the checksum verification to prevent man-in-the-middle attacks, but
always uses the -f flag).
With this commit, it only skips check_arch and check_libc besides the
package up to date check and the help output mentions this.
provider_priority is a number which determines what priority a package should be
given when solving a dependency graph using a provides entry instead of a direct
package, in the event of conflicts.
This adds an env option REQUIRE_CHECK to require testsuites to
be run. This does not clutter getopts so it can be safely removed
afterwards when we enforce tests globally. This will allow our CI
infrastructure to enforce testsuites where possible.
Guile uses ELF is internal object format, and creates them as
"no machine" and "standlone" OSABI. Scanelf supports printing
OSABI, so use that to filter these out.
See: https://github.com/alpinelinux/aports/pull/1714
This also removes unneccessary 'sed' from the pipeline as it's
simple to read each field outputted by scanelf.
After the first dep is printed, `shift` is called to avoid the special
case where the first dep cannot have a comma prepended. However,
if there are no deps for a package (seen early on in the aports main
repo in acf-jquery), $# is 0. POSIX specifies that `shift` has two
options when the shift operand (1) is greater than $#:
- if non-interactive, it can exit the shell
- if it does not exit the shell, it must return a non-zero exit code
Since we run the shell with -e, the second case folds in to the first.
BusyBox ash does not implement this behaviour, but bash does when called
as /bin/sh or when the `posix` shopt is set.
Currently is hard to discover what abuild version was used on a build log.
This lack of information makes it hard to reproduce a buld failure.
This change simply adds the abuild version at all logs.
Due to fakeroot being very permissive with regards to file permissions,
some package testsuites that explicitly check for failing permission checks
will fail.
For those testsuites, give the APKBUILD the option to run the tests outside
of a fakeroot environment.
This patch allows to set a nice description for the APKINDEX, in case
the aport that is being built is not inside a git repository.
I have tested it, and it behaves exactly like without the patch,
even when git is not installed, or the folder is not inside a git repository:
The `|| true` at the end of the DESCRIPTION= line makes sure, aport
does not get aborted, just like it does not get aborted in that case
as of now, as the `git describe` command gets executed in a subshell.
Currently abuild does not understand .tar.lz packages, which blocks
the inclusion of certain packages in Alpine Linux.
I found this issue when adding 'ed' package to the repository. With
this change, abuild package will now depend on lzip package. I might
send a patch for abuild's APKBUILD.
Just checking the build, host and target triplet is not enough
due to various different combinations used in the early bootstrap.
So use detect specifical bootstrap setting.
The check() function is an APKBUILD overridable function which runs a testsuite. The packages listed in
$checkdepends are packages which will be installed at build time only if check() will be run.
Currently, if aarch64 exists in config.guess, it is not updated.
This breaks spl, which has aarch64 entry, but not ppc64le.
update_config_guess should update config.guess if any of those
does not exists.
This script uses variable expansion / pattern replacement, which is not
defined in POSIX-sh. Also APKBUILDs are not required to be strictly
POSIX-sh compatible. If someone run abuild on system with e.g. dash as
/bin/sh, then it fails (and someone reported exactly that on IRC
today). Therefore abuild should explicitly use /bin/ash in shebang and
not /bin/sh.
this fixes cross deps such as "CHOST=armhf abuild deps"
to work properly.
if makedepends is not defined the following default
will be used (as that's the definition cross-build
aware apkbuilds use):
makedepends="$makedepends_build $makedepends_host"
supporting these features with cross building is non-trivial
and they do are generally not useful features, so remove them.
as result the abuildrepo is also removed to write out paths.
all package generation paths now use $REPODEST/$repo/$CARCH
which allows easily writing packages to correct $subpkgarch
in future commits.
For proper cross-build support, the subpackage arch needs to
be known before invoking the split function. This implements
a way to do that. This also changes to write the actual subpkg
arch to .PKGINFO - apk index --rewrite-arch still overwrites
index to have machine arch for noarch packages.
This is in preparation to support subpackages="pkg:split:arch"
syntax, and just makes the current code ignore anything after
the second colon if it exists. This allows to use the new syntax
in aports git without running experimental abuild on the official
builders.
The previous deletion of file did not work properly at all. Fix
this instead to rename the file, so we keep the previous download.
This allows builders to redownload upstream file if the checksum
has been changed in aports.
"apk del" now returns error if the package name does not exist.
Thus when cross compiling, always install the virtual
.makedepends-$pkgname name for chroot also since undeps() will
try to uninstall it always.
A majority of APKBUILDs currently use exactly the prepare() provided by
newapkbuild verbatim, even for packages where no patches exist, which
has created the expectation that patches listed as sources are applied
by default.
This rev simply adds that default prepare() to abuild (and exposes it as
default_prepare), with no configurability. If needed, default behaviour
can be overridden by providing explicitly prepare().
For git repositories the $reporev variable is not really used since the
$_rev variable is passed to git and $_rev was only set when $reporev was
empty.
The problem is that gzip refuses to run if it detects that a file has
more than 1 link. Our existing solution (removing hardlinks, compressing
the man page and recreating the hardlinks) made certain assumptions
about inode order that are only given on Unix v7 like filesystems
meaning it didn't work properly on 'tree-based' filesystems like BTRFS
or ZFS.
This patch has a different more bulletproof approach: It simply replaces
all hardlinks with symlinks. This is way easier because symlinks (unlike
hardlinks) can point to a file that doesn't exist, therefore we can
update all links before compressing the file in an easy way.
This commit also fixes incorrect behavior in case where an absolute
symlink points to a file installed on the build host but which is
missing from the package.
we might need add $pkgname as a dependency for $pkgname-dev and then we
should not install ourselves in case makedepends="$depends_dev"
This is needed foor bootstraping.
flock(2) on an NFS mount will on the server side convert the lock to a
POSIX lock (fcntl(F_SETLK)). This means that abuild running on NFS
server and client will create different locks and they will both try
download same file at same time.
We fix this by creating a small abuild-fetch application that will
create a POSIX lock which works with NFS.
This is a feature to prefix all so:* provides with a tag. This was done
so that the openjdk8 so depends would only be satisfied by openjdk8
provides and not openjdk7, which might provide same so:lib*.so file.
To use it do: sonameprefix="openjdk8:"
Force dependency scanning when main pkg has noarch but subpackage
overrides it. We need to save the arch setting set in split func for
later if it differs from main pkg's arch in a temp file because the
splitfunc runs in a subprocess.
fixes#4491
Natanael Copa
Joseph Benden
Richard Mortier
tcely
Kaarle Ritvanen
Oliver Smith
Leo
kpcyrd
Pete Dietl
Keith Maxwell
Kevin Daudt
Mike Sullivan
Sören Tempel
Chloe Kudryavtsev
Disassembler
Carlo Landmeter
Tiago Ilieve
Timo Teräs
Jonathan Sieber
Manuel Tiago Pereira
Jonathan Neuschäfer
William Pitcock
Carlo Landmeter
A. Wilcox
tmpfile
Breno Leitao
Andrej
Shiz
G.J.R. Timmer
Tuomas Jaakola
Valery Kartel
Breno Leitao
xentec
Przemyslaw Pawelczyk
Jakub Jirutka
Paul Lorenz
Michael Zuo
Christian Kampka