check_whence.py requires the current tree to be a git repo so run it only
if that is indeed the case.
Signed-off-by: Juerg Haefliger <juerg.haefliger@canonical.com>
Currently we use f(ile) and d(irectory), over the more common ones t(arget) and
l(ink). Rename things appropriately.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
The check_whence.py script ensures that links defined in WHENCE are not in-tree.
Since we're calling the script, we no longer need the convoluted path and
associated --prune tag.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
With previous commit we call check_whence.py, which ensures that all files
listed are available. Drop the now dead code.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Currently ./check_whence.py is used when submitting new firmware, while
copy-firmware.sh when the firmware is to be consumed.
Since the latter does (very little) validation, having a malformed WHENCE file
can lead to all sorted of problems. From the obvious, where it errors out, to
more serious one where it overwrites or executes something it should not have.
Just call check_whence.py and error out. It takes 0.2s on my 5 year old
mid-range laptop, so the overhead is negligible.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
If the user provides an existing non-empty folder (their /usr/lib/firmware/ or otherwise)
there is a high chance we'll silently overwrite existing files. That may or may not be what
they wanted, so throw a warning so highlight that.
v2:
- use printf instead of echo
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Currently we don't reset/override the destdir variable, so we end up inheriting
whatever the caller's environment has for it. While it may work, it's not
particularly consistent (be that within the script or other tools) nor is it
obvious.
While in here, ensure we handle the variable with test -z/-n instead of varying
other constructs.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
This reverts commit 2bad80e7ed.
The commit effectively added accidental command injection, while it was aiming
to control the compression flags.
In practise you'd want to use ZSTD_CLEVEL and ZSTD_NBTHREADS for zstd. As
documented in zstd(1) it allows for up-to level 19, which is fine since the
kernel does not support higher levels.
Arch, Alpine and likely other distributions have been using this approach
since day one.
The other compressors like xz have equivalent.
Cc: Juerg Haefliger <juerg.haefliger@canonical.com>
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Flesh out the de-duplication logic in separate script. The copy-firmware.sh is
already complex enough and de-duplication doesn't really fit in there.
In the process we migrate away from the open-coded `ln --relative`. We also
avoid touching symlinks, which are not created by rdfind. Otherwise we end up
"fixing" the folder to folder symlinks (created earlier in the process) and
things explode.
As result we also get a few bonuses:
- the COPYOPTS shell injection is gone - the variable was never used
- people can dedup as separate step if/when they choose to do so
Aside: based on the noise in git log and around distros ... I'm wondering if
having the de-duplication as opt-in, would have been better. Is it too late to
change or the ship has sailed?
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
If copy-firwmware is told to compress files, it blindly assumes that
the target is indeed compressed (unless it's a directory) and creates
any symlinks accordingly.
This leads to broken symlinks for links that point at RawFiles:
adspr.jsn.zst -> ../qcm6490/adspr.jsn.zst
adsps.jsn.zst -> ../qcm6490/adsps.jsn.zst
That should be:
adspr.jsn -> ../qcm6490/adspr.jsn
adsps.jsn -> ../qcm6490/adsps.jsn
Fix that by checking if the target (directory or file) exists. And while
at it, add a check for broken symlinks at the end.
Fixes: 541f96c0fa ("qcom: qcm6490: add ADSP and CDSP firmware")
Signed-off-by: Juerg Haefliger <juerg.haefliger@canonical.com>
A number of distributions don't include rdfind by default. This is an
optimization, so make it optional via a --ignore-duplicates option and a
new make target.
Distributions that include rdfind should add it as a dependency for
their linux-firmware package. Those that don't should use the new
target or option.
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
The "which" utility is not guaranteed to be installed either, and if it
is, its behavior is not portable either. This means that when rdfind /
pre-commit are installed, the `which` check will report a fatal error
because the which tool did not exist and the shell returned a nonzero
status when attempting to fork+exec. If it did exist, it might not be an
implementation of `which` that returns nonzero when commands do not
exist.
Conversely, the "command -v" shell builtin is required to exist in all
POSIX 2008 compliant shells, and is thus guaranteed to work everywhere.
For some in-depth discussions on the topic, see:
- https://mywiki.wooledge.org/BashFAQ/081
- https://unix.stackexchange.com/questions/85249/why-not-use-which-what-to-use-then/85250#85250
Examples of open-source shells likely to be installed as /bin/sh on
Linux, which implement the 15-year-old standard: ash, bash, busybox,
dash, ksh, mksh and zsh.
A side benefit of using the POSIX portable option is that it requires
neither an external disk executable, nor (because unlike "which", the
exit code is reliable) a subshell fork. This therefore represents a mild
speedup.
Signed-off-by: Eli Schwartz <eschwartz93@gmail.com>
rdfind was trying to duplicate symlinks created by WHENCE causing
problems. Move the call before WHENCE creates symlinks.
Reported-by: Timur Tabi <ttabi@nvidia.com>
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Currently, 'Raw: <filename>' needs to be added to WHENCE for files that
must not be compressed. This means that such files need to be listed
twice ('File: <filename>' and 'Raw: <filename>') which is not pretty and
error prone. Also, the current implementation is broken for filenames
that contain quotes (") and it also requires grep'ing of WHENCE for every
processed line which slows things down.
Fix all of the above by introducing a new keyword 'RawFile' so that such
files are listed only once, quotes are handled correctly and grep is no
longer needed.
Signed-off-by: Juerg Haefliger <juerg.haefliger@canonical.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
Any options following the compressor option --zstd or --xz are passed on
to the compressor as additional options so that - for example - the
compression level can be specified:
$ ./copy-firmware.sh --zst -19 <destdir>
Signed-off-by: Juerg Haefliger <juerg.haefliger@canonical.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
When `copy-firmware` is called with `--xz` or `--zstd` it will create
broken symlinks for directories:
```
$ ./copy-firmware -v --zstd $dir
[...]
creating link qcom/LENOVO/21BX.zst -> ../sc8280xp/LENOVO/21BX.zst
```
The original target `../sc8280xp/LENOVO/21BX` is a directory. Adding the
compression extension to the directory name breaks the link. The
directory `qcom/sc8280xp/LENOVO/21BX` exists but
`qcom/sc8280xp/LENOVO/21BX.zst` does not exist.
The relative symlink needs to be resolved. If it points to a directory,
create the symlink without the compression extension.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
dash is the default /bin/sh on Debian based system.
```
dash ./copy-firmware.sh --zstd example
./copy-firmware.sh: 37: test: unexpected operator
```
`=` should be used with the test command for POSIX conformance.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
The kernel has supported compressed firmware for quite some time. So
let's add a couple of targets to produce that. In practical terms this
means it we'll use ~5x times less space on disk.
Reportedly the amd ucode, needs to be uncompressed _within_ the
initrd in order to work. Using compressed ucode in late load just works.
Ideally this will be addressed by the initrd generators, but considering
the files are tiny in size let's skip the compression.
v2
- commit message, skip compression for files annotated as Raw
v3
- rebase
[Drop extra verbose statement in zstd case, Josh Boyer
<jwboyer@kernel.org>]
Cc: David Woodhouse <dwmw2@infradead.org>
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
As mentioned by shellcheck backticks are considered legacy. Plus we
should ensure the output is quoted, otherwise we'll get word splitting.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
Add space between the arguments and pattern and combine patterns where
possible.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
Properly quote, otherwise we'll get word splitting. In other words:
Files might end up installed to /some/foo/, where /some path/foo bar/ is
expected.
v2:
- rebase/split && also quote the $() output
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
mkdir -p creates paths that are bound to user's settings and therefore
can lead to different file mode bits of the base paths accross different
machines.
Use install instead, as this tool is not prone to such behavior.
Signed-off-by: Konrad Weihmann <kweihmann@outlook.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
File: entries in the WHENCE file overwrite existing files in the target
directory, however, Link: entries are skipped if the file exists in the
target directory. This can cause issues if the Link: entry is updated, but
the target directory contains an old symlink. Do not skip writing Link:
entries if the file exists, always create the symlink. This matches the
behavior of File: entries and ensures symlinks will contain values from
the WHENCE file.
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
Rather than require symlinks to be created in the filesystem and have
duplicate Link: entries in the WHENCE file, make copy-firmware.sh create
the symlinks on the fly.
Signed-off-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
The current make-install procedure leaves lots of garbage files that
aren't really firmware files in /lib/firmware.
Instead of copy-all-and-prune approach, copy only the listed files and
links in WHENCE by make-install for assuring only the proper firmware
files.
Signed-off-by: Takashi Iwai <tiwai@suse.de>
Signed-off-by: Josh Boyer <jwboyer@kernel.org>
Juerg Haefliger
Emil Velikov
Josh Boyer
Eli Schwartz
Mario Limonciello
Benjamin Drung
Emil Velikov
Konrad Weihmann
Tony Nguyen
Thierry Reding
Takashi Iwai