Yulia reported a bug with module.patch which can be seen with
CONFIG_PROVE_RCU:
BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:49
in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 1053, name: grep
preempt_count: 0, expected: 0
RCU nest depth: 1, expected: 0
2 locks held by grep/1053:
#0: ffff8881079bf1a0 (&p->lock){+.+.}-{3:3}, at: seq_read_iter+0x55/0x460
#1: ffffffff8275d620 (rcu_read_lock){....}-{1:2}, at: cache_seq_start_rcu+0x5/0x140 [sunrpc]
CPU: 3 PID: 1053 Comm: grep Tainted: G OE K 6.2.0 #57
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc37 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x5b/0x77
__might_resched.cold+0xff/0x13a
cpus_read_lock+0x16/0xd0
static_key_disable+0xe/0x20
e_show+0x5b/0xd70 [livepatch_module]
seq_read_iter+0x127/0x460
seq_read+0xa3/0xd0
proc_reg_read+0x52/0xa0
vfs_read+0xc9/0x2f0
? __do_sys_newfstat+0x57/0x60
? lock_is_held_type+0xe8/0x140
ksys_read+0x6c/0xf0
do_syscall_64+0x37/0x90
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fbd314f7af0
Code: b6 fe ff ff 48 8d 3d 2f 78 09 00 48 83 ec 08 e8 96 25 02 00 66 0f 1f 44 00 00 83 3d d9 db 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 7e e3 01 00 48 89 04 24
RSP: 002b:00007ffcd6f1c708 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 0000000000008000 RCX: 00007fbd314f7af0
RDX: 0000000000008000 RSI: 0000563c751ea000 RDI: 0000000000000003
RBP: 0000000000008000 R08: 0000000000009008 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000563c751ea000
R13: 0000000000000003 R14: 0000000000000003 R15: 0000563c751ea000
The problem is that the patched module's seq_operations take an RCU read
lock before calling e_show(), which is patched to call
static_key_disable(), which can sleep.
Fix the issue by moving all the module test code to a different module
(xfs) which doesn't take an RCU lock.
Also enable the pr_debug() test and make the static branch test
unconditional since they're supported by newer kernels and this test is
for 6.2.
Reported-by: Yulia Kopkova <ykopkova@redhat.com>
Debugged-by: Joe Lawrence <joe.lawrence@redhat.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Add patches rebased on top of upstream 6.2.0.
Integration tests for these can be ran as this:
$ make PATCH_DIR="linux-6.2.0" KPATCH_BUILD_OPTS="--non-replace --sourcedir /path/to/src/linux-6.2.0" integration-slow
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Calling __flush_tlb_local() may result in a kernel warning:
STATIC_NOPV void native_flush_tlb_local(void)
{
/*
* Preemption or interrupts must be disabled to protect the access
* to the per CPU variable and to prevent being preempted between
* read_cr3() and write_cr3().
*/
WARN_ON_ONCE(preemptible());
so use another paravirt call like slow_down_io() instead.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Add patches rebased on top of upstream 5.18.0.
Integration tests for these can be ran as this:
$ make PATCH_DIR="linux-5.18.0" KPATCH_BUILD_OPTS="--non-replace --sourcedir /path/to/src/linux-5.18.0" integration-slow
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
The yum-builddep utility doesn't always install all the kernel build
dependencies. Using the --skip-unavailable flag (rhel8+) lets us skip
over packages that it can't find, but continue to install the ones it
can. (Unavailable packages are usually platform-specific or not strictly
necessary for kpatch-build's kernel build invocation.)
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
EPEL ships an epel-release-latest-X.noarch.rpm per rhel-X major release.
Install the package matching the system release.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Clean out any integration tests that no longer exercise their original
use cases.
Suggested-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
As I noticed, commit eaaced191 has added
partial support for openEuler.
This patch enables usage in openEuler like:
kpatch-build xxxx.patch
I test it in openEuler 21.09, for people who
want to use kpatch in openEuler, two more steps
are needed.
1) add repo source
source rpm package of openEuler kernel are put in
two places. One is
https://repo.openeuler.org/openEuler-21.09/source/
Another one is
https://repo.openeuler.org/openEuler-21.09/update/source/
The latter one is not inclued in rpm repo lists by default.
2) compile kernel with CONFIG_LIVEPATCH_PER_TASK_CONSISTENCY enabled
openEuler has its own strategy when trying to apply patches.
We can use the klp_enable_patch function only when
CONFIG_LIVEPATCH_PER_TASK_CONSISTENCY is enabled.
Signed-off-by: anatasluo <luolongjuna@gmail.com>
* gcc-constprop.patch
In v4.20, 33e26418193f ("y2038: make do_gettimeofday() and get_seconds()
inline"), do_gettimeofdat() no longer exists as a stand alone function
in kernel/time/timekeeping.c.
* gcc-static-local-var-4.patch
Unlike on rhel-8.4, _always_inline put_aio_ring_file() is causing too
many inlines and results in modified, but not ftrace-able,
__do_sys_io_submit() and __do_sys_io_setup(). Remove the annotation
from this function.
* module.patch
In v4.20, 9ceddd9da134 ("knfsd: Allow lockless lookups of the exports"),
the nfs_exports_op seq_operations converted to RCU protected lookups.
Calling yield() from a kpatched e_show() results in a kernel warning,
"Voluntary context switch within RCU read-side critical section!"
Substitute with single_task_running(), which does not context switch or
have any other side effects.
In v5.10, ec6347bb4339 ("x86, powerpc: Rename memcpy_mcsafe() to
copy_mc_to_{user, kernel}()") did away with the mcsafe_key. Use
another exported static key like context_tracking_enabled.
In v5.13, a0e2bf7cb700 ("x86/paravirt: Switch time pvops functions to
use static_call()"), paravirt_sched_clock() was converted from a
paravirt call to a non-exported static call. Update the x86 code to
instead call __flush_tlb_local() (which is still a paravirt call).
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
module-call-external.patch and new-function.patch fail to build on
ppc64le with "Found an unsupported sibling call". Add
fno-optimize-sibling-calls attributes to functions in question.
Signed-off-by: Artem Savkov <asavkov@redhat.com>
Depending on environment, "Permission denied" error may occur
when /sys/kernel/debug/dynamic_debug/control setting.
Signed-off-by: Keiya Nobuta <nobuta.keiya@fujitsu.com>
gcc-static-local-var-4.patch is disabled on this distribution, disable
the test as well as it will always fail during 'slow' integration test
runs.
Signed-off-by: Artem Savkov <asavkov@redhat.com>
Update the test/integration/Makefile to pass a KPATCH_BUILD_OPTS
variable to kpatch-test. This allows the user better control over the
kpatch build process, for example, building non-atomic replace .ko files
on kernels that do support atomic-replace:
% make integration KPATCH_BUILD_OPTS="--non-replace"
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Rebased to kernel-4.18.0-304.el8.
Note: since RHEL-8.4 dropped klp.arch support, we can now re-enable
those tests that reference static keys defined in vmlinux.
Also, adjust for adjust for ppc64le inlining:
Building gcc-static-local-var-4.patch on ppc64le results in test
failure, as the kpatch .ko now contains a 'free_ioctx' symbol (the test
expects to NOT see one).
From the build log:
aio.o: changed function: free_ioctx
aio.o: new function: put_aio_ring_file << now un-inlined?
aio.o: changed function: aio_free_ring
aio.o: changed function: ioctx_alloc
aio.o: changed function: aio_prep_rw
aio.o: changed function: aio_read_events
aio.o: new function: kpatch_aio_foo << expected new function
and a source code change to free_ioctx():
% diff -upr \
<(objdump -D -j .text.free_ioctx ~/.kpatch/tmp/orig/fs/aio.o) \
<(objdump -D -j .text.free_ioctx ~/.kpatch/tmp/patched/fs/aio.o)
--- /dev/fd/63 2020-10-26 14:28:18.086236019 -0400
+++ /dev/fd/62 2020-10-26 14:28:18.086236019 -0400
@@ -1,5 +1,5 @@
-/root/.kpatch/tmp/orig/fs/aio.o: file format elf64-powerpcle
+/root/.kpatch/tmp/patched/fs/aio.o: file format elf64-powerpcle
Disassembly of section .text.free_ioctx:
@@ -53,7 +53,7 @@ Disassembly of section .text.free_ioctx:
b0: 00 00 82 3c addis r4,r2,0
b4: 00 00 84 e8 ld r4,0(r4)
b8: 78 fb e6 7f mr r6,r31
- bc: e0 00 63 38 addi r3,r3,224
+ bc: 38 00 63 38 addi r3,r3,56
c0: 01 00 00 48 bl c0 <free_ioctx+0xb8>
c4: 00 00 00 60 nop
c8: 70 ff ff 4b b 38 <free_ioctx+0x30>
Marking put_aio_ring_file() as __always_inline keeps the r3 / 224
offset value, leaving free_ioctx() unchanged. Since it's no longer
included in the resulting .ko, gcc-static-local-var-4.test will pass
once again.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Add patches rebased on top of upstream 5.10.11 tarball. Integration
tests for these can be ran as this:
make PATCH_DIR="linux-5.10.11" KPATCHBUILD_OPTS="-s /path/to/src/linux-5.10.11" integration-slow
Signed-off-by: Artem Savkov <asavkov@redhat.com>
Rebased against kernel-3.10.0-1160.el7.
data-read-mostly.patch.disabled remains disabled as we hit several build
errors like:
"Found a jump label at __netif_receive_skb_core()+0x50, using key
netstamp_needed. Jump labels aren't supported with this kernel. Use
static_key_enabled() instead."
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Add a RHEL reproducer that causes a kpatch-build to fail when the final
module references a symbol with a different CRC than the original
kernel.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
The kpatch-test script clears the kernel log buffer to distinguish
between old and new dmesg entries. Wiping out the old buffer may
surprise some users, but isn't too hard to avoid:
- save the last dmesg line
- run the tests
- filter out old dmesg lines until after finding the saved entry
- if no saved entry is found, the buffer most likely overflowed
- inform the user to increase the log size
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Defer clearing the kernel buffer until we're ready to start the tests,
otherwise we increase the likelihood of catching stray errors while
we're building the modules and preparing the tests.
Fixes#1133 ("kpatch-test should ignore older kernel log messages")
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
The commands used to build the livepatches and to load or unload them
are currently hard-coded in kpatch-test.
This patch adds 2 options to kpatch-test to make it easier to use custom
kpatch and kpatch-build commands:
* --system-kpatch-tools - if set, 'sudo kpatch' will be used to
load/unload the patches; 'kpatch-build' - to build them.
To use custom tools here, the user can adjust $PATH.
If the option is not set, kpatch-test will assume it is in kpatch source
tree, same as before this commit, and will use the tools from there.
* --kpatch-build-opts="..." - additional options to pass to
kpatch-build.
Example:
./kpatch-test \
--system-kpatch-tools \
--kpatch-build-opts="-s ./linux-src -c ./config -v ./vmlinux" \
-d my_kpatch_tests/test/integration/v01
In this case, kpatch and kpatch-build installed in the system will be used,
and kpatch-build will look for the kernel source tree, configuration
file and vmlinux binary in the current directory.
Signed-off-by: Evgenii Shatokhin <eshatokhin@virtuozzo.com>
RHELs older than 7.7 are missing 5279631271b3 "module: fix ddebug_remove_module()"
which leads to a crash if dynamic debug is used with livepatching.
Disable recent dynamic debug addition on those distros.
Signed-off-by: Artem Savkov <asavkov@redhat.com>
Fixing the remaining warnings suggesting to quote the output subshell
invocation would hinder readability. Also the results of dirname and
"type -p" used in the subshell should return spaceless strings in kpatch
usecases.
Ignore the warnings for now.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
It wouldn't be bash without pondering over what to quote or not to
quote.
Shellcheck reports a bunch of quoting issues in our scripts. Fix what
can be fixed.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Shellcheck emmits the following warning:
SC2035: Use ./*glob* or -- *glob* so names with dashes won't become options.
Which seems like a fair recommendation.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Shellcheck reports the following error:
SC2145: Argument mixes string and array.
Lets pretend that types are a thing in bash and use the list of
arguments as a single string instead of the array of arguments.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Shellcheck complains with the following warning:
SC2230: which is non-standard.
It is probably fair to assume that which is available for now, so just
ignore the warning.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Shellcheck complains with the following warning:
SC1091: Not following: /etc/os-release was not specified as input (see shellcheck -x)
Which is already silenced in lib.sh.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
On RHEL-8, version agnostic python-devel package does not exist.
On previous RHEL releases, python-devel is for python2.
On RHEL-8, the platform-python (which provides the needed utilities)
is python3.
Assuming this will be the same for future RHEL releases, specify the
python major as part of the kpatch dependencies.
Signed-off-by: Julien Thierry <jthierry@redhat.com>
Enable dynamic debug prints in klp_try_switch_task() function before
going through with load tests and switch back to a previous state
afterwards.
Signed-off-by: Artem Savkov <asavkov@redhat.com>
For consistency, disable the shadow-newpid-LOADED integration test
script since its corresponding patch already is already disabled.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
