Upstream 4.15 kernels provide support for pre and post (un)patch
callbacks, inspired by the kpatch load hooks. Add support for them
in the livepatch-patch-hook.
At the same time, convert the kpatch hooks to use the same API.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Restructure kpatch's sysfs interface and mirror the sysfs tree after
livepatch's sysfs layout. With the current sysfs layout, we cannot
distinguish which object a function belongs to, and we cannot tell which
modules/objects are patched. Therefore, restructure the kpatch sysfs tree
such that module/object information is available. With the new layout, each
patched object has its own directory, with each function being a
subdirectory of its object.
Implement this by embedding a kobject struct within the kpatch_module,
kpatch_func, and kpatch_object structs and supplying their ktypes and
kobject release methods.
Before:
/sys/kernel/kpatch
└── patches
└── <patch_module>
├── checksum
├── enabled
└── functions
├── <function> # from <object1>
│ ├── new_addr
│ └── old_addr
├── <function> # from <object2>
│ ├── new_addr
│ └── old_addr
└─── <function> # from <object3>
├── new_addr
└── old_addr
After:
/sys/kernel/kpatch
└── <patch_module>
├── <object1>
│ └── <function,sympos>
│ ├── new_addr
│ └── old_addr
├── <object2>
│ └── <function,sympos>
│ ├── new_addr
│ └── old_addr
├── checksum
├── enabled
└── <object3>
└── <function,sympos>
├── new_addr
└── old_addr
Backport the symbol lookup and checking code from upstream livepatch
code that relies on a symbol position enumeration rather than a fixed
memory address.
Fixes#617.
When patching a kernel module, if we can't find a needed dynrela symbol,
we currently assume it's exported. However, it's also possible that
it's provided by another .o in the patch module. Add support for that.
Fixes#445.
This adds support for shadow variables, which allow you to add new
"shadow" fields to existing data structures.
To allow patches to call the shadow functions in the core module, I had
to add a funky hack to use --warn-unresolved-symbols when linking, which
allows the patched vmlinux to link with the missing symbols. I also
added greps to the log file to ensure that only unresolved symbols to
kpatch_shadow_* are allowed. We can remove this hack once the core
module gets moved into the kernel tree.
Fixes#314.
To reduce redundancy, remove/change the old_offset fields in the
kpatch_func and kpatch_patch_func structs to just old_addr. Since
old_offset is being used as a placeholder for old_addr, might as well
consolidate it to just one variable.
Some functions in the kernel are always on the stack of some thread
in the system. Attempts to patch these function will currently always
fail the activeness safety check.
However, through human inspection, it can be determined that, for a
particular function, consistency is maintained even if the old and new
versions of the function run concurrently.
This commit introduces a KPATCH_FORCE_UNSAFE() macro to define patched
functions that such be exempted from the activeness safety check.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
This commit enables the ability to create user-defined hooks as part of
the normal code patch that can do preparatory work for the application
of the patch. This work could include, but is not limited to, changing
data structure semantics.
The user may define a new function as part of the patch and mark it as a
load-time or unload-time hook with the kpatch_load_hook() and
kpatch_unload_hook() macros. These macros are in an include file that
gets copied into the source tree at include/linux/kpatch-hooks.h at
patch build time. The signature for both hooks is "int kpatch_unload_hook(void)".
For now, the return code is ignored. The hooks may not fail. They also
run in stop_machine() context and may not sleep. These hooks, more or
less, must follow all the rules of interrupt context code.
When patching module A, if one of the new function's relas reference a
symbol in module B, we currently just leave it as a normal rela. But if
module B hasn't been loaded yet, the patch module will fail to load due
to the rela's reference to an undefined symbol.
The fix is to convert these relas to dynrelas, which can be resolved
later in the module notifier when A is loaded.
Also added support for the R_X86_64_NONE relocation type, needed for
dynrelas which reference __fentry__.
The recent module patching code has exposed some problems with our data
structures. We currently patch the funcs and dynrelas individually,
which is kind of scary now that different objects can be patched at
different times. Instead it's cleaner and safer to group them by
patched object.
This patch implements per-object patching and relocations by refactoring
the interfaces:
- Completely separate the create-diff-object <-> patch module interface
from the patch module <-> core module interface. create-diff-object
will include "kpatch-patch.h" but not "kpatch.h". Thus,
create-diff-object has no knowledge about the core module's
interfaces, and the core module has no knowledge about the patch
module's special sections.
- Newly added kpatch-patch.h defines the format of the patch module
special sections. It's used by create-diff-object to create the
special sections and used by the patch module to read them.
- kpatch.h still defines the core module interfaces. Each kpatch_module
has a list of kpatch_objects for each module object to be patched.
Each kpatch_object has a list of kpatch_funcs and a list of
kpatch_dynrelas. The patch module creates these lists when populating
kpatch_module.
This way of structuring the data allows us to patch funcs and dynrelas
on a per patched object basis, which will allow us to catch more error
scenarios and make the code easier to manage going forward. It also
allows the use of much more common code between kpatch_register() and
kpatch_module_notify().
This allows a patch module to contain patched functions for modules
which haven't been loaded yet. If/when the module is loaded later, it
will be patched from the module notifier function.
Create a list of registered patch modules, which will be used in the
module notifier to patch future loaded modules. It will also be used to
fix the kpatch replace race condition where it calls module_put too
early.
The kpatch_internal struct is a good idea, in that it documents which
parts of kpatch_module shouldn't be used by the patch module. But it
creates extra code and will require more extra code if we want to keep a
list of kpmods, which is needed to create a module notifier for module
patching of future loaded modules.
Embedding the private data directly in the public struct allows the code
to be simpler: no extra kmallocs/kfrees, no need to store pointers
between the public and private structs. I think the simpler code is
worth the tradeoff (exposing implementation detail). Kernel code
usually doesn't bother with hiding a internal struct data from other
kernel code anyway. For example, see ftrace_ops or struct kprobe.
The private fields are documented with a "private" comment.
This commit adds module patching support to create-diff-object by:
1) generalizing the vmlinux CLI parameter
2) adding the kernel object name to each patch and dynrela
3) adding slightly different logic for vmlinux/module in the dynrela
creation
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Make old addresses relative to the start address of the relocatable
kernel or module.
This commit has no functional effect; it just prepares the code for
future acceptance of the module patching support.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
This commit introduces functionality to verify the location of symbols
used in both the patch and dynrelas sections. It adds significant
protection from mismatches between the base and running kernels.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
This adds dynamic linking support for the patch modules. It is the
first step toward supporting patching module code and relocatable
kernels.
Rela entries that reference non-included local and non-exported global
symbols are converted to "dynrelas". These dynrelas are relocations
that are done by the core module, not the kernel module linker. This
allows the core module to apply offsets to the base addresses found
in the base vmlinux or module.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Conflicts:
kpatch-build/kpatch-build
We merged PR #186 a little too hastily. It seg faults with the new
parainstructions-section.patch in the integration test suite. Reverting
it for now until we get it figured out.
This reverts commit e1177e3a03.
This reverts commit 880e271841.
This reverts commit 2de5f6cbfb.
This reverts commit 38b7ac74ad.
This reverts commit 108cd9f95e.
This adds dynamic linking support for the patch modules. It is the
first step toward supporting patching module code and relocatable
kernels.
Rela entries that reference non-included local and non-exported global
symbols are converted to "dynrelas". These dynrelas are relocations
that are done by the core module, not the kernel module linker. This
allows the core module to apply offsets to the base addresses found
in the base vmlinux or module.
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Make kpatch_funs truly internal by:
Defining it in core.c
Adding a struct kpatch_internal, declared in kpatch.h and defined in
core.c, that contains per patch module internal data.
Adding an "internal" field to struct kpatch_modules.
Allocating internal and funcs data in core.c, not in the patch module,
since the patch module has no knowledge of kpatch_func anymore.
Adding a "patch" field to kpatch_func that points directly to the
kpatch_patch provided by the module (rather than a field-by-field copy)
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Allow the user to atomically replace all existing modules with a new
"kpatch replace" command. This provides a safe way to do atomic
upgrades for cumulative patch module updates.
- checkpatch doesn't like the FSF address since it's subject to change
- checkpatch doesn't like strings split by line
- whitespace fix
- sparse suggested to change some variables and functions to static
This is an attempt to both simplify and improve the correctness of the
NMI synchronization code.
There's a race in kpatch_ftrace_handler() between the kpatch_get_func()
and kpatch_finish_status() calls which could result in func being NULL.
The retry was supposed to fix this. However, this race would still be a
problem in the repatching case (if the function had already been
previously patched), in which case func would not be NULL, but could
instead point to the previously patched version of the function. In
this case it wouldn't retry and it would be possible for the previous
version of the function to run.
The fix is to use a memory barrier between accesses of the func hash and
the status variable, and then just call kpatch_get_func() *after*
accessing the status variable. For OP_PATCH, if status is SUCCESS, then
func is guaranteed to point to the new function. If status is FAILURE,
func might point to the new function, in which case we can use
get_prev_func to get the previous version of the function.
I also made some pretty big changes to try to simplify the design so
that there are less moving parts and so that it's hopefully easier to
understand. I moved the OP field into the kpatch_func struct. This
allows us to merge the two global state variables (status + op) into a
single global state variable (state), which helps make the code quite a
bit simpler. I turned it into a proper state machine and documented the
meaning of each state in the comments.
Moving the OP field to the kpatch_func struct also paves the way for an
upcoming pull request which will allow patch modules to be atomically
replaced ("kpatch load --replace <module>").
Currently the patch module calls kpatch_unregister in the patch module
exit path. If the activeness safety check fails in kpatch_unregister,
it's too late for the patch module to stop exiting, so all it can do is
panic.
Prevent this scenario by requiring the user to disable the patch module
via sysfs before allowing the module to be unloaded. The sysfs write
will fail if the activeness safety check fails. An rmmod will fail if
the patch is still enabled.
Also add support for this new unloading model in "kpatch unload".
Put funcs, num_funcs, and mod in their own struct called kpatch_module.
This allows us to keep patch module specific variables in one place (and
we'll have more of these variables soon).
Support live patching on NMI handlers. This adds checks for
possible inconsistency of live patching on NMI handlers.
The inconsistency problem means that any concurrent execution
of old function and new function, which can lead unexpected results.
Current kpatch checks possible inconsistency problem with
stop_machine, which can cover only threads and normal interrupts.
However, beacuse NMI can not stop with it, stop_machine is not
enough for live patching on NMI handlers or sub-functions which are
invoked in the NMI context.
To check for possible inconsistency of live patching on those
functions, add an atomic flag to count patching target functions
invoked in NMI context while updating kpatch hash table. If the
flag is set by the target functions in NMI, we can not ensure
there is no concurrent execution on it.
This fixes the issue #65.
Changes from v5:
- Fix to add a NULL check in kpatch_get_committed_func().
Changes from v4:
- Change kpatch_operation to atomic_t.
- Use smp_rmb/wmb barriers between kpatch_operation and kpatch_status.
- Check in_nmi() first and if true, access kpatch_operation.
Changes from v3:
- Fix kpatch_apply/remove_patch to return 0 if succeeded.
Changes from v2:
- Clean up kpatch_get_committed_func as same style of kpatch_get_func.
- Rename opr to op in kpatch_ftrace_handler.
- Consolidate in_nmi() and kpatch_operation check into one condition.
- Fix UNPATCH/PATCH mistype in kpatch_register.
Changes from v1:
- Rename inconsistent_flag to kpatch_status.
- Introduce new enums and helper functions for kpatch_status.
- Use hash_del_rcu instead of hlist_del_rcu.
- Rename get_committed_func to kpatch_get_committed_func.
- Use ACCESS_ONCE for kpatch_operation to prevent compiler optimization.
- Fix to remove (!func || func->updating) condition from NMI check.
- Add more precise comments.
- Fix setting order of kpatch_status and kpatch_operation.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
My apologies for the size of this commit. I combined these two features
(updating API and using a hash table) into a single commit because their
implementations are tightly coupled and I didn't want to have to add
support for the old kpatch_funcs array with the new API just for the
sake of splitting up the commit :-)
- Update the core module API to get a more clear separation between core
module and patch module. This is cleaner and will help our case for
getting the core module merged upstream into the kernel.
- Convert the old kpatch_funcs array into a hash table. This is so much
nicer performance-wise and everything-else-wise than that ugly old
array.
- Do the incremental patching in stop machine. This ensures that the
funcs hash is up to date and we don't miss anything.
- Disable preemption in the ftrace handler when accessing the func hash.
That way we don't get conflicts with the stop_machine handler updating
the hash.
Joe Lawrence
Jessica Yu
Josh Poimboeuf
Jessica Yu
Seth Jennings
Masami Hiramatsu