2014-02-17 15:54:25 +00:00
|
|
|
kpatch: dynamic kernel patching
|
|
|
|
===============================
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-19 21:49:15 +00:00
|
|
|
kpatch is a Linux dynamic kernel patching tool which allows you to patch a
|
|
|
|
running kernel without rebooting or restarting any processes. It enables
|
2014-02-20 03:32:07 +00:00
|
|
|
sysadmins to apply critical security patches to the kernel immediately, without
|
2014-02-19 21:49:15 +00:00
|
|
|
having to wait for long-running tasks to complete, users to log off, or
|
2014-02-20 03:32:07 +00:00
|
|
|
for scheduled reboot windows. It gives more control over uptime without
|
2014-02-19 21:49:15 +00:00
|
|
|
sacrificing security or stability.
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-19 03:32:24 +00:00
|
|
|
kpatch is currently in active development. For now, it should _not_ be used
|
2014-02-19 21:49:15 +00:00
|
|
|
in production environments.
|
2014-02-18 16:15:13 +00:00
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
**WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss
|
|
|
|
may occur!**
|
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
Installation
|
|
|
|
------------
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
*NOTE: These installation instructions are currently Fedora-specific. Support
|
|
|
|
for other distributions is planned soon.*
|
|
|
|
|
|
|
|
Install the dependencies for compiling kpatch:
|
|
|
|
|
|
|
|
sudo yum install gcc kernel-devel elfutils elfutils-devel
|
|
|
|
|
|
|
|
*NOTE: Ensure you have elfutils-0.158 or newer.*
|
|
|
|
|
2014-03-10 14:24:40 +00:00
|
|
|
Install the dependencies for the "kpatch-build" command:
|
2014-02-18 16:15:13 +00:00
|
|
|
|
|
|
|
sudo yum install rpmdevtools pesign
|
|
|
|
sudo yum-builddep kernel
|
|
|
|
|
2014-02-18 21:56:47 +00:00
|
|
|
# optional, but highly recommended
|
|
|
|
sudo yum install ccache
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
Compile kpatch:
|
2014-02-13 16:04:35 +00:00
|
|
|
|
|
|
|
make
|
2014-02-18 16:15:13 +00:00
|
|
|
|
|
|
|
Install kpatch to /usr/local:
|
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
sudo make install
|
|
|
|
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
Quick start
|
2014-02-17 15:54:25 +00:00
|
|
|
-----------
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
*NOTE: While kpatch is designed to work with any recent Linux
|
2014-03-10 14:24:40 +00:00
|
|
|
kernel on any distribution, the "kpatch-build" command currently
|
2014-02-17 15:54:25 +00:00
|
|
|
only works on Fedora.*
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
Load the kpatch core module:
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
sudo insmod /usr/local/lib/modules/$(uname -r)/kpatch/kpatch.ko
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
Make a source patch against the kernel tree:
|
|
|
|
|
|
|
|
# from a kernel git tree:
|
|
|
|
git diff > /path/to/foo.patch
|
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
Build the patch module:
|
2014-02-18 04:37:07 +00:00
|
|
|
|
2014-03-10 14:24:40 +00:00
|
|
|
kpatch-build /path/to/foo.patch
|
2014-02-18 04:37:07 +00:00
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
This outputs a patch module named `kpatch-foo.ko` in the current
|
2014-02-18 04:37:07 +00:00
|
|
|
directory. Now apply it to the running kernel:
|
|
|
|
|
|
|
|
sudo insmod kpatch-foo.ko
|
|
|
|
|
|
|
|
Done! The kernel is now patched.
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
How it works
|
2014-02-17 15:54:25 +00:00
|
|
|
------------
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-02-19 03:32:24 +00:00
|
|
|
kpatch works at a function granularity: old functions are replaced with new
|
|
|
|
ones. It has four main components:
|
|
|
|
|
|
|
|
- **kpatch-build**: a collection of tools which convert a source diff patch to
|
2014-04-01 21:40:28 +00:00
|
|
|
a patch module. They work by compiling the kernel both with and without
|
|
|
|
the source patch, comparing the binaries, and generating a patch module
|
2014-02-19 03:32:24 +00:00
|
|
|
which includes new binary versions of the functions to be replaced.
|
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
- **patch module**: a kernel module (.ko file) which includes the
|
2014-02-19 03:32:24 +00:00
|
|
|
replacement functions and metadata about the original functions.
|
|
|
|
|
|
|
|
- **kpatch core module**: a kernel module (.ko file) which provides an
|
2014-04-01 21:40:28 +00:00
|
|
|
interface for the patch modules to register new functions for
|
2014-02-19 03:32:24 +00:00
|
|
|
replacement. It uses the kernel ftrace subsystem to hook into the original
|
|
|
|
function's mcount call instruction, so that a call to the original function
|
|
|
|
is redirected to the replacement function.
|
|
|
|
|
|
|
|
- **kpatch utility:** a command-line tool which allows a user to manage a
|
2014-04-01 21:40:28 +00:00
|
|
|
collection of patch modules. One or more patch modules may be
|
2014-02-19 15:50:56 +00:00
|
|
|
configured to load at boot time, so that a system can remain patched
|
2014-02-19 03:32:24 +00:00
|
|
|
even after a reboot into the same version of the kernel.
|
|
|
|
|
|
|
|
|
2014-03-10 14:24:40 +00:00
|
|
|
### kpatch-build
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
The "kpatch-build" command converts a source-level diff patch file to a kernel
|
|
|
|
patch module. Most of its work is performed by the kpatch-build script
|
2014-02-18 16:15:13 +00:00
|
|
|
which uses a collection of utilities: `create-diff-object`,
|
|
|
|
`add-patch-section`, and `link-vmlinux-syms`.
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
The primary steps in kpatch-build are:
|
|
|
|
- Build the unstripped vmlinux for the kernel
|
|
|
|
- Patch the source tree
|
|
|
|
- Rebuild vmlinux and monitor which objects are being rebuilt.
|
2014-02-11 18:00:17 +00:00
|
|
|
These are the "changed objects".
|
2014-02-18 16:15:13 +00:00
|
|
|
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
|
2014-02-11 18:00:17 +00:00
|
|
|
resulting in the changed patched objects
|
|
|
|
- Unpatch the source tree
|
2014-02-18 16:15:13 +00:00
|
|
|
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
|
2014-02-13 13:49:02 +00:00
|
|
|
resulting in the changed original objects
|
2014-02-18 16:15:13 +00:00
|
|
|
- Use `create-diff-object` to analyze each original/patched object pair
|
2014-02-11 18:00:17 +00:00
|
|
|
for patchability and generate an output object containing modified
|
|
|
|
sections
|
2014-02-18 16:15:13 +00:00
|
|
|
- Link all the output objects into a cumulative object
|
|
|
|
- Use `add-patches-section` to add the .patches section that the
|
2014-04-01 21:40:28 +00:00
|
|
|
kpatch core module uses to determine the list of functions that need
|
2014-02-11 18:00:17 +00:00
|
|
|
to be redirected using ftrace
|
2014-04-01 21:40:28 +00:00
|
|
|
- Generate the patch module
|
2014-02-18 16:15:13 +00:00
|
|
|
- Use `link-vmlinux-syms` to hardcode non-exported kernel symbols
|
2014-04-01 21:40:28 +00:00
|
|
|
into the symbol table of the patch module
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
### Patching
|
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
The patch modules register with the core module (`kpatch.ko`).
|
2014-02-18 16:15:13 +00:00
|
|
|
They provide information about original functions that need to be replaced, and
|
|
|
|
corresponding function pointers to the replacement functions.
|
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
The core module registers a trampoline function with ftrace. The
|
2014-02-18 16:15:13 +00:00
|
|
|
trampoline function is called by ftrace immediately before the original
|
|
|
|
function begins executing. This occurs with the help of the reserved mcount
|
|
|
|
call at the beginning of every function, created by the gcc `-mfentry` flag.
|
|
|
|
The trampoline function then modifies the return instruction pointer (IP)
|
|
|
|
address on the stack and returns to ftrace, which then restores the original
|
|
|
|
function's arguments and stack, and "returns" to the new function.
|
|
|
|
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
Limitations
|
|
|
|
-----------
|
|
|
|
|
2014-03-21 19:57:52 +00:00
|
|
|
- Patches which modify kernel modules are not supported (yet). Only
|
|
|
|
functions in the vmlinux file (listed in System.map) can be patched.
|
|
|
|
|
|
|
|
- Patches to functions which are always on the stack of at least one
|
|
|
|
process in the system are not supported. Examples: schedule(),
|
|
|
|
sys_poll(), sys_select(), sys_read(), sys_nanosleep(). Attempting to
|
|
|
|
apply such a patch will cause the insmod of the patch module to return
|
|
|
|
an error.
|
|
|
|
|
|
|
|
- Patches which modify init functions (annotated with `__init`) are not
|
|
|
|
supported. kpatch-build will return an error if the patch attempts
|
|
|
|
to do so.
|
|
|
|
|
|
|
|
- Patches which modify statically allocated data are not supported.
|
|
|
|
kpatch-build will detect that and return an error. (In the future
|
|
|
|
we will add a facility to support it. It will probably require the
|
2014-04-01 21:40:28 +00:00
|
|
|
user to write code which runs at patch module loading time which manually
|
2014-03-21 19:57:52 +00:00
|
|
|
updates the data.)
|
|
|
|
|
|
|
|
- Patches which change the way a function interacts with dynamically
|
|
|
|
allocated data might be safe, or might not. It isn't possible for
|
|
|
|
kpatch-build to verify the safety of this kind of patch. It's up to
|
|
|
|
the user to understand what the patch does, whether the new functions
|
|
|
|
interact with dynamically allocated data in a different way than the
|
|
|
|
old functions did, and whether it would be safe to atomically apply
|
|
|
|
such a patch to a running kernel.
|
2014-02-18 23:33:20 +00:00
|
|
|
|
|
|
|
|
2014-02-19 21:49:15 +00:00
|
|
|
Frequently Asked Questions
|
|
|
|
--------------------------
|
|
|
|
|
|
|
|
**Q. Isn't this just a virus/rootkit injection framework?**
|
|
|
|
|
|
|
|
kpatch uses kernel modules to replace code. It requires the `CAP_SYS_MODULE`
|
|
|
|
capability. If you already have that capability, then you already have the
|
|
|
|
ability to arbitrarily modify the kernel, with or without kpatch.
|
|
|
|
|
|
|
|
**Q. How can I detect if somebody has patched the kernel?**
|
|
|
|
|
2014-04-01 21:40:28 +00:00
|
|
|
We hope to create a new kernel TAINT flag which will get set whenever a patch
|
2014-02-19 21:49:15 +00:00
|
|
|
module is loaded.
|
|
|
|
|
|
|
|
Also, many distros ship with cryptographically signed kernel modules, and will
|
|
|
|
taint the kernel anyway if you load an unsigned module.
|
|
|
|
|
|
|
|
**Q. Will it destabilize my system?**
|
|
|
|
|
|
|
|
No, as long as the patch is chosen carefully. See the Limitations section
|
|
|
|
above.
|
|
|
|
|
|
|
|
**Q. Why does kpatch use ftrace to jump to the replacement function instead of
|
|
|
|
adding the jump directly?**
|
|
|
|
|
|
|
|
ftrace owns the first "call mcount" instruction of every kernel function. In
|
|
|
|
order to keep compatibility with ftrace, we go through ftrace rather than
|
|
|
|
updating the instruction directly.
|
|
|
|
|
|
|
|
**Q Is kpatch compatible with \<insert kernel debugging subsystem here\>?**
|
|
|
|
|
|
|
|
We aim to be good kernel citizens and maintain compatibility. A hot patch
|
|
|
|
replacement function is no different than a function loaded by any other kernel
|
|
|
|
module. Each replacement function has its own symbol name and kallsyms entry,
|
|
|
|
so it looks like a normal function to the kernel.
|
|
|
|
|
|
|
|
- **oops stack traces**: Yes. If the replacement function is involved in an
|
|
|
|
oops, the stack trace will show the function and kernel module name of the
|
|
|
|
replacement function, just like any other kernel module function. The oops
|
|
|
|
message will also show the taint flag. [TODO: taint flag]
|
|
|
|
- **kdump/crash**: Yes. Replacement functions are normal functions, so crash
|
|
|
|
will have no issues. [TODO: create patch module debuginfo symbols and crash
|
|
|
|
warning message]
|
|
|
|
- **ftrace**: Yes, see previous question.
|
|
|
|
- **systemtap/kprobes**: TODO: try it out
|
|
|
|
- **perf**: TODO: try it out
|
|
|
|
|
|
|
|
**Q. Why not use something like kexec instead?**
|
|
|
|
|
|
|
|
If you want to avoid a hardware reboot, but are ok with restarting processes,
|
|
|
|
kexec is a good alternative.
|
|
|
|
|
|
|
|
**Q. If an application can't handle a reboot, it's designed wrong.**
|
|
|
|
|
|
|
|
That's a good poi... [system reboots]
|
|
|
|
|
|
|
|
**Q. What changes are needed in other upstream projects?**
|
|
|
|
|
|
|
|
We hope to make the following changes to other projects:
|
|
|
|
|
|
|
|
- kernel:
|
|
|
|
- ftrace improvements to close any windows that would allow a patch to
|
|
|
|
be inadvertently disabled
|
|
|
|
- hot patch taint flag
|
|
|
|
- possibly the kpatch core module itself
|
|
|
|
|
|
|
|
- crash:
|
|
|
|
- make it glaringly obvious that you're debugging a patched kernel
|
|
|
|
- point it to where the patch modules and corresponding debug symbols
|
|
|
|
live on the file system
|
|
|
|
|
|
|
|
**Q: Is it possible to register a function that gets called atomically with
|
|
|
|
`stop_machine` when the patch module loads and unloads?**
|
|
|
|
|
|
|
|
We do have plans to implement something like that.
|
|
|
|
|
2014-02-21 03:10:09 +00:00
|
|
|
**Q. What kernels are supported?**
|
|
|
|
|
|
|
|
kpatch needs gcc >= 4.6 and Linux >= 3.7 for use of the -mfentry flag.
|
|
|
|
|
|
|
|
**Q. Is it possible to remove a patch?**
|
|
|
|
|
|
|
|
Yes. Just unload the patch module and the original function will be restored.
|
|
|
|
|
|
|
|
**Q. Can you apply multiple patches?**
|
|
|
|
|
|
|
|
Yes. Also, a single function can even be patched multiple times if needed.
|
|
|
|
|
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
Demonstration
|
|
|
|
-------------
|
|
|
|
|
|
|
|
A low-level demonstration of kpatch is available on Youtube:
|
2014-02-11 20:34:19 +00:00
|
|
|
|
|
|
|
http://www.youtube.com/watch?v=WeSmG-XirC4
|
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
This demonstration completes each step in the previous section in a manual
|
2014-02-18 16:15:13 +00:00
|
|
|
fashion. However, from a end-user perspective, most of these steps are hidden
|
2014-03-10 14:24:40 +00:00
|
|
|
by the "kpatch-build" command.
|
2014-02-18 16:15:13 +00:00
|
|
|
|
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
Get involved
|
|
|
|
------------
|
|
|
|
|
2014-03-19 15:29:07 +00:00
|
|
|
If you have questions or feedback, join the
|
|
|
|
[mailing list](https://www.redhat.com/mailman/listinfo/kpatch) and say hi.
|
2014-02-18 23:33:20 +00:00
|
|
|
|
2014-03-19 15:29:07 +00:00
|
|
|
Contributions are very welcome. Feel free to open issues or PRs on github.
|
|
|
|
For big PRs, it's a good idea to discuss them first in github issues or on the
|
|
|
|
[mailing list](https://www.redhat.com/mailman/listinfo/kpatch) before you write
|
|
|
|
a lot of code.
|
2014-02-18 23:33:20 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
License
|
|
|
|
-------
|
|
|
|
|
|
|
|
kpatch is under the GPLv2 license.
|
|
|
|
|
|
|
|
This program is free software; you can redistribute it and/or
|
|
|
|
modify it under the terms of the GNU General Public License
|
|
|
|
as published by the Free Software Foundation; either version 2
|
|
|
|
of the License, or (at your option) any later version.
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
along with this program; if not, write to the Free Software
|
|
|
|
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|