2014-02-17 15:54:25 +00:00
|
|
|
kpatch: dynamic kernel patching
|
|
|
|
===============================
|
2014-02-11 20:34:19 +00:00
|
|
|
|
|
|
|
kpatch is a tool for the generation and application of kernel
|
|
|
|
modules that patch a running Linux kernel while in operation,
|
|
|
|
without requiring a reboot. This is very valuable in cases
|
|
|
|
where critical workloads, which do not have high availability via
|
|
|
|
scale-out, run on a single machine and are very downtime
|
|
|
|
sensitive or require a heavyweight approval process and
|
|
|
|
notification of workload users in the event of downtime.
|
|
|
|
|
2014-02-19 03:32:24 +00:00
|
|
|
kpatch is currently in active development. For now, it should _not_ be used
|
2014-02-18 16:15:13 +00:00
|
|
|
in production environments until significantly more testing on various
|
|
|
|
patches and environments is conducted.
|
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
**WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss
|
|
|
|
may occur!**
|
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
Installation
|
|
|
|
------------
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
*NOTE: These installation instructions are currently Fedora-specific. Support
|
|
|
|
for other distributions is planned soon.*
|
|
|
|
|
|
|
|
Install the dependencies for compiling kpatch:
|
|
|
|
|
|
|
|
sudo yum install gcc kernel-devel elfutils elfutils-devel
|
|
|
|
|
|
|
|
*NOTE: Ensure you have elfutils-0.158 or newer.*
|
|
|
|
|
|
|
|
Install the dependencies for the "kpatch build" command:
|
|
|
|
|
|
|
|
sudo yum install rpmdevtools pesign
|
|
|
|
sudo yum-builddep kernel
|
|
|
|
|
2014-02-18 21:56:47 +00:00
|
|
|
# optional, but highly recommended
|
|
|
|
sudo yum install ccache
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
Compile kpatch:
|
2014-02-13 16:04:35 +00:00
|
|
|
|
|
|
|
make
|
2014-02-18 16:15:13 +00:00
|
|
|
|
|
|
|
Install kpatch to /usr/local:
|
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
sudo make install
|
|
|
|
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
Quick start
|
2014-02-17 15:54:25 +00:00
|
|
|
-----------
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
*NOTE: While kpatch is designed to work with any recent Linux
|
2014-02-13 16:04:35 +00:00
|
|
|
kernel on any distribution, the "kpatch build" command currently
|
2014-02-17 15:54:25 +00:00
|
|
|
only works on Fedora.*
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
Load the kpatch core module:
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
sudo insmod /usr/local/lib/modules/$(uname -r)/kpatch/kpatch.ko
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-18 04:37:07 +00:00
|
|
|
Make a source patch against the kernel tree:
|
|
|
|
|
|
|
|
# from a kernel git tree:
|
|
|
|
git diff > /path/to/foo.patch
|
|
|
|
|
|
|
|
Build the hot patch kernel module:
|
|
|
|
|
|
|
|
kpatch build /path/to/foo.patch
|
|
|
|
|
|
|
|
This outputs a hot patch module named `kpatch-foo.ko` in the current
|
|
|
|
directory. Now apply it to the running kernel:
|
|
|
|
|
|
|
|
sudo insmod kpatch-foo.ko
|
|
|
|
|
|
|
|
Done! The kernel is now patched.
|
2014-02-13 03:53:05 +00:00
|
|
|
|
2014-02-13 16:04:35 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
How it works
|
2014-02-17 15:54:25 +00:00
|
|
|
------------
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-02-19 03:32:24 +00:00
|
|
|
kpatch works at a function granularity: old functions are replaced with new
|
|
|
|
ones. It has four main components:
|
|
|
|
|
|
|
|
- **kpatch-build**: a collection of tools which convert a source diff patch to
|
|
|
|
a hot patch module. They work by compiling the kernel both with and without
|
|
|
|
the source patch, comparing the binaries, and generating a hot patch module
|
|
|
|
which includes new binary versions of the functions to be replaced.
|
|
|
|
|
|
|
|
- **hot patch module**: a kernel module (.ko file) which includes the
|
|
|
|
replacement functions and metadata about the original functions.
|
|
|
|
|
|
|
|
- **kpatch core module**: a kernel module (.ko file) which provides an
|
|
|
|
interface for the hot patch modules to register new functions for
|
|
|
|
replacement. It uses the kernel ftrace subsystem to hook into the original
|
|
|
|
function's mcount call instruction, so that a call to the original function
|
|
|
|
is redirected to the replacement function.
|
|
|
|
|
|
|
|
- **kpatch utility:** a command-line tool which allows a user to manage a
|
|
|
|
collection of hot patch modules. One or more hot patch modules may be
|
|
|
|
configured to be loaded at boot time, so that a system can remain patched
|
|
|
|
even after a reboot into the same version of the kernel.
|
|
|
|
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
### kpatch build
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
The "kpatch build" command converts a source-level diff patch file to a hot
|
|
|
|
patch kernel module. Most of its work is performed by the kpatch-build script
|
|
|
|
which uses a collection of utilities: `create-diff-object`,
|
|
|
|
`add-patch-section`, and `link-vmlinux-syms`.
|
2014-02-11 18:00:17 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
The primary steps in kpatch-build are:
|
|
|
|
- Build the unstripped vmlinux for the kernel
|
|
|
|
- Patch the source tree
|
|
|
|
- Rebuild vmlinux and monitor which objects are being rebuilt.
|
2014-02-11 18:00:17 +00:00
|
|
|
These are the "changed objects".
|
2014-02-18 16:15:13 +00:00
|
|
|
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
|
2014-02-11 18:00:17 +00:00
|
|
|
resulting in the changed patched objects
|
|
|
|
- Unpatch the source tree
|
2014-02-18 16:15:13 +00:00
|
|
|
- Recompile each changed object with `-ffunction-sections -fdata-sections`,
|
2014-02-13 13:49:02 +00:00
|
|
|
resulting in the changed original objects
|
2014-02-18 16:15:13 +00:00
|
|
|
- Use `create-diff-object` to analyze each original/patched object pair
|
2014-02-11 18:00:17 +00:00
|
|
|
for patchability and generate an output object containing modified
|
|
|
|
sections
|
2014-02-18 16:15:13 +00:00
|
|
|
- Link all the output objects into a cumulative object
|
|
|
|
- Use `add-patches-section` to add the .patches section that the
|
2014-02-11 18:00:17 +00:00
|
|
|
core kpatch module uses to determine the list of functions that need
|
|
|
|
to be redirected using ftrace
|
2014-02-13 16:51:00 +00:00
|
|
|
- Generate the patch kernel module
|
2014-02-18 16:15:13 +00:00
|
|
|
- Use `link-vmlinux-syms` to hardcode non-exported kernel symbols
|
2014-02-13 16:51:00 +00:00
|
|
|
into the symbol table of the patch kernel module
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
### Patching
|
|
|
|
|
|
|
|
The hot patch kernel modules register with the core module (`kpatch.ko`).
|
|
|
|
They provide information about original functions that need to be replaced, and
|
|
|
|
corresponding function pointers to the replacement functions.
|
|
|
|
|
|
|
|
The kpatch core module registers a trampoline function with ftrace. The
|
|
|
|
trampoline function is called by ftrace immediately before the original
|
|
|
|
function begins executing. This occurs with the help of the reserved mcount
|
|
|
|
call at the beginning of every function, created by the gcc `-mfentry` flag.
|
|
|
|
The trampoline function then modifies the return instruction pointer (IP)
|
|
|
|
address on the stack and returns to ftrace, which then restores the original
|
|
|
|
function's arguments and stack, and "returns" to the new function.
|
|
|
|
|
2014-02-11 20:34:19 +00:00
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
Limitations
|
|
|
|
-----------
|
|
|
|
|
|
|
|
- kpatch can't detect when a patch changes the contents of a dynamically
|
|
|
|
allocated data structure, and isn't able to determine whether such patches
|
|
|
|
are safe to apply. It's the user's responsibility to analyze any such
|
|
|
|
patches for safety before applying them.
|
|
|
|
- Patches which change the contents of static data structures are not currently
|
|
|
|
supported. kpatch build will detect such changes and report an error.
|
|
|
|
- Patches to functions which are always in the call stack of a task, such as
|
|
|
|
schedule(), will fail to apply at runtime.
|
|
|
|
- Patches which change functions that are only called in the kernel init path
|
|
|
|
will have no effect (obviously).
|
|
|
|
|
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
Demonstration
|
|
|
|
-------------
|
|
|
|
|
|
|
|
A low-level demonstration of kpatch is available on Youtube:
|
2014-02-11 20:34:19 +00:00
|
|
|
|
|
|
|
http://www.youtube.com/watch?v=WeSmG-XirC4
|
|
|
|
|
2014-02-17 15:54:25 +00:00
|
|
|
This demonstration completes each step in the previous section in a manual
|
2014-02-18 16:15:13 +00:00
|
|
|
fashion. However, from a end-user perspective, most of these steps are hidden
|
|
|
|
by the "kpatch build" command.
|
|
|
|
|
|
|
|
|
2014-02-18 23:33:20 +00:00
|
|
|
Get involved
|
|
|
|
------------
|
|
|
|
|
2014-02-19 03:32:24 +00:00
|
|
|
If you have questions, feedback, or you'd like to contribute, feel free to join
|
|
|
|
the mailing list at https://www.redhat.com/mailman/listinfo/kpatch and say hi.
|
2014-02-18 23:33:20 +00:00
|
|
|
|
|
|
|
|
2014-02-18 16:15:13 +00:00
|
|
|
License
|
|
|
|
-------
|
|
|
|
|
|
|
|
kpatch is under the GPLv2 license.
|
|
|
|
|
|
|
|
This program is free software; you can redistribute it and/or
|
|
|
|
modify it under the terms of the GNU General Public License
|
|
|
|
as published by the Free Software Foundation; either version 2
|
|
|
|
of the License, or (at your option) any later version.
|
|
|
|
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
|
|
along with this program; if not, write to the Free Software
|
|
|
|
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|