In ARM64, every function section should have its own pfe section.
It is a bug in GCC 11/12 which will only generate a single pfe
section for all functions. The bug has been fixed in GCC 13.1.
As the create-diff-object is generating the pfe sections on its own,
we should also fix this bug, instead of try to repeat the bug.
--
Adjusted whitespace in Zimao's proposed code.
Signed-off-by: Pete Swain <swine@google.com>
If CONFIG_UBSAN is enabled, ubsan section (.data..Lubsan_{data,type})
can be created. Keep them unconditionally.
NOTE: This patch needs to be verified.
Signed-off-by: Misono Tomohiro <misono.tomohiro@jp.fujitsu.com>
Generalized kpatch_line_macro_change_only() & insn_is_load_immediate()
to collapse the aarch64 support back into parent.
I'm assuming the 3rd start1 of the original
/* Verify mov w2 <line number> */
if (((start1[offset] & 0b11111) != 0x2) || (start1[offset+3] != 0x52) ||
((start1[offset] & 0b11111) != 0x2) || (start2[offset+3] != 0x52))
was a typo for start2.
That's now absorbed into insn_is_load_immediate() leaving just one
aarch64-specific piece: thinning out the match-list for diagnosing a
__LINE__ reference, to just "__warn_printf".
New toolchain/arch, new conventions for section/label/etc names
gcc's .LCx symbols point to string literals in '.rodata.<func>.str1.*'
sections. Clang creates similar .Ltmp%d symbols in '.rodata.str'
The function is_string_literal_section()
generalized (too much?) to match either
- clang's/arm64 /^\.rodata\.str$/
- gcc's /^\.rodata\./ && /\.str1\./
Various matchers for .data.unlikely .bss.unlikely
replaced by is_data_unlikely_section()
generalized to match
- gcc's ".data.unlikely"
- clang's ".(data|bss).module_name.unlikely"
.data.once handled similarly
Signed-off-by: Pete Swain <swine@google.com>
On arm64, kpatch_find_func_profiling_calls() was skipping
leaf functions, with no relocations, so they weren't patchable.
Here other archs need to walk a function's reloc entries to
check for __fentry__ or __mcount, so it's valid to skip over
functions without sym->sec->rela, because they cannot be patchable,
else they would have at least an __fentry__ call relocation.
But arm64 marks functions patchable in a different way, with per-func
__patchable_function_entries sections referring _to_ the func,
not relocations _within_ the func, so a function w/o relocations
for text or data can still be patchable.
Move the sym->sec->rela check to the per-arch paths.
This allows gcc-static-local-var-5.patch
to generate livepatch, on arm64 & x86
Suggested-By: Bill Wendling <morbo@google.com>
Signed-off-by: Pete Swain <swine@google.com>
new clang toolchain on arm64 produces individual __patchable_function_entries
sections for each patchable func, in -ffunction-sections mode,
rather than traditional single __mcount_loc section.
Bend the existing logic to detect this multiplicity in the incoming
kelf objects, and allocate N identical one-entry sections.
These are retrieved as needed by a new function: find_nth_section_by_name()
and attached to the .text sections they describe.
These __pfe section are not actually arm64-specific, but a generic
enhancement across gcc & clang, to allow better garbage collection of
unreferenced object sections, and mcount/pfe objects which refer to them.
The __pfe sections are combined in kernel-or-module final link,
from 5.19.9's 9440155ccb948f8e3ce5308907a2e7378799be60.
From clang-11, __pfe is supported for x86, though not yet used by kernel
The split between allocate/populate phases here is necessary to
enumerate/populate the outgoing section-headers before beginning to
produce output sections
Also adds some missing \n to log_debug()s
Signed-off-by: Pete Swain <swine@google.com>
It seems mapping symbols in aarch64 elf has section size of 0.
So, exclude it in section symbol replacing code just like
kpatch_correlate_symbols().
This fixes the data-read-mostly unit test on aarch64.
Signed-off-by: Misono Tomohiro <misono.tomohiro@jp.fujitsu.com>
Add the final support required for aarch64 and enable building on that arch.
Signed-off-by: Suraj Jitindar Singh <surajjs@amazon.com>
---
V1->V2:
- Add # shellcheck disable=SC2086
- Add comment to kpatch_is_mapping_symbol()
The __mcount_loc section contains the addresses of patchable ftrace sites which
is used by the ftrace infrastructure in the kernel to create a list of tracable
functions and to know where to patch to enable tracing of them. On aarch64 this
section is called __patchable_function_entries and is generated by the
compiler. Either of __mcount_loc or __patchable_function_entries is recognised
by the kernel but for aarch64 use __patchable_function_entries as it is what
is expected.
Add aarch64 support to kpatch_alloc_mcount_sections(). The SHF_LINK_ORDER
section flag must be copied to ensure that it matches to avoid the following:
ld: __patchable_function_entries has both ordered [...] and unordered [...] sections
Add aarch64 support to kpatch_populate_mcount_sections(). Check for the 2
required NOP instructions on function entry, which may be preceded by a BTI C
instruction depending on whether the function is a leaf function. This
determines the offset of the patch site.
Signed-off-by: Suraj Jitindar Singh <surajjs@amazon.com>
---
V1->V2:
- Don't preserve the __patchable_function_entries section from the patched elf
as this is already verified by kpatch_check_func_profiling_calls()
- Instead get the patch entry offset by checking for a preceding BTI C instr
- Copy the section flags for __patchable_function_entries
---
rebased, added sh_link fix from Suraj's later commit
"kpatch-build: Enable ARM64 support"
Signed-off-by: Pete Swain <swine@google.com>
The function kpatch_create_mcount_sections() allocates the __mcount_loc section
and then populates it with functions which have a patchable entry.
The following patch will add aarch64 support to this function where the
allocation will have to be done before the kelf_patched is torn down.
Thus split this function so that the allocation can be performed earlier and
the populating as before.
No intended functional change.
Signed-off-by: Suraj Jitindar Singh <surajjs@amazon.com>
---
V1->V2:
- Add patch to series
The "has_function_profiling" support field in the symbol struct is used to show
that a function symbol is able to be patched. This is necessary to check that
functions which need to be patched are able to be.
On arm64 this means the presence of 2 NOP instructions at function entry
which are patched by ftrace to call the ftrace handling code. These 2 NOPs
are inserted by the compiler and the location of them is recorded in a
section called "__patchable_function_entries". Check whether a symbol has a
corresponding entry in the "__patchable_function_entries" section and if so
mark it as "has_func_profiling".
Signed-off-by: Suraj Jitindar Singh <surajjs@amazon.com>
---
V1->V2:
- Make error message standard across architectures when no patchable entry
- Don't store __patchable_function_entries section in
kpatch_find_func_profiling_calls(), instead find it each time
Do we need more robust architecture protection (Issue #1356)
The elf.h reloc-type constants are not unique across archs
#define R_PPC64_REL24 10 /* PC relative 26 bit */
#define R_X86_64_32 10 /* Direct 32 bit zero extended */
so to avoid any unexpected aliasing, guard all R_arch_type refs
with a check on kelf->arch, or a global default arch set from
the first elf encountered.
Kernel v6.2+ commit bea75b33895f ("x86/Kconfig: Introduce function
padding") added 16 bytes of NOP padding in front of each function.
For objects built with --function-sections, this means that function
symbols no longer sit at the beginning of their respective ELF sections,
but 16 bytes offset.
In the same release, kernel v6.2+ commit 9f2899fe36a6 ("objtool: Add
option to generate prefix symbols") adds ELF function symbols with
prefix "__pfx_" to indicate the start of a function, inclusive of
NOP-padding.
For example:
$ objdump -Dr -j.text.cmdline_proc_show fs/proc/cmdline.o
...
Disassembly of section .text.cmdline_proc_show:
0000000000000000 <__pfx_cmdline_proc_show>:
0: 90 nop
1: 90 nop
2: 90 nop
3: 90 nop
4: 90 nop
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
0000000000000010 <cmdline_proc_show>:
10: e8 00 00 00 00 callq 15 <cmdline_proc_show+0x5>
11: R_X86_64_PLT32 __fentry__-0x4
15: 55 push %rbp
16: 48 8b 35 00 00 00 00 mov 0x0(%rip),%rsi # 1d <cmdline_proc_show+0xd>
19: R_X86_64_PC32 saved_command_line-0x4
1d: 48 89 fd mov %rdi,%rbp
20: e8 00 00 00 00 callq 25 <cmdline_proc_show+0x15>
21: R_X86_64_PLT32 seq_puts-0x4
25: 48 89 ef mov %rbp,%rdi
28: be 0a 00 00 00 mov $0xa,%esi
2d: e8 00 00 00 00 callq 32 <cmdline_proc_show+0x22>
2e: R_X86_64_PLT32 seq_putc-0x4
32: 31 c0 xor %eax,%eax
34: 5d pop %rbp
35: e9 00 00 00 00 jmpq 3a <cmdline_proc_show+0x2a>
36: R_X86_64_PLT32 __x86_return_thunk-0x4
Kpatch-build needs to gracefully handle NOP-padding when it is present.
At the same time, it should include "__pfx_<function>" symbols when
their respective functions change, but not treat prefix such functions
as first-class functions.
This also adds support for CONFIG_CFI_CLANG, which also creates prefixed
symbols with the name "__cfi_<function>".
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Kernel v6.2+ commit bea75b33895f ("x86/Kconfig: Introduce function
padding") introduces the -fpatchable-function-entry=16,16 build flag on
x86. This leverages compiler support for generating a
__patchable_function_entries section similar to __mcount_loc.
That said, x86 still utilizes __mcount_loc even when
__patchable_function_entries exists. The latter point to the __pfx
symbols, but the section is discarded in the vmlinux link and isn't used
regardless, for ftrace or for any other purpose.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Like unsupported jump labels and static call sites, batch report all
unsupported sibling calls so the kpatch developer doesn't need to
iteratively find them all.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Kernel v6.2+ commits 00abd3840812 ("objtool: Add .call_sites section")
and e81dc127ef69 ("x86/callthunks: Add call patching for call depth
tracking") added .call_sites sections to object files. These are filled
with an array of s32 values.
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
The term "dynrela" was invented before klp relocations ever existed.
They're basically the same thing: special livepatch-specific relocations
which get applied when the patched object gets loaded or patched.
They're necessary due to a) the need to access unexported symbols; and
b) late module patching.
The different names are confusing. Consolidate them by replacing
"dynrela" with "klp_reloc" (code) or "klp relocation" (English).
Note there's still some antiquated code in the kpatch core module and in
the pre-4.7 klp patch template which still use the "dynrela" naming.
That code is mostly dead anyway so I've just left it alone.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Otherwise on recent distros it appends the errno to the error message,
like:
create-diff-object: ERROR: x86.o: kpatch_regenerate_special_section: 2633: Found 1 unsupported static call(s) in the patched code. Use KPATCH_STATIC_CALL() instead.: Success
which is not what we want in most cases.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Similar to jump labels, static calls aren't supported when the static
call key was originally defined in a module rather than in vmlinux.
Detect those cases and either remove them (in the case of tracepoints)
or error out.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Convert the hard-coded should_keep_jump_label() to a proper callback,
since static calls will need a similar filter.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Align the s390 special_section initializers to improve readability and
for consistency with the rest.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
kpatch_mangled_strcmp() only ignores the digits after the period, but in
the case of __UNIQUE_ID(), the symbol names have random digits before
the period due to the use of `__COUNTER__`. Make sure such symbols are
properly correlated.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Clang FDO adds a new, ignorable ELF section, .llvm.call-graph-profile
Generalize to ignore all .llvm.*
Signed-off-by: Pete Swain <swine@google.com>
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com> [subject line]
If two sections want to be the same, they need to satisfy
two conditions:
1) the result of memcmp is zero, which means they
have the same content.
2) they have the same relocation entries.
In one specific situation, two sections have the same content.
But one section has relocation entries while the other one has
no relocation entries. For example, in X86, consider the
following code:
original code
```
__noreturn noinline int kpatch_func(void)
{
while(1) {};
}
```
patched code
```
__noreturn notrace noinline int kpatch_func(void)
{
asm(".byte 0xe8, 0x00, 0x00, 0x00, 0x00");
while(1){};
}
```
Since the original code has a fentry call, these two functions have
the same compile result. But obviously, they are different functions.
Currently, kpatch would not find their differences since the patched
code has no relocation entries.
For the situation that one section has relocation entries while the
other one doesn't have, it should be set to be changed directly.
Cooperated-by: Zongwu Li <lizongwu@huawei.com>
Signed-off-by: Longjun Luo <luolongjuna@gmail.com>
After patched, rela information for some sections could
disappear. For example, a function like the following:
"
notrace noinline static int version_proc_show(struct seq_file *m,
void *v)
{
return 0;
}
"
Apart from common rela entries, trace and return thunk mechanism
will generate rela information. Use `notrace` to remove the
effect of trace. Make CONFIG_RETHUNK=n can remove the effect of
return thunk.
Discovered-by: Zongwu Li <lizongwu@huawei.com>
Signed-off-by: Longjun Luo <luolongjuna@gmail.com>
Recent toolchains only create a section symbol if it's needed, i.e. if
there's a reference to it. If there's a missing section symbol in
kpatch_create_intermediate_sections(), create one instead of erroring
out.
Fixes#1272.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
* Add s390 specific checks
* Identify patchable functions.
* Dont mark expolines as dynrelas. These expolines are always included
in final kernel module. This ensures that expoline functions and the
kernel itself are not too far apart and avoids out of range
relocation. However, this isnt a problem for other functions, as these
relocations are performed via R_390_PLT32DBL using gcc option
-mno-pic-data-is-text-relative.
* s390 maintains expoline tables to locate the expoline thunks. If
needed, the module loader could later replace these expoline thunks
with normal indirect branch. Each element in the expoline table is of 4
bytes. If there is a changed function in rela.s390_return*, then mark
that specific rela symbol as included. This is already performed in the
processing of special sections. Hence include it.
Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>
If a section reference can't be converted to a symbol reference, error
out to try to prevent unexpected behavior later on.
There are a few sections for which a symbol is optional: .rodata and
string literal sections. Don't warn about those.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
rela_insn() only cares about the base section. Convert it to take a
non-rela section as its argument instead of a relasec.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Add support for R_X86_64_NONE. With an upstream kernel, it's quite
rare, only used for a few jump labels. With older kernels it was used
for fentry hooks. Either way, it should be treated like a PC-relative
relocation.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
It doesn't make sense to disassemble a data section. That just happened
to work by accident. PC-relative offsets only need adjusting when
associated with an instruction.
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
