time when analyzing dumpfiles/systems with extremely large task
counts. For example, running with a dumpfile containing over a
million tasks, startup time and "ps" processing time was reduced
from 45 minutes to less then 40 seconds.
(gthelen@google.com)
calculating phys_base and the mapped kernel offset for KASLR-enabled
kernels on SADUMP dumpfiles by using a technique developed by Takao
Indoh. Originally, the patchset included support for kdumps, but this
was dropped in v2, as it was deemed unnecessary due to the upstream
implementation of the "vmcoreinfo device" in QEMU. However, there
are still several reasons for which the vmcoreinfo device may not be
present at the time when a memory dump is taken from a VM, ranging
from a host running older QEMU/libvirt versions, to misconfigured VMs
or environments running Hypervisors that doesn't support this device.
This patchset generalizes the KASLR-related functions from sadump.c
and moves them to kaslr_helper.c, and makes kdump analysis fall back
to KASLR offset calculation if vmcoreinfo data is missing.
(slp@redhat.com)
when the VM was suspended. This patch enables crash to read the
relevant registers from each vCPU state for use as the starting hooks
by the "bt" command. Also, support for "help -[D|n]" to display
dumpfile contents, and "help -r" to display vCPU register sets has
been implemented. This is also the first step towards implementing
automatic KASLR offset calculations for VMSS dumpfiles.
(slp@redhat.com)
the kernel is configured with CONFIG_SPARSEMEM_VMEMMAP, the plugin
function optimizes the mem_section search, reducing the computation
effort and time consumed by commands that repeatedly call the
is_page_ptr() function on large-memory systems.
(k-hagio@ab.jp.nec.com)
the maximum SPARSEMEM section number during initialization, and
use it as the topmost delimeter in subsequent mem_section searches.
Also allow for per-architecture machdep->is_page_ptr() plugin functions.
(anderson@redhat.com)
later kernels. Without the patch, the backtraces of the active
tasks in a kdump-generated dumpfile are truncated. Without the
patch, the panic task will just show the "crash_kexec" frame
and the kernel-entry user-space exception frame; the non-panic
tasks will show their backtraces starting from the stackframe
addresses captured in the per-cpu NT_PRSTATUS notes, and will
not display the exception frame generated by the NMI callback,
nor any stackframes on the IRQ stack.
(anderson@redhat.com)
patch is a cleanup/collaboration of the original logic used by the
various vtop functions, where several new common functions have been
added for extracting page table entries from PGD, P4D, PUD, PMD and
PTE pages. The usage of the former PML4 and UPML pages have been
replaced with the use of the common PGD page, and use the PUD page
in 4-level page table translation. Support for 5-level page tables
has been incorporated into the the existing x86_64_kvtop() and
x86_64_uvtop_level4() functions. Backwards compatibility for older
legacy kernels has been maintained. The third phase of support will
automatically detect whether the kernel proper, and whether an
individual user task, is utilizing 5-level page tables. This patch
enables support for kernel-only 5-level page tables by entering the
command line option "--machdep vm=5level".
(douly.fnst@cn.fujitsu.com)
SADUMP dumpfile facility, where this patch fixes a problem when Page
Table Isolation(PTI) is enabled. When PTI is enabled, bit 12 of CR3
register is used to split user space and kernel space. Also bit 11:0
is used for Process Context IDentifiers(PCID). To open an SADUMP
dumpfile, the value of CR3 is used to calculate KASLR offset and
phys_base; this patch masks the CR3 register value correctly for
a PTI enabled kernel.
(indou.takao@jp.fujitsu.com)
"bt" command may indicate "bt: cannot transition from exception stack
to current process stack" if the crash callback NMI occurred while an
active task was running on the new entry trampoline stack. This has
only been tested on the RHEL7 backport of the upstream patch because
as of this commit, crash does not run on 4.15-rc kernels. Further
changes may be required for upstream kernels, and distributions that
implement the kernel changes differently than upstream.
(anderson@redhat.com)
backports of, kernel commit 4950d6d48a0c43cc61d0bbb76fb10e0214b79c66,
titled "x86/dumpstack: Remove 64-byte gap at end of irq stack".
Without the patch, backtraces fail to transition from the IRQ stack
back to the process stack, showing an error message such as
"bt: cannot transition exception stack to IRQ stack to current
process stack".
(anderson@redhat.com)
commit e8cfbc245e24887e3c30235f71e9e9405e0cfc39, titled "pid: remove
pidhash". The kernel's traditional usage of a pid_hash[] array to
store PIDs has been replaced by an IDR radix tree, requiring a new
crash plug-in function to gather the system's task set. Without the
patch, the crash session fails during initialization with the error
message "crash: cannot resolve init_task_union".
(anderson@redhat.com)
hash page table geometry accomodates a larger virtual address range.
Without the patch, the virtual-to-physical translation of user space
virtual addresses by "vm -p", "vtop", and "rd -u" may generate an
invalid translation or otherwise fail.
(hbathini@linux.vnet.ibm.com)
force the embedded gdb module to complete its two-stage strategy
used for reading debuginfo symbol tables from module object files,
and then retry the line number extraction. This automatically does
what the "mod -r" or "crash --readnow" options accomplish.
(anderson@redhat.com)
contain commit 2482ddec670fb83717d129012bc558777cb159f7, titled
"mm: add SLUB free list pointer obfuscation". Without the patch,
there will numerous error messages of the type "kmem: <cache name>
slab: <address> invalid freepointer: <obfuscated address>".
(anderson@redhat.com)
dumpfile facility. SADUMP dumpfile headers do not contain phys_base
or VMCOREINFO notes, so without this patch, the crash session fails
during initialization with the message "crash: seek error: kernel
virtual address: <address> type: "page_offset_base". This patch
calculates the phys_base value and the KASLR offset using the IDTR
and CR3 registers from the dumpfile header.
(indou.takao@jp.fujitsu.com)
Without the patch, a message indicating "crash: builtin stackframe.sp
offset incorrect!" is issued during session initialization, and the
"bt" command fails with the error message "bt: invalid structure
member offset: task_struct_thread_context_sp".
(anderson@redhat.com)
x86_64 "bt" command. Kernels configured with CONFIG_ORC_UNWINDER
contain .orc_unwind and .orc_unwind_ip sections that can be queried
to determine the stack frame size of any text address within a kernel
function. For kernels not configured with CONFIG_FRAME_POINTER,
the crash utility does frame size calculation by disassembling a
function from its beginning to the specified text address, counting
the push, pop, and add/sub rsp instructions, accounting for retq
instructions that occur in the middle of a function. With this patch,
access to the new ORC sections has been plugged into the existing
frame size calculator, resulting in a more efficient and accurate
manner of determining frame sizes, and as a result, more accurate
backtraces.
(anderson@redhat.com)
frame at the top of the stack in Linux 4.7 and later kernels.
Without the patch, the contents of the user mode exception frame are
invalid due to the miscalculation of the starting address of the
pt_regs structure on the kernel stack.
(anderson@redhat.com)
the max real address to 53 bits. Without this patch, the warning
message "WARNING: cannot access vmalloc'd module memory" appears
during initialization, and any command that attempts to read a
vmalloc'd kernel virtual address will fail and display "read error"
messages.
(hbathini@linux.vnet.ibm.com)
sets of virtual memory offsets have been #define'd and helper macros
and placeholder functions for the p4d page tables have been added.
The only functional changes with this patchset are dynamically-set
PGDIR_SHIFT and PHYSICAL_MASK_SHIFT values that are based upon the
kernel configuration.
(anderson@redhat.com)
containing kernel commit 0100301bfdf56a2a370c7157b5ab0fbf9313e1cd,
subject "sched/x86: Rewrite the switch_to() code". Without the
patch, backtraces for inactive (sleeping) tasks fail with the message
"bt: invalid structure member offset: task_struct_thread_eip".
(anderson@redhat.com)
by the "kmem" command when SLAB_RED_ZONE has been enabled in Linux 4.6 and later
kernels. By default, CONFIG_SLUB object addresses that are displayed by the
"kmem" command will point to the SLAB_RED_ZONE padding inserted at the
beginning of the object. As an alternative, a new "redzone" environment
variable has been addedd that can be toggled on or off. If "set redzone off"
is entered, the object addresses will point to the address that gets returned
to the allocator.
(hirofumi@mail.parknet.co.jp, anderson@redhat.com)
in Linux 4.6 and later kernels due to upstream changes in the radix
tree facility. Without the patch, the commands will fail with the
message "radix trees do not exist or have changed their format".
(hirofumi@mail.parknet.co.jp)
"taint/module: Clean up global and module taint flags handling".
Without the patch, when running against Linux 4.10-rc1 and later
kernels, the crash utility fails during session initialization with
the message "crash: invalid structure size: tnt".
(panand@redhat.com)
for Linux 4.6 commit a7f8de168ace487fa7b88cb154e413cf40e87fc6, which
allows the ARM64 kernel image to be loaded anywhere in physical
memory. Without the patch, attempting to run live on an ARM64
Linux 4.6 and later kernel may display the warning message "WARNING:
cannot read linux_banner string", and then fails with the message
"crash: vmlinux and /dev/crash do not match!". Version 1.3 of the
crash driver is required, which introduces a new ioctl command that
retrieves the ARM64-only "kimage_voffset" value that is required for
virtual-to-physical address translation.
(anderson@redhat.com)
in Linux 4.9 and later kernels configured with CONFIG_RANDOMIZE_BASE.
Without the patch, the command shows a value of ffffea0000000000 next
to "KERNEL VMEMMAP BASE".
(anderson@redhat.com)
kernels configured with CONFIG_RANDOMIZE_BASE. Without the patch,
the crash session may fail during initialization with the message
"crash: vmlinux and vmcore do not match!".
(anderson@redhat.com)
dumpfiles:
(1) If ELF notes are not available, read them from the kernel's
crash_notes.
(2) If an online CPUs did not save its ELF notes, then adjust
the mapping of each ELF note to its CPU accordingly.
(rabinv@axis.com)
and c65eacbe290b8141554c71b2c94489e73ade8c8d, which have introduced a
new CONFIG_THREAD_INFO_IN_TASK configuration. This configuration
moves each task's thread_info structure from the base of its kernel
stack into its task_struct. Without the patch, the crash session
fails during initialization with the error "crash: invalid structure
member offset: thread_info_cpu".
(anderson@redhat.com)
whose device driver uses the blk-mq interface. Currently "dev -d"
always displays 0 in all fields for the blk-mq disk because blk-mq
does not increment/decrement request_list.count[2] on I/O creation
and I/O completion. The following values are used in blk-mq in such
situations:
- I/O creation: blk_mq_ctx.rq_dispatched[2]
- I/O completion: blk_mq_ctx.rq_completed[2]
So, we can get the counter of in-progress I/Os as follows:
in progress I/Os == rq_dispatched - rq_completed
This patch displays the result of above calculation for the disk.
It determines whether the device driver uses blk-mq if the
request_queue.mq_ops is not NULL. The "DRV" field is displayed as
"N/A(MQ)" if the value for in-flight in the device driver does not
exist for blk-mq.
(m.mizuma@jp.fujitsu.com)
that are not declared as "char *" types. Change two prior direct
callers of resizebuf() to use RESIZEBUF(), and fix two prior users of
RESIZEBUF() to correctly calculate the need to resize their buffers.
(anderson@redhat.com)
As both radix and hash MMU are supported in a single kernel on
Power ISA 3.0 based server processors, identify the current MMU
type and set page table index values accordingly. Also, in Linux
4.7 and later kernels, PPC64/BOOK3S uses the same masked bit values
in page table entries for 4K and 64K page sizes.
(hbathini@linux.vnet.ibm.com)
in which Thomas Gleixner redesigned the kernel timer mechanism to
switch to a non-cascading wheel. Without the patch, the "timer"
command fails with the message "timer: zero-size memory allocation!
(called from <address>)"
(anderson@redhat.com)
changes in kernel page table management accommodating it. This patch
series makes appropriate changes here to work for such kernels.
Also, this series fixes a few bugs along the way:
ppc64: fix vtop page translation for 4K pages
ppc64: Use kernel terminology for each level in 4-level page table
ppc64/book3s: address changes in kernel v4.5
ppc64/book3s: address change in page flags for PowerISA v3.0
ppc64: use physical addresses and unfold pud for 64K page size
ppc64/book3s: support big endian Linux page tables
The patches are needed for Linux v4.5 and later kernels on all
ppc64 hardware.
(hbathini@linux.vnet.ibm.com)
that are configured with CONFIG_RANDOMIZE_MEMORY, which randomizes
the base addresses of the kernel's unity-map address (PAGE_OFFSET),
and the vmalloc region. Without the patch, the crash utility fails
with a segmentation violation during session initialization.
(anderson@redhat.com)
each command's -s option, but instead of parsing gdb output, member
values are read directly from memory, so the command is much faster
for 1-, 2-, 4-, and 8-byte members.
(Alexandr_Terekhov@epam.com)
"bt" option, which can be accessed using "bt -o", and where "bt -O"
will toggle the original and optional methods as the default. The
original backtrace method has adopted two changes/features from
the optional method:
(1) ORIG_X0 and SYSCALLNO registers are not displayed in kernel
exception frames.
(2) stackframe entry text locations are modified to be the PC
address of the branch instruction instead of the subsequent
"return" PC address contained in the stackframe link register.
Accordingly, these are the essential differences between the original
and optional methods:
(1) optional: the backtrace will start with the IPI exception frame
located on the process stack.
(2) original: the starting point of backtraces for the active,
non-crashing, tasks, will continue to have crash_save_cpu()
on the IRQ stack as the starting point.
(3) optional: the exception entry stackframe adjusted to be located
farther down in the IRQ stack.
(4) optional: bt -f does not display IRQ stack memory above the
adjusted exception entry stackframe.
(5) optional: may display "(Next exception frame might be wrong)".
(takahiro.akashi@linaro.org, anderson@redhat.com)
attached to it. While kernel threads typically do not have an
mm_struct referencing a user-space virtual address space, they can
either temporarily reference one for a user-space copy operation, or
in the case of KVM "vhost" kernel threads, keep a reference to the
user space of the "quem-kvm" task that created them. Without the
patch, they will be mistaken for user tasks; the "bt" command will
display an invalid kernel-entry exception frame that indicates
"[exception RIP: unknown or invalid address]", the "ps" command
will not enclose the command name with brackets, and the "ps -[uk]"
and "foreach [user|kernel]" options will show the kernel thread as
a user task.
(anderson@redhat.com)
all tasks for evidence of stack overflows. It does so by verifying
the thread_info.task address, ensuring the thread_info.cpu value is
a valid cpu number, and checking the end of the stack for the
STACK_END_MAGIC value.
(anderson@redhat.com)
page tables in crash-7.1.5, which fails to translate user space
virtual addresses. Without the patch, "vtop <user-space address>"
fails to translate all user-space addresses, and any command that
needs to either translate or read user-space memory, such as "vm -p",
"ps -a", and "rd -u" will fail.
(anderson@redhat.com)
layout and associated KASLR support that was introduced in Linux 4.6.
The kernel text and static data has been moved from unity-mapped
memory into the vmalloc region, and its start address can be
randomized if CONFIG_RANDOMIZE_BASE is configured. Related support
is being put into the kernel's kdump code, the kexec-tools package,
and makedumpfile(8); with that in place, the analysis of Linux 4.6
ARM64 dumpfiles with or without KASLR enabled should work normally
by entering "crash vmlinux vmcore". On live systems, Linux 4.6 ARM64
kernels will only work automatically if CONFIG_RANDOMIZE_BASE is not
configured. Unfortunately, if CONFIG_RANDOMIZE_BASE is configured
on a live system, two --machdep command line arguments are required,
at least for the time being. The arguments are:
--machdep phys_offset=<base physical address>
--machdep kimage_voffset=<kernel kimage_voffset value>
Without the patch, any attempt to analyze a Linux 4.6 ARM64 kernel
fails during initialization with a stream of "read error" messages
followed by "crash: vmlinux and vmcore do not match!".
(takahiro.akashi@linaro.org)
are not configured with CONFIG_FUNCTION_GRAPH_TRACER. Without the
patch, backtraces that originate from a per-cpu IRQ stack will dump
an invalid exception frame before transitioning to the process stack.
(anderson@redhat.com)
are specified by the QEMU mem-path argument of a memory-backend-file
object. This allows the running of a live crash session against a
QEMU guest from the host machine. In this example, the /tmp/MEM file
on a QEMU host represents the guest's physical memory:
$ qemu-kvm ...other-options... \
-object memory-backend-file,id=MEM,size=128m,mem-path=/tmp/MEM,share=on \
-numa node,memdev=MEM -m 128
and a live session run can be run against the guest kernel like so:
$ crash <path-to-guest-vmlinux> live:/tmp/MEM@0
By prepending the ramdump image name with "live:", the crash session will
act as if it were running a normal live session.
(oleg@redhat.com)
version supports running against a live kernel. Compressed kdump
support is also here, but the crash dump support for the kernel,
kexec-tools, and makedumpfile is still pending. Initial work was
done by Karl Volz with help from Bob Picco.
(dave.kleikamp@oracle.com)
option searches for data structures of a specified size or within a
range of specified sizes. The -m option searches for data structures
that contain a member of a given type. If a structure contains
another structure, the members of the embedded structure will also
be subject to the search. The type string may be a substring of the
data type name. The output displays the size and name of the data
structure.
(Alexandr_Terekhov@epam.com, anderson@redhat.com)
kmem_cache shown by "kmem -s" in kernels configured with CONFIG_SLUB.
Without the patch, the values under the ALLOCATED column may be too
large because cached per-cpu objects are counted as allocated.
(vinayakm.list@gmail.com)
which were introduced in Linux 4.5 by this commit:
commit 132cd887b5c54758d04bf25c52fa48f45e843a30
arm64: Modify stack trace and dump for use with irq_stack
Without the patch, if an active task was operating on its per-cpu
IRQ stack on dumpfiles generated by kdump, its backtrace would start
at the exception frame that was laid down on the process stack.
This patch also adds support for "bt -E" to search IRQ stacks for
exception frames, and the "mach" command displays the addresses
of each per-cpu IRQ stack.
(anderson@redhat.com)
commit ccbf62d8a284cf181ac28c8e8407dd077d90dd4b, which changed the
task_struct.start_time member from a struct timespec to a u64.
Without the patch, the "RUN TIME" value is nonsensical.
(anderson@redhat.com)
region between 0xffffffffff600000 and 0xffffffffffe00000. Without
the patch, the reading of addresses in that region returns invalid
data; in addition, the "vtop" command for an address in that region
shows an invalid physical address under the "PHYSICAL" column.
(nakajima.akira@nttcom.co.jp, anderson@redhat.com)
during session initialization. This includes the system's cpu maps,
xtime, and utsname data. These require at least "-d1" as a command
line option value, and are primarily useful as an aide for debugging
suspect dumpfiles that fail during session initialization.
(anderson@redhat.com)
conjunction with "-s", and requires that the "start" address is the
address of a list_head, or other similar list linkage structure whose
first member points to the next linkage structure. The "-l <offset>"
argument is the offset of the embedded list linkage structure in the
specified "-s" data structure; it can be either a number of bytes or
expressed in "struct.member" format.
(anderson@redhat.com)
the kernel has dynamically downsized from the size indicated by the
debuginfo data. At this time, only "kmem_cache" and "task_struct"
structures that have been downsized are registered, but others may be
added in the future. If a downsized data structure is passed to gdb
for display, gdb will request a read of the "full" data structure,
which may flow into a memory region that was either filtered by
makedumpfile(8), or perhaps into non-existent memory, thereby killing
the generating command immediately due to a partial read. With this
patch, commands such as "struct" and "task" that reference downsized
data structures will have their reads flagged to return successfully
if partial read error occurs.
(anderson@redhat.com)
which have offline cpus, and therefore will not contain associated
NT_PRSTATUS notes in the dumpfile header. Without the patch, if
there are any offline cpus, a segmentation violation is generated
during session initialization.
(anderson@redhat.com)
and compressed kdump clones. The patch utilizes the NT_PRSTATUS
notes in the dumpfile headers instead of reading them from the
kernel's "crash_notes", which are not initialized when QEMU generates
a dumpfile. Without the patch, these warning messages are displayed
during session initialization:
WARNING: invalid note (n_type != NT_PRSTATUS)
WARNING: cannot retrieve registers for active tasks
and running "bt" on an active task causes a segmentation violation.
(drjones@redhat.com)
containing this Linux 4.4 commit:
commit 9d372c9fab34cd8803141871195141995f85c7f7
arm64: Add page size to the kernel image header
Without the patch, the kernel page size is calculated by looking
at the size of the "swapper_pg_dir" page directory. With this
update, the page size can be determined by checking a flag built
into the kernel image header, available in the "_kernel_flags_le"
absolute symbol.
(drjones@redhat.com)
a system without the crash.ko memory driver, and whose kernel was
configured with CONFIG_STRICT_DEVMEM. Without the patch, if any
-d<value> is entered on the command line, the crash session fails
during initialization.
(dmair@suse.com)
which contain this kernel commit:
commit 1d798ca3f16437c71ff63e36597ff07f9c12e4d6
mm: make compound_head() robust
The commit above removes the PG_tail and PG_compound page.flags bits
and the page.first_page member, and introduces a page.compound_head
member, which is a pointer to the head page and whose bit 0 acts as
the tail flag. Without the patch, a SLAB or SLUB warning message
that indicates "cannot determine how compound pages are linked" is
displayed during initialization, and any command that tracks compound
pages will be affected.
(anderson@redhat.com)
header of compressed kdumps, which is set by the new -e option to
the makedumpfile(8) facility. The -e option excludes kernel pages
that contain nothing but kernel page structures for pages that are
not being included in the dump. If the bit is set in the dumpfile,
the crash utility will issue a warning that the dumpfile is known to
be incomplete during initialization, just prior to the system banner
display.
(anderson@redhat.com)
"dis -s" option if the kernel source code is not located in the
standard location that is compiled into the kernel's debuginfo data.
The directory argument should point to the top-level directory of the
kernel source tree.
(anderson@redhat.com)
contain this kernel commit that modifies the tvec_root and tvec
data structures:
commit bc7a34b8b9ebfb0f4b8a35a72a0b134fd6c5ef50
timer: Use hlist for the timer wheel hash buckets
Without the patch, the "timer" command will spew messages indicating
"timer: invalid list entry: 0", followed by "timer: ignoring faulty
timer list at index <number> of timer array".
(anderson@redhat.com)
instructions. Without the patch, "dis [-f] <function>" may continue
beyond the end of a function, disassembling the memory that is in
between the target function and the next function.
(anderson@redhat.com)
the "snap.so" extension module as "(ACTIVE)" in the STATE field of
the initial system banner and the "set" command. Without the patch,
the task's STATE field shows it as the "(PANIC)" task.
(anderson@redhat.com)
returns a count of symbols with the same name. Export a new
is_symbol_text() function, which checks whether specified symbol
entry is a type 't' or 'T'.
(atomlin@redhat.com, anderson@redhat.com)
option is context-sensitive, similar to the the regular "files"
command when used without an argument, but replaces the FILE and
DENTRY columns with I_MAPPING and NRPAGES columns that reflect
each open file's inode.i_mapping address_space structure address,
and the address_space.nrpages count within it; this shows how
many of each open file's pages are currently in the system's
page cache. The "files -p <inode>" option takes the address
of an inode, and dumps all of its pages that are currently in the
system's page cache, borrowing the "kmem -p" page structure output.
(yangoliver@gmail.com)
backtraces of the active tasks start at the function that is saved
in each per-cpu ELF note. With the patch, the backtrace will start
at the "crash_kexec" function on the panicking cpu, and at the
"crash_save_cpu" function on the other active cpus. By doing so,
the backtrace will display the exception handling functions leading
to crash_kexec() or crash_save_cpu(), as well as the exception frame
register set as it was at the time of the fatal exception on the
panic cpu, or when the shutdown IPI was received on the other cpus.
(anderson@redhat.com)
"union", "task", "list" and "tree" commands. If a specified
structure member contains an embedded structure, the output may
be restricted to just the embedded structure by expressing the
.member argument as "member.member". If a specified structure
member is an array, the output may be restricted to a single array
element by expressing the .member argument as "member[index]".
Furthermore, these embedded member specifications may extend beyond
one level deep, for example, by expressing the member argument as
"member.member.member", or "member[index].member".
(Alexandr_Terekhov@epam.com, anderson@redhat.com)
and "vtop" commands. Without the patch, if "vm -p" references a
swapped-out page on Linux 4.0 and later kernels, the SWAP location
may indicate "(unknown swap location)", and will show an invalid
OFFSET value; on Linux 3.13 and later kernels, running "vtop" on a
user virtual address incorrectly translates the PTE contents of
swapped out pages by showing a PHYSICAL address and FLAGS translation
instead of the SWAP device and OFFSET. It is possible that there may
be PTE bit translation errors on other kernel versions; the patch
addresses the changes in ARM64 PTE bit definitions made in Linux
3.11, 3.13, and 4.0 kernels.
(anderson@redhat.com)
but it allows the user to specify the page struct members to be
displayed. The option takes a comma-separated list of one or
more page struct members, which will be displayed following the
page structure address. The "flags" member will always be expressed
in hexadecimal format, and the "_count" and "_mapcount" members will
always be expressed in decimal format. Otherwise, all other members
will be displayed in hexadecimal format unless the current output
radix is 10 and the member is a signed/unsigned integer. Members
that are data structures may be specified by the data structure's
member name, or expanded to specify a member of that data structure.
For example, "-m lru" refers to a list_head data structure, in which
case both the list_head.next and list_head.prev pointer values will
be displayed; if "-m lru.next" is specified, just the list_head.next
value will be displayed.
(atomlin@redhat.com, anderson@redhat.com)
(1) The MIPS general purpose registers in the elf_gregset_t
don't start at index 0 but at index 6.
(2) Adjust for the kernel's pt_regs structure changes between
kernel versions. For example, fields are inserted into the
middle based on build time options, and the amount of padding
at the head of the structure was changed relatively recently.
To handle this, split the structure definition into two parts
and get the offsets of these two parts dynamically.
(3) Do not display each parsed kernel symbol during initialization
when invoked with "crash -d8".
(4) Add support for loading raw MIPS ramdump dumpfiles.
(5) Add support for compressed kdump dumpfiles.
(rabinv@axis.com)
data of specified cpus. It can be used in conjunction with all runq
command options. The cpus must be specified in a comma- and/or
dash-separated list; for examples, "3", "1,8,9", "1-23", or "1,8-15".
(anderson@redhat.com)
a bundle of data that describes a structure member. The function
receives a pointer to a struct_member_data structure, in which the
caller has initialized the "structure" and "member" name pointers:
struct struct_member_data {
char *structure;
char *member;
long type;
long unsigned_type;
long length;
long offset;
long bitpos;
long bitsize;
};
A gdb "printm" command is crafted using those two fields, and the
output of the command is used to initialize the remaining six fields.
Adapted from Qiao Nuohan's "pstruct" extension module.
(anderson@redhat.com, qiaonuohan@cn.fujitsu.com)
string into a buffer allocated with GETBUF(). As is the case with
any buffer allocated with GETBUF(), it is only meant to exist during
the life-span of the current command. If it is not explicitly freed
via FREEBUF(), then it will be freed automatically prior to the next
command.
(anderson@redhat.com)
layout introduced in Linux 3.17. The vmalloc region end address, and
the vmemmap start and end addresses are now calculated at kernel
build time, because they depend upon the size of a struct page.
Accordingly, the crash utility needs to calculate those three address
values dynamically, after the embedded gdb module has initialized.
Without the patch, reads of page structures return invalid data due
to incorrect virtual-to-physical translations of memory in the
vmemmap range. This in turn causes commands that require page
structure contents to fail or show invalid data, such as "kmem -p",
"kmem -[sS]", and the "kmem -[fF]" options.
(anderson@redhat.com)
Similar to all other supported dumpfile types, it is invoked as:
$ crash vmlinux <vmname>.vmss
A "<vmname>.vmss" file created by the VMware vSphere ESX hypervisor
contains a header and the full memory image. A "<vmname>.vmss" file
created by the VMware Workstation facility only contains the header,
and must be accompanied by a companion "<vmname>.vmem" memory image
that is located in the same directory as the "<vmname>.vmss" file.
(hfu@vmware.com)
architecture. For compressed kdump and ELF kdump dumpfiles, the
elf_prstatus structure in each per-cpu NT_PRSTATUS note will be
translated.
(anderson@redhat.com)
The "help -r" option has been extended to dump the PPC64 registers
stored in each per-cpu NT_PRSTATUS note in compressed kdump and
ELF kdump dumpfiles.
(anderson@redhat.com)
initial support is restricted to 32-bit MIPS kernels that are
configured as little-endian. With respect to dumpfile types, only
ELF vmcores are recognized. In addition to building crash as a
32-bit MIPS binary, it is also possible to build crash as an x86
binary on an x86 or x86_64 host so that crash analysis of MIPS
dumpfiles can be performed on an x86 or x86_64 host. The x86 binary
can be built by entering "make target=MIPS" for the initial build;
subsequent builds with MIPS support can be accomplished by entering
"make" alone.
(rabin@rab.in)
adds support for displaying the new s390x vector registers. For
ELF dumps, the registers are taken from the VX ELF notes; for s390
dumps. the registers are taken from memory. The option produces the
same output as the -a option, but also displays the vector registers
for all active tasks.
(holzheu@linux.vnet.ibm.com)
architectures. For compressed kdumps, the elf_prstatus structure in
each per-cpu NT_PRSTATUS note will be translated. For ELF kdumps,
the elf_prstatus structure in each per-cpu NT_PRSTATUS note, and
the QEMUCPUState structure in each per-cpu QEMU note, will be
translated.
(zhouwj-fnst@cn.fujitsu.com, anderson@redhat.com)
kernels that have the following Linux 3.18 commit, which removes the
special per-cpu exception stack for handling stack segment faults:
commit 6f442be2fb22be02cafa606f1769fa1e6f894441
x86_64, traps: Stop using IST for #SS
Without this patch, backtraces that originate on any of the other 4
per-cpu exception stacks will be mis-labeled at the transition point
back to the previous stack. For example, backtraces that that
originate on the NMI stack will indicate that they are coming from
the "DOUBLEFAULT" stack. The patch examines all idt_table entries
during initialization, looking for gate descriptors that have
non-zero index values, and when found, pulls out out the handler
function address; from that information, the exception stack name
string array is properly initialized rather than being hard-coded.
This fix also properly labels the exception stack names on x86_64
CONFIG_PREEMPT_RT realtime kernels, which only utilize 3 exception
stacks instead of the traditional 5 (now 4 with this kernel commit),
instead of just showing "RT". Also, without the patch, the "mach"
command will mis-label the stack names when it displays the base
addresses of each per-cpu exception stack.
(anderson@redhat.com)
the mm_struct address pointer in its task_struct is NULL'd out, and
as a result, the "vm" command looks like this:
crash> vm
PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres"
MM PGD RSS TOTAL_VM
0 0 0k 0k
However, the mm_struct address can be retrieved from the task's
kernel stack and entered manually with this option, which allows the
"vm" command to attempt to dump the virtual memory data of the task.
It may, or may not, work, depending upon how far the virtual memory
deconstruction has proceeded. This option only verifies that the
address entered is from the "mm_struct" slab cache, and that
its mm_struct.mm_count is non-zero.
(qiaonuohan@cn.fujitsu.com, anderson@redhat.com)
information, which will be appended to the traditional output of
the command. For example:
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 1965332 7.5 GB ----
FREE 78080 305 MB 3% of TOTAL MEM
USED 1887252 7.2 GB 96% of TOTAL MEM
SHARED 789954 3 GB 40% of TOTAL MEM
BUFFERS 110606 432.1 MB 5% of TOTAL MEM
CACHED 1212645 4.6 GB 61% of TOTAL MEM
SLAB 146563 572.5 MB 7% of TOTAL MEM
TOTAL SWAP 1970175 7.5 GB ----
SWAP USED 5 20 KB 0% of TOTAL SWAP
SWAP FREE 1970170 7.5 GB 99% of TOTAL SWAP
COMMIT LIMIT 2952841 11.3 GB ----
COMMITTED 1150595 4.4 GB 38% of TOTAL LIMIT
The COMMIT LIMIT and COMMITTED information is similar to that
displayed by the CommitLimit and Committed_AS lines in /proc/meminfo.
(atomlin@redhat.com)
of network devices with respect the network namespace of the current
context, or that of a task specified by the optional "pid" or "task"
argument. The former "net -n <address>" option that translates
an IPv4 address expressed as a decimal or hexadecimal value into a
standard numbers-and-dots notation has been changed to "net -N".
(ws@parallels.com)
the active tasks in x86_64 ELF or compressed dumpfiles created by the
KVM "virsh dump --memory-only" facility. Without the patch, the
backtraces of active tasks may show an invalid starting frame that
indicates "__schedule". The fix displays the exception RIP and dumps
the register contents that are stored in the dumpfile header. If the
active task was operating in the kernel, the backtrace continues from
there; if the task was operating in user-space, the backtrace is
complete at that point.
(anderson@redhat.com)
"virsh dump --memory-only --format <compression-type>" command,
where the compression-type is either "kdump-zlib", "kdump-lzo" or
"kdump-snappy". Without the patch, if an x86_64 guest kernel was loaded
with a non-zero "phys_base", the "--machdep phys_base=<offset>" command
line option was required as a workaround or the crash session would fail
with the warning message "WARNING: cannot read linux_banner string"
followed by the fatal error message "crash: vmlinux and <dumpfile name>
do not match!".
(anderson@redhat.com)
configured with CONFIG_SLAB:
commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
mm/slab: use percpu allocator for cpu cache
The commit above redesigned the kmem_cache.array_cache[] from a
hardwired array to a per-cpu pointer referencing external array_cache
structures. Without the patch, the crash session would fail during
initialization with the message "crash: cannot resolve cache_cache".
Note that it could be worked around by using the "--no_kmem_cache"
command line option, with a resulting loss of functionality for
commands requiring slab-related data.
(anderson@redhat.com)
the header of compressed kdumps, and the new DUMP_ELF_INCOMPLETE flag
in the header of ELF kdumps. If the makedumpfile(8) facility fails
to complete the creation of compressed or ELF kdump vmcore files
due to ENOSPC or other error, it will mark the vmcore as incomplete.
If either flag is set, the crash utility will issue a warning that
the dumpfile is known to be incomplete during initialization, just
prior to the system banner display. When reads are attempted on
missing data, a read error will be returned. As an alternative,
zero-filled data will be returned if the "--zero_excluded" command
line flag is used, or the "zero_excluded" runtime variable is set
to "on". In either case, the read errors or zero-filled memory
may cause the crash session to fail entirely, cause commands to
fail, or may result in other unpredictable runtime behavior.
(anderson@redhat.com, zhouwj-fnst@cn.fujitsu.com)
either "show" (the default) or "hide". When set to "hide", certain
command output associated with offline cpus will be hidden from view,
and the output will indicate that the cpu is "[OFFLINE]". The new
variable can be set during invocation on the crash command line via
the option "--offline [show|hide]". During runtime, or in a .crashrc
or other crash input file, the variable can be set by entering
"set offline [show|hide]". The commands or options that are affected
when the variable is set to "hide" are as follows:
o On X86_64 machines, the "bt -E" option will not search exception
stacks associated with offline cpus.
o On X86_64 machines, the "mach" command will append "[OFFLINE]"
to the addresses of IRQ and exception stacks associated with
offline cpus.
o On X86_64 machines, the "mach -c" command will not display the
cpuinfo_x86 data structure associated with offline cpus.
o The "help -r" option has been fixed so as to not attempt to
display register sets of offline cpus from ELF kdump vmcores,
compressed kdump vmcores, and ELF kdump clones created by
"virsh dump --memory-only".
o The "bt -c" option will not accept an offline cpu number.
o The "set -c" option will not accept an offline cpu number.
o The "irq -s" option will not display statistics associated with
offline cpus.
o The "timer" command will not display hrtimer data associated
with offline cpus.
o The "timer -r" option will not display hrtimer data associated
with offline cpus.
o The "ptov" command will append "[OFFLINE]" when translating a
per-cpu address offset to a virtal address of an offline cpu.
o The "kmem -o" option will append "[OFFLINE]" to the base per-cpu
virtual address of an offline cpu.
o The "kmem -S" option in CONFIG_SLUB kernels will not display
per-cpu data associated with offline cpus.
o When a per-cpu address reference is passed to the "struct"
command, the data structure will not be displayed for offline
cpus.
o When a per-cpu symbol and cpu reference is passed to the "p"
command, the data will not be displayed for offline cpus.
o When the "ps -[l|m]" option is passed the optional "-C [cpus]"
option, the tasks queued on offline cpus are not shown.
o The "runq" command and the "runq [-t/-m/-g/-d]" options will not
display runqueue data for offline cpus.
o The "ps" command will replace the ">" active task indicator to
a "-" for offline cpus.
The initial system information banner and the "sys" command will
display the total number of cpus as before, but will append the count
of offline cpus. Lastly, a fix has been made for the initialization
time determination of the maximum number of per-cpu objects queued
in a CONFIG_SLAB kmem_cache so as to continue checking all cpus
higher than the first offline cpu. These changes in behavior are not
dependent upon the setting of the crash "offline" variable.
(qiaonuohan@cn.fujitsu.com)
utility source tree on PPC and PPC64 machines. Without the patch,
both PPC and PPC64 will get #define'd if the extension module build
procedure does not #define one or the other, which in turn causes
multiple conflicting declarations.
(anderson@redhat.com)
a crash-7.0.4 patch which added per-thread task_struct.rss_stat page
counts to the task's mm_struct.rss_stat page counts in order to show
an accurate/synchronized RSS value. Without the patch, the "ps"
command performance would degrade as the number of tasks increased,
most notably when there were thousands of tasks.
(panfy.fnst@cn.fujitsu.com, anderson@redhat.com)
more "ramdump" files may be entered on the crash command line
in an ordered pair format consisting of the RAM dump filename
and the starting physical address expressed in hexadecimal,
connected with an ampersand:
$ crash vmlinux ramdump@address [ramdump@address]
A temporary ELF header will be created in /var/tmp, and the
combination of the header and the ramdump file(s) will be handled
like a normal ELF vmcore. The ELF header will only exist during
the crash session. If desired, an optional "-o <filename>"
may be entered to create a permanent ELF vmcore file from the
ramdump file(s).
(vinayakm.list@gmail.com, paawan1982@yahoo.com, anderson@redhat.com)
configured with CONFIG_SLUB to display the address of each per-cpu
kmem_cache_cpu address and the contents of its per-cpu partial list.
(qiaonuohan@cn.fujitsu.com)
kernel's VA_BITS value. It currently is hardwired in the kernel to
one of two values depending upon whether 4K or 64K pages are
configured. However, there are plans to support 16K paqes, to make
VA_BITS a configurable value, and to make the number of page-table
levels configurable. Towards that end, the crash utility has been
changed to determine the VA_BITS value based upon known kernel
virtual addresses, and to then calculate the relevant kernel virtual
address ranges on that value instead of hardwiring them based upon
the page size.
(anderson@redhat.com)
CONFIG_ARM_LPAE. The patch implements the virtual-to-physical
address translation of 64-bit PTEs used by ARM LPAE kernels.
(sdu.liu@huawei.com, weijitao@huawei.com)
the cgroup_name() function now utilizes kernfs_name(). Without the
patch, the command fails with the error message "runq: invalid
structure member offset: cgroup_dentry".
(anderson@redhat.com)
option to "ps -l", but which translates the task timestamp value from
a decimal or hexadecimal nanoseconds value into a more human-readable
string consisting of the number of days, hours, minutes, seconds and
milliseconds that have elapsed since the task started executing on a
cpu. More accurately described, it is the time difference between
the timestamp copied from the per-cpu runqueue clock when the task
last started executing compared to the most current value of the
per-cpu runqueue clock.
(anderson@redhat.com, bud.brown@redhat.com)
In addition, a new "ps -C <cpu-specifier>" option has been added
that can only be used with "ps -l" and "ps -m", which sorts the
global task list into per-cpu blocks; the cpu-specifier uses the
standard comma or dash separated list, expressed as "-C 1,3,5",
"-C 1-3", "-C 1,3,5-7,10", or "-Call" or "-Ca" for all cpus.
(anderson@redhat.com)
properly display exception frame register contents on NMI stacks.
Kernel commit 3f3c8b8c4b2a34776c3470142a7c8baafcda6eb0 added 12 more
values to the NMI exception stack to handle nested NMIs caused by
page faults or breakpoints that could occur while handling an NMI
exception. The fix has two parts:
1. Determine if this kernel has the nested NMI layout and set a
machine-specific flag (NESTED_NMI) if it does.
2. When backtracing an NMI stack, use the saved values instead of
those found at the top of stack.
Kernel commit 28696f434fef0efa97534b59986ad33b9c4df7f8 changed
the stack layout again, swapping the location of the "saved" and
"copied" registers. This can be detected automatically, because the
"copied" registers contain either a copy of the "saved" registers,
or point to "repeat_nmi". So, if "repeat_nmi" is found as the return
address, assume that this is the old layout, and adjust the stack
pointer again. Without the patch, incorrect register values are
displayed in the exception frame dump in the NMI stack backtrace.
(ptesarik@suse.cz)
cpus offline themselves during a kdump procedure. This causes an
invalid cpu count determination during crash session initialization
from an ARM vmcore. The patch utilizes the cpu count found in the
cpu_active_map if it is greater than the count in the cpu_online_map.
In addition, the maximum NR_CPUS value for the ARM architecture has
been raised from 4 to 32.
(sdu.liu@huawei.com)
can be now be readily identified because of new kernel symbols that
have been added. For those kernels, the new "--kaslr=<offset>"
and/or "--kaslr=auto" options are not necessary for ELF or compressed
kdump vmcores, or for live systems that have /proc/kallsyms showing
the relocated symbol values. A new KASLR initialization function
called kaslr_init() is now called by symtab_init() prior to the
initial symbol-sorting operation. If kaslr_init() determines that
KASLR may be in effect, it will trigger a search for the relevant
vmlinux symbols during the sorting operation, which in turn will
cause the relocation value to be automatically calculated.
(anderson@redhat.com)
functions from netdump.c and kdump.c via a new read_vmcoreinfo()
method in the global program_context structure. The function
get_log_from_vmcoreinfo() will access vmcoreinfo data via the
new pointer instead of requiring its callers to pass pointers to
their dumpfile-specific function.
(anderson@redhat.com)
for the Linux 3.13 increase of the ARM64 virtual address space size
from 39 to 42 bits when 64K pages are configured. Without the patch,
the warning message "WARNING: cannot access vmalloc'd module memory"
is displayed during session initialization.
(anderson@redhat.com)
several command options still under development. In-kernel exception
frames are only dumped if the exception handler function is contained
within the symbol boundaries from "__exception_text_start" to
"__exception_text_end"; when ARM64 kdump is eventually implemented,
further exception-related work will be resumed.
(anderson@redhat.com)
kernels that expand to a 42-bit address space when 64K pages are
configured. This is also the first crash version that has been
tested on a live ARM64 system with 4K pages, where it cleanly
make it to the "crash>" prompt. However, it should be noted that
some commands (most notably "bt") still do not work as of yet.
(anderson@redhat.com)
is necessary to immediately perform a function for each entry in a
list while the list is being traversed. A callback function, and an
option callback data pointer, can be registered in the list_data
structure. The address of each entry in the list along with the
optional callback data pointer will be passed to the callback
function. If desired, the callback function may also dictate that
do_list() should stop the list traversal and return immediately to
its caller.
(anderson@redhat.com)
The hash queue is used for gathering and verifying lists, and the
original count of 128 may be overwhelmed if a list is extremely
large. For example, on a 256GB system with 192GB of free pages,
the "kmem -f" command takes hours to complete; with this patch,
the time is reduced to a few minutes. In addition, a new command
line option "--hash <count>" has been added to allow a user to
override the default hash queue head count of 32768.
(anderson@redhat.com)
that that are configured with CONFIG_RANDOMIZE_BASE. When set to
"auto", the KASLR relocation value will be determined automatically
by comparing the "_stext" symbol value compiled into the vmlinux file
with the _stext symbol value stored in kdump vmcoreinfo data; on live
systems the comparison will be made with the "_stext" symbol value
that is found in /proc/kallsyms.
(ahonig@google.com, anderson@redhat.com)
X86_64 kernels that are configured with CONFIG_RANDOMIZE_BASE.
The offset value must be equal to the difference between the
symbol values compiled into the vmlinux file and their relocated
value.
(ahonig@google.com, anderson@redhat.com)
the function frame that called into a function that was interrupted.
Without the patch, the first frame just above an IRQ exception frame
register dump may show an invalid/stale function.
(anderson@redhat.com)
a slab object, consisting of the slab cache name and the address
value, separated by a colon, and encompassed in brackets:
[slab-cache-name:address]
Enhanced the "bt -F" option such that if "-F" is entered twice,
and if the stack frame contents reference a slab cache object, both
the slab cache name and the stack contents will be displayed within
brackets.
Enhanced the "rd -S" option such that if "-S" is entered twice,
and if the memory contents reference a slab cache object, both the
slab cache name and the memory contents will be displayed within
brackets.
(anderson@redhat.com)
