running live against /proc/kcore. Without the patch, the message
"WARNING: cannot access vmalloc'd module memory" may be displayed
during session initialization, and vmalloc/module memory will be
unaccessible. It should be noted that at the time of this patch,
the upstream (4.16.0) version of /proc/kcore does not work correctly
for ARM64, because PT_LOAD segments for unity-mapped blocks of
physical memory are not generated.
(anderson@redhat.com)
calculating phys_base and the mapped kernel offset for KASLR-enabled
kernels on SADUMP dumpfiles by using a technique developed by Takao
Indoh. Originally, the patchset included support for kdumps, but this
was dropped in v2, as it was deemed unnecessary due to the upstream
implementation of the "vmcoreinfo device" in QEMU. However, there
are still several reasons for which the vmcoreinfo device may not be
present at the time when a memory dump is taken from a VM, ranging
from a host running older QEMU/libvirt versions, to misconfigured VMs
or environments running Hypervisors that doesn't support this device.
This patchset generalizes the KASLR-related functions from sadump.c
and moves them to kaslr_helper.c, and makes kdump analysis fall back
to KASLR offset calculation if vmcoreinfo data is missing.
(slp@redhat.com)
"kimage_voffset" value in the ELF header. Without the patch, it is
necessary to use the "--machdep kvimage_offset=<value>" command line
option, or the session fails with the message "crash: vmlinux and
vmcore do not match!".
(anderson@redhat.com)
offset value in the dumpfile header for kernels that are compiled
with CONFIG_RANDOMIZE_BASE. Without the patch, it is necessary to
use the "--kaslr=<offset>" command line option, or the session
fails with the message "WARNING: cannot read linux_banner string",
followed by "crash: vmlinux and vmcore do not match!".
(anderson@redhat.com)
and later x86_64 kernels configured with CONFIG_RANDOMIZE_BASE, which
randomizes the unity-mapping PAGE_OFFSET value. Without the patch,
the crash session fails during session initialization with the error
message "crash: seek error: kernel virtual address: <address>
type: page_offset_base".
(anderson@redhat.com)
dumpfiles:
(1) If ELF notes are not available, read them from the kernel's
crash_notes.
(2) If an online CPUs did not save its ELF notes, then adjust
the mapping of each ELF note to its CPU accordingly.
(rabinv@axis.com)
--memory-only" facility if the guest kernel was not configured with
CONFIG_KEXEC, or CONFIG_KEXEC_CORE in Linux 4.3 and later kernels.
Without the patch, the crash session fails during initialization with
the message "crash: cannot resolve kexec_crash_image".
(hirofumi@mail.parknet.co.jp)
to a block device. When running the crash utility against such a
block device, a misleading warning message such as this is displayed:
WARNING: /dev/sda1: may be truncated or incomplete
PT_LOAD p_offset: 16384
p_filesz: 5497558138880
bytes required: 5497558155264
dumpfile size: 0
With the patch, the warning message above will be replaced by a note
using this format:
NOTE: /dev/sda1: No dump complete check for block devices
(holzheu@linux.vnet.ibm.com)
and compressed kdump clones. The patch utilizes the NT_PRSTATUS
notes in the dumpfile headers instead of reading them from the
kernel's "crash_notes", which are not initialized when QEMU generates
a dumpfile. Without the patch, these warning messages are displayed
during session initialization:
WARNING: invalid note (n_type != NT_PRSTATUS)
WARNING: cannot retrieve registers for active tasks
and running "bt" on an active task causes a segmentation violation.
(drjones@redhat.com)
Since makedumpfile commit 349a0ed1, it is now possible to save Xen
dumps in compressed kdump format. This patch set adds support for
these files. Two new files, xen_dom0.c and xen_dom0.h, have been
added to provide the common functionality required by both ELF and
compressed kdump formats.
(ptesarik@suse.cz)
the "snap.so" extension module as "(ACTIVE)" in the STATE field of
the initial system banner and the "set" command. Without the patch,
the task's STATE field shows it as the "(PANIC)" task.
(anderson@redhat.com)
(1) The MIPS general purpose registers in the elf_gregset_t
don't start at index 0 but at index 6.
(2) Adjust for the kernel's pt_regs structure changes between
kernel versions. For example, fields are inserted into the
middle based on build time options, and the amount of padding
at the head of the structure was changed relatively recently.
To handle this, split the structure definition into two parts
and get the offsets of these two parts dynamically.
(3) Do not display each parsed kernel symbol during initialization
when invoked with "crash -d8".
(4) Add support for loading raw MIPS ramdump dumpfiles.
(5) Add support for compressed kdump dumpfiles.
(rabinv@axis.com)
of their respective memory segments are not laid out sequentially
from low to high in the dumpfile. This has only been seen in ELF
vmcore files created by the VMware vmss2core facility. Without the
patch, the crash session may fail during initialization, either with
the message "cannot malloc ELF header buffer", or "crash: <dumpfile>:
not a supported file format".
(anderson@redhat.com)
architecture. For compressed kdump and ELF kdump dumpfiles, the
elf_prstatus structure in each per-cpu NT_PRSTATUS note will be
translated.
(anderson@redhat.com)
The "help -r" option has been extended to dump the PPC64 registers
stored in each per-cpu NT_PRSTATUS note in compressed kdump and
ELF kdump dumpfiles.
(anderson@redhat.com)
initial support is restricted to 32-bit MIPS kernels that are
configured as little-endian. With respect to dumpfile types, only
ELF vmcores are recognized. In addition to building crash as a
32-bit MIPS binary, it is also possible to build crash as an x86
binary on an x86 or x86_64 host so that crash analysis of MIPS
dumpfiles can be performed on an x86 or x86_64 host. The x86 binary
can be built by entering "make target=MIPS" for the initial build;
subsequent builds with MIPS support can be accomplished by entering
"make" alone.
(rabin@rab.in)
architectures. For compressed kdumps generated by "virsh dump
--memory-only", the QEMUCPUState structure in each per-cpu QEMU
note will be translated, and the dumpfile offset address of each
QEMU note will be displayed.
(qiaonuohan@cn.fujitsu.com, anderson@redhat.com)
64-bit ELF vmcore format generated by "virsh dump --memory-only".
Without the patch, the QEMUCPUState structures in QEMU notes are not
translated.
(qiaonuohan@cn.fujitsu.com)
adds support for displaying the new s390x vector registers. For
ELF dumps, the registers are taken from the VX ELF notes; for s390
dumps. the registers are taken from memory. The option produces the
same output as the -a option, but also displays the vector registers
for all active tasks.
(holzheu@linux.vnet.ibm.com)
architectures. For compressed kdumps, the elf_prstatus structure in
each per-cpu NT_PRSTATUS note will be translated. For ELF kdumps,
the elf_prstatus structure in each per-cpu NT_PRSTATUS note, and
the QEMUCPUState structure in each per-cpu QEMU note, will be
translated.
(zhouwj-fnst@cn.fujitsu.com, anderson@redhat.com)
the active tasks in x86_64 ELF or compressed dumpfiles created by the
KVM "virsh dump --memory-only" facility. Without the patch, the
backtraces of active tasks may show an invalid starting frame that
indicates "__schedule". The fix displays the exception RIP and dumps
the register contents that are stored in the dumpfile header. If the
active task was operating in the kernel, the backtrace continues from
there; if the task was operating in user-space, the backtrace is
complete at that point.
(anderson@redhat.com)
"virsh dump --memory-only --format <compression-type>" command,
where the compression-type is either "kdump-zlib", "kdump-lzo" or
"kdump-snappy". Without the patch, if an x86_64 guest kernel was loaded
with a non-zero "phys_base", the "--machdep phys_base=<offset>" command
line option was required as a workaround or the crash session would fail
with the warning message "WARNING: cannot read linux_banner string"
followed by the fatal error message "crash: vmlinux and <dumpfile name>
do not match!".
(anderson@redhat.com)
the header of compressed kdumps, and the new DUMP_ELF_INCOMPLETE flag
in the header of ELF kdumps. If the makedumpfile(8) facility fails
to complete the creation of compressed or ELF kdump vmcore files
due to ENOSPC or other error, it will mark the vmcore as incomplete.
If either flag is set, the crash utility will issue a warning that
the dumpfile is known to be incomplete during initialization, just
prior to the system banner display. When reads are attempted on
missing data, a read error will be returned. As an alternative,
zero-filled data will be returned if the "--zero_excluded" command
line flag is used, or the "zero_excluded" runtime variable is set
to "on". In either case, the read errors or zero-filled memory
may cause the crash session to fail entirely, cause commands to
fail, or may result in other unpredictable runtime behavior.
(anderson@redhat.com, zhouwj-fnst@cn.fujitsu.com)
either "show" (the default) or "hide". When set to "hide", certain
command output associated with offline cpus will be hidden from view,
and the output will indicate that the cpu is "[OFFLINE]". The new
variable can be set during invocation on the crash command line via
the option "--offline [show|hide]". During runtime, or in a .crashrc
or other crash input file, the variable can be set by entering
"set offline [show|hide]". The commands or options that are affected
when the variable is set to "hide" are as follows:
o On X86_64 machines, the "bt -E" option will not search exception
stacks associated with offline cpus.
o On X86_64 machines, the "mach" command will append "[OFFLINE]"
to the addresses of IRQ and exception stacks associated with
offline cpus.
o On X86_64 machines, the "mach -c" command will not display the
cpuinfo_x86 data structure associated with offline cpus.
o The "help -r" option has been fixed so as to not attempt to
display register sets of offline cpus from ELF kdump vmcores,
compressed kdump vmcores, and ELF kdump clones created by
"virsh dump --memory-only".
o The "bt -c" option will not accept an offline cpu number.
o The "set -c" option will not accept an offline cpu number.
o The "irq -s" option will not display statistics associated with
offline cpus.
o The "timer" command will not display hrtimer data associated
with offline cpus.
o The "timer -r" option will not display hrtimer data associated
with offline cpus.
o The "ptov" command will append "[OFFLINE]" when translating a
per-cpu address offset to a virtal address of an offline cpu.
o The "kmem -o" option will append "[OFFLINE]" to the base per-cpu
virtual address of an offline cpu.
o The "kmem -S" option in CONFIG_SLUB kernels will not display
per-cpu data associated with offline cpus.
o When a per-cpu address reference is passed to the "struct"
command, the data structure will not be displayed for offline
cpus.
o When a per-cpu symbol and cpu reference is passed to the "p"
command, the data will not be displayed for offline cpus.
o When the "ps -[l|m]" option is passed the optional "-C [cpus]"
option, the tasks queued on offline cpus are not shown.
o The "runq" command and the "runq [-t/-m/-g/-d]" options will not
display runqueue data for offline cpus.
o The "ps" command will replace the ">" active task indicator to
a "-" for offline cpus.
The initial system information banner and the "sys" command will
display the total number of cpus as before, but will append the count
of offline cpus. Lastly, a fix has been made for the initialization
time determination of the maximum number of per-cpu objects queued
in a CONFIG_SLAB kmem_cache so as to continue checking all cpus
higher than the first offline cpu. These changes in behavior are not
dependent upon the setting of the crash "offline" variable.
(qiaonuohan@cn.fujitsu.com)
EM_ARM and EM_AARCH values as "e_machine" types, and ELFOSABI_LINUX
as an "e_ident[EI_OSABI]" type. Without the patch, the e_machine
translation would show "40 (unsupported)" for 32-bit ARM, or
"183 (unsupported)" on ARM64; and the ELFOSABI_LINUX type would
be translated as "3 (?)".
(anderson@redhat.com)
more "ramdump" files may be entered on the crash command line
in an ordered pair format consisting of the RAM dump filename
and the starting physical address expressed in hexadecimal,
connected with an ampersand:
$ crash vmlinux ramdump@address [ramdump@address]
A temporary ELF header will be created in /var/tmp, and the
combination of the header and the ramdump file(s) will be handled
like a normal ELF vmcore. The ELF header will only exist during
the crash session. If desired, an optional "-o <filename>"
may be entered to create a permanent ELF vmcore file from the
ramdump file(s).
(vinayakm.list@gmail.com, paawan1982@yahoo.com, anderson@redhat.com)
cpus offline themselves during a kdump procedure. This causes an
invalid cpu count determination during crash session initialization
from an ARM vmcore. The patch utilizes the cpu count found in the
cpu_active_map if it is greater than the count in the cpu_online_map.
In addition, the maximum NR_CPUS value for the ARM architecture has
been raised from 4 to 32.
(sdu.liu@huawei.com)
can be now be readily identified because of new kernel symbols that
have been added. For those kernels, the new "--kaslr=<offset>"
and/or "--kaslr=auto" options are not necessary for ELF or compressed
kdump vmcores, or for live systems that have /proc/kallsyms showing
the relocated symbol values. A new KASLR initialization function
called kaslr_init() is now called by symtab_init() prior to the
initial symbol-sorting operation. If kaslr_init() determines that
KASLR may be in effect, it will trigger a search for the relevant
vmlinux symbols during the sorting operation, which in turn will
cause the relocation value to be automatically calculated.
(anderson@redhat.com)
functions from netdump.c and kdump.c via a new read_vmcoreinfo()
method in the global program_context structure. The function
get_log_from_vmcoreinfo() will access vmcoreinfo data via the
new pointer instead of requiring its callers to pass pointers to
their dumpfile-specific function.
(anderson@redhat.com)
session fails with the warning message "WARNING: cannot read
linux_banner string" followed by the fatal error message "crash:
vmlinux and <dumpfile name> do not match!". At this point in
time, the kernel requires a patch to the ARM64 kern_addr_valid()
function to properly allow memory to be read from the kernel logical
memory map region.
(anderson@redhat.com)
that that are configured with CONFIG_RANDOMIZE_BASE. When set to
"auto", the KASLR relocation value will be determined automatically
by comparing the "_stext" symbol value compiled into the vmlinux file
with the _stext symbol value stored in kdump vmcoreinfo data; on live
systems the comparison will be made with the "_stext" symbol value
that is found in /proc/kallsyms.
(ahonig@google.com, anderson@redhat.com)
