"virsh dump --memory-only --format <compression-type>" command,
where the compression-type is either "kdump-zlib", "kdump-lzo" or
"kdump-snappy". Without the patch, if an x86_64 guest kernel was loaded
with a non-zero "phys_base", the "--machdep phys_base=<offset>" command
line option was required as a workaround or the crash session would fail
with the warning message "WARNING: cannot read linux_banner string"
followed by the fatal error message "crash: vmlinux and <dumpfile name>
do not match!".
(anderson@redhat.com)
configured with CONFIG_SLAB:
commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
mm/slab: use percpu allocator for cpu cache
The commit above redesigned the kmem_cache.array_cache[] from a
hardwired array to a per-cpu pointer referencing external array_cache
structures. Without the patch, the crash session would fail during
initialization with the message "crash: cannot resolve cache_cache".
Note that it could be worked around by using the "--no_kmem_cache"
command line option, with a resulting loss of functionality for
commands requiring slab-related data.
(anderson@redhat.com)
the header of compressed kdumps, and the new DUMP_ELF_INCOMPLETE flag
in the header of ELF kdumps. If the makedumpfile(8) facility fails
to complete the creation of compressed or ELF kdump vmcore files
due to ENOSPC or other error, it will mark the vmcore as incomplete.
If either flag is set, the crash utility will issue a warning that
the dumpfile is known to be incomplete during initialization, just
prior to the system banner display. When reads are attempted on
missing data, a read error will be returned. As an alternative,
zero-filled data will be returned if the "--zero_excluded" command
line flag is used, or the "zero_excluded" runtime variable is set
to "on". In either case, the read errors or zero-filled memory
may cause the crash session to fail entirely, cause commands to
fail, or may result in other unpredictable runtime behavior.
(anderson@redhat.com, zhouwj-fnst@cn.fujitsu.com)
either "show" (the default) or "hide". When set to "hide", certain
command output associated with offline cpus will be hidden from view,
and the output will indicate that the cpu is "[OFFLINE]". The new
variable can be set during invocation on the crash command line via
the option "--offline [show|hide]". During runtime, or in a .crashrc
or other crash input file, the variable can be set by entering
"set offline [show|hide]". The commands or options that are affected
when the variable is set to "hide" are as follows:
o On X86_64 machines, the "bt -E" option will not search exception
stacks associated with offline cpus.
o On X86_64 machines, the "mach" command will append "[OFFLINE]"
to the addresses of IRQ and exception stacks associated with
offline cpus.
o On X86_64 machines, the "mach -c" command will not display the
cpuinfo_x86 data structure associated with offline cpus.
o The "help -r" option has been fixed so as to not attempt to
display register sets of offline cpus from ELF kdump vmcores,
compressed kdump vmcores, and ELF kdump clones created by
"virsh dump --memory-only".
o The "bt -c" option will not accept an offline cpu number.
o The "set -c" option will not accept an offline cpu number.
o The "irq -s" option will not display statistics associated with
offline cpus.
o The "timer" command will not display hrtimer data associated
with offline cpus.
o The "timer -r" option will not display hrtimer data associated
with offline cpus.
o The "ptov" command will append "[OFFLINE]" when translating a
per-cpu address offset to a virtal address of an offline cpu.
o The "kmem -o" option will append "[OFFLINE]" to the base per-cpu
virtual address of an offline cpu.
o The "kmem -S" option in CONFIG_SLUB kernels will not display
per-cpu data associated with offline cpus.
o When a per-cpu address reference is passed to the "struct"
command, the data structure will not be displayed for offline
cpus.
o When a per-cpu symbol and cpu reference is passed to the "p"
command, the data will not be displayed for offline cpus.
o When the "ps -[l|m]" option is passed the optional "-C [cpus]"
option, the tasks queued on offline cpus are not shown.
o The "runq" command and the "runq [-t/-m/-g/-d]" options will not
display runqueue data for offline cpus.
o The "ps" command will replace the ">" active task indicator to
a "-" for offline cpus.
The initial system information banner and the "sys" command will
display the total number of cpus as before, but will append the count
of offline cpus. Lastly, a fix has been made for the initialization
time determination of the maximum number of per-cpu objects queued
in a CONFIG_SLAB kmem_cache so as to continue checking all cpus
higher than the first offline cpu. These changes in behavior are not
dependent upon the setting of the crash "offline" variable.
(qiaonuohan@cn.fujitsu.com)
utility source tree on PPC and PPC64 machines. Without the patch,
both PPC and PPC64 will get #define'd if the extension module build
procedure does not #define one or the other, which in turn causes
multiple conflicting declarations.
(anderson@redhat.com)
a crash-7.0.4 patch which added per-thread task_struct.rss_stat page
counts to the task's mm_struct.rss_stat page counts in order to show
an accurate/synchronized RSS value. Without the patch, the "ps"
command performance would degrade as the number of tasks increased,
most notably when there were thousands of tasks.
(panfy.fnst@cn.fujitsu.com, anderson@redhat.com)
more "ramdump" files may be entered on the crash command line
in an ordered pair format consisting of the RAM dump filename
and the starting physical address expressed in hexadecimal,
connected with an ampersand:
$ crash vmlinux ramdump@address [ramdump@address]
A temporary ELF header will be created in /var/tmp, and the
combination of the header and the ramdump file(s) will be handled
like a normal ELF vmcore. The ELF header will only exist during
the crash session. If desired, an optional "-o <filename>"
may be entered to create a permanent ELF vmcore file from the
ramdump file(s).
(vinayakm.list@gmail.com, paawan1982@yahoo.com, anderson@redhat.com)
configured with CONFIG_SLUB to display the address of each per-cpu
kmem_cache_cpu address and the contents of its per-cpu partial list.
(qiaonuohan@cn.fujitsu.com)
kernel's VA_BITS value. It currently is hardwired in the kernel to
one of two values depending upon whether 4K or 64K pages are
configured. However, there are plans to support 16K paqes, to make
VA_BITS a configurable value, and to make the number of page-table
levels configurable. Towards that end, the crash utility has been
changed to determine the VA_BITS value based upon known kernel
virtual addresses, and to then calculate the relevant kernel virtual
address ranges on that value instead of hardwiring them based upon
the page size.
(anderson@redhat.com)
CONFIG_ARM_LPAE. The patch implements the virtual-to-physical
address translation of 64-bit PTEs used by ARM LPAE kernels.
(sdu.liu@huawei.com, weijitao@huawei.com)
the cgroup_name() function now utilizes kernfs_name(). Without the
patch, the command fails with the error message "runq: invalid
structure member offset: cgroup_dentry".
(anderson@redhat.com)
option to "ps -l", but which translates the task timestamp value from
a decimal or hexadecimal nanoseconds value into a more human-readable
string consisting of the number of days, hours, minutes, seconds and
milliseconds that have elapsed since the task started executing on a
cpu. More accurately described, it is the time difference between
the timestamp copied from the per-cpu runqueue clock when the task
last started executing compared to the most current value of the
per-cpu runqueue clock.
(anderson@redhat.com, bud.brown@redhat.com)
In addition, a new "ps -C <cpu-specifier>" option has been added
that can only be used with "ps -l" and "ps -m", which sorts the
global task list into per-cpu blocks; the cpu-specifier uses the
standard comma or dash separated list, expressed as "-C 1,3,5",
"-C 1-3", "-C 1,3,5-7,10", or "-Call" or "-Ca" for all cpus.
(anderson@redhat.com)
properly display exception frame register contents on NMI stacks.
Kernel commit 3f3c8b8c4b2a34776c3470142a7c8baafcda6eb0 added 12 more
values to the NMI exception stack to handle nested NMIs caused by
page faults or breakpoints that could occur while handling an NMI
exception. The fix has two parts:
1. Determine if this kernel has the nested NMI layout and set a
machine-specific flag (NESTED_NMI) if it does.
2. When backtracing an NMI stack, use the saved values instead of
those found at the top of stack.
Kernel commit 28696f434fef0efa97534b59986ad33b9c4df7f8 changed
the stack layout again, swapping the location of the "saved" and
"copied" registers. This can be detected automatically, because the
"copied" registers contain either a copy of the "saved" registers,
or point to "repeat_nmi". So, if "repeat_nmi" is found as the return
address, assume that this is the old layout, and adjust the stack
pointer again. Without the patch, incorrect register values are
displayed in the exception frame dump in the NMI stack backtrace.
(ptesarik@suse.cz)
cpus offline themselves during a kdump procedure. This causes an
invalid cpu count determination during crash session initialization
from an ARM vmcore. The patch utilizes the cpu count found in the
cpu_active_map if it is greater than the count in the cpu_online_map.
In addition, the maximum NR_CPUS value for the ARM architecture has
been raised from 4 to 32.
(sdu.liu@huawei.com)
can be now be readily identified because of new kernel symbols that
have been added. For those kernels, the new "--kaslr=<offset>"
and/or "--kaslr=auto" options are not necessary for ELF or compressed
kdump vmcores, or for live systems that have /proc/kallsyms showing
the relocated symbol values. A new KASLR initialization function
called kaslr_init() is now called by symtab_init() prior to the
initial symbol-sorting operation. If kaslr_init() determines that
KASLR may be in effect, it will trigger a search for the relevant
vmlinux symbols during the sorting operation, which in turn will
cause the relocation value to be automatically calculated.
(anderson@redhat.com)
functions from netdump.c and kdump.c via a new read_vmcoreinfo()
method in the global program_context structure. The function
get_log_from_vmcoreinfo() will access vmcoreinfo data via the
new pointer instead of requiring its callers to pass pointers to
their dumpfile-specific function.
(anderson@redhat.com)
for the Linux 3.13 increase of the ARM64 virtual address space size
from 39 to 42 bits when 64K pages are configured. Without the patch,
the warning message "WARNING: cannot access vmalloc'd module memory"
is displayed during session initialization.
(anderson@redhat.com)
several command options still under development. In-kernel exception
frames are only dumped if the exception handler function is contained
within the symbol boundaries from "__exception_text_start" to
"__exception_text_end"; when ARM64 kdump is eventually implemented,
further exception-related work will be resumed.
(anderson@redhat.com)
kernels that expand to a 42-bit address space when 64K pages are
configured. This is also the first crash version that has been
tested on a live ARM64 system with 4K pages, where it cleanly
make it to the "crash>" prompt. However, it should be noted that
some commands (most notably "bt") still do not work as of yet.
(anderson@redhat.com)
is necessary to immediately perform a function for each entry in a
list while the list is being traversed. A callback function, and an
option callback data pointer, can be registered in the list_data
structure. The address of each entry in the list along with the
optional callback data pointer will be passed to the callback
function. If desired, the callback function may also dictate that
do_list() should stop the list traversal and return immediately to
its caller.
(anderson@redhat.com)
The hash queue is used for gathering and verifying lists, and the
original count of 128 may be overwhelmed if a list is extremely
large. For example, on a 256GB system with 192GB of free pages,
the "kmem -f" command takes hours to complete; with this patch,
the time is reduced to a few minutes. In addition, a new command
line option "--hash <count>" has been added to allow a user to
override the default hash queue head count of 32768.
(anderson@redhat.com)
that that are configured with CONFIG_RANDOMIZE_BASE. When set to
"auto", the KASLR relocation value will be determined automatically
by comparing the "_stext" symbol value compiled into the vmlinux file
with the _stext symbol value stored in kdump vmcoreinfo data; on live
systems the comparison will be made with the "_stext" symbol value
that is found in /proc/kallsyms.
(ahonig@google.com, anderson@redhat.com)
X86_64 kernels that are configured with CONFIG_RANDOMIZE_BASE.
The offset value must be equal to the difference between the
symbol values compiled into the vmlinux file and their relocated
value.
(ahonig@google.com, anderson@redhat.com)
the function frame that called into a function that was interrupted.
Without the patch, the first frame just above an IRQ exception frame
register dump may show an invalid/stale function.
(anderson@redhat.com)
a slab object, consisting of the slab cache name and the address
value, separated by a colon, and encompassed in brackets:
[slab-cache-name:address]
Enhanced the "bt -F" option such that if "-F" is entered twice,
and if the stack frame contents reference a slab cache object, both
the slab cache name and the stack contents will be displayed within
brackets.
Enhanced the "rd -S" option such that if "-S" is entered twice,
and if the memory contents reference a slab cache object, both the
slab cache name and the memory contents will be displayed within
brackets.
(anderson@redhat.com)
