are specified by the QEMU mem-path argument of a memory-backend-file
object. This allows the running of a live crash session against a
QEMU guest from the host machine. In this example, the /tmp/MEM file
on a QEMU host represents the guest's physical memory:
$ qemu-kvm ...other-options... \
-object memory-backend-file,id=MEM,size=128m,mem-path=/tmp/MEM,share=on \
-numa node,memdev=MEM -m 128
and a live session run can be run against the guest kernel like so:
$ crash <path-to-guest-vmlinux> live:/tmp/MEM@0
By prepending the ramdump image name with "live:", the crash session will
act as if it were running a normal live session.
(oleg@redhat.com)
version supports running against a live kernel. Compressed kdump
support is also here, but the crash dump support for the kernel,
kexec-tools, and makedumpfile is still pending. Initial work was
done by Karl Volz with help from Bob Picco.
(dave.kleikamp@oracle.com)
commit ccbf62d8a284cf181ac28c8e8407dd077d90dd4b, which changed the
task_struct.start_time member from a struct timespec to a u64.
Without the patch, the "RUN TIME" value is nonsensical.
(anderson@redhat.com)
the kernel has dynamically downsized from the size indicated by the
debuginfo data. At this time, only "kmem_cache" and "task_struct"
structures that have been downsized are registered, but others may be
added in the future. If a downsized data structure is passed to gdb
for display, gdb will request a read of the "full" data structure,
which may flow into a memory region that was either filtered by
makedumpfile(8), or perhaps into non-existent memory, thereby killing
the generating command immediately due to a partial read. With this
patch, commands such as "struct" and "task" that reference downsized
data structures will have their reads flagged to return successfully
if partial read error occurs.
(anderson@redhat.com)
Linux 4.2 and later kernels, which contain these commits:
commit 5aaeb5c01c5b6c0be7b7aadbf3ace9f3a4458c3d
x86/fpu, sched: Introduce CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT and
use it on x86
commit 0c8c0f03e3a292e031596484275c14cf39c0ab7a
x86/fpu, sched: Dynamically allocate 'struct fpu'
Without the patch, when running on a filtered kdump dumpfile, it is
possible that error messages like this will be seen when gathering
the tasks running on a system: "crash: page excluded: kernel virtual
address: <task_struct address> type: "fill_task_struct".
(ats-kumagai@wm.jp.nec.com)
"live" s390x dumpfiles created by the VMDUMP, stand-alone dump, or
"virsh dump" facilities, none of which explicitly mark the dumpfile
as a "live dump", run a standard "bt" backtrace on each kernel stack
instead of the text-address-only "bt -t". Without the patch, an
invalid text reference may be found in a task's kernel stack due to
the common zero-based user and kernel virtual address space ranges of
the s390x, causing the task to be mistakenly set as the "PANIC" task.
(holzheu@linux.vnet.ibm.com)
Do not search for a panic task in s390x dumpfiles that are marked
as a "live dump"...
The first part prevented a search of the active tasks; this part
prevents the last-ditch search of all tasks.
(anderson@redhat.com)
the "snap.so" extension module as "(ACTIVE)" in the STATE field of
the initial system banner and the "set" command. Without the patch,
the task's STATE field shows it as the "(PANIC)" task.
(anderson@redhat.com)
a "live dump". Without the patch, an exhaustive, unnecessary, search
of all kernel stacks that looks for evidence of a system crash may
find an invalid reference in a task's kernel stack due to the common
zero-based user and kernel virtual address space ranges of the s390x,
causing the task to be mistakenly set as the "PANIC" task.
(holzheu@linux.vnet.ibm.com, anderson@redhat.com)
Without the patch, if the target kernel was not configured with both
CONFIG_FAIR_GROUP_SCHED and CONFIG_RT_GROUP_SCHED, the command fails
with the message "runq: -g option not supported or applicable on this
architecture or kernel". With this patch, if the kernel was built
with either CONFIG_FAIR_GROUP_SCHED or CONFIG_RT_GROUP_SCHED, the
command will execute.
(rabinv@axis.com)
option is context-sensitive, similar to the the regular "files"
command when used without an argument, but replaces the FILE and
DENTRY columns with I_MAPPING and NRPAGES columns that reflect
each open file's inode.i_mapping address_space structure address,
and the address_space.nrpages count within it; this shows how
many of each open file's pages are currently in the system's
page cache. The "files -p <inode>" option takes the address
of an inode, and dumps all of its pages that are currently in the
system's page cache, borrowing the "kmem -p" page structure output.
(yangoliver@gmail.com)
state. Without the patch:
(1) The "ps" command's ST column shows "??" for tasks in the
TASK_WAKING state.
(2) The "ps" command's ST column shows "??" for tasks in the
TASK_PARKED state in Linux 3.14 and later kernels.
(3) The STATE field of the initial system banner and the "set"
command are incorrect if the task state has the TASK_WAKING,
TASK_WAKEKILL modifier, or TASK_PARKED bits set in Linux 3.14
and later kernels.
(4) The "foreach DE" task identifier fails if a task with a PID
number of 0xDE (222) exists.
(5) The "foreach" command's "SW", "PA", "TR" and "DE" task
identifiers inadvertently select all tasks in kernel versions
that do not have those states.
(6) The "help -t" output would display incorrect values for the
TASK_WAKEKILL, TASK_WAKING and TASK_PARKED states in Linux 3.14
and later kernels.
Lastly, support for the TASK_NOLOAD modifier introduced in Linux 4.2
has been added to STATE field of the "set" command and the initial
system banner.
(anderson@redhat.com)
"union", "task", "list" and "tree" commands. If a specified
structure member contains an embedded structure, the output may
be restricted to just the embedded structure by expressing the
.member argument as "member.member". If a specified structure
member is an array, the output may be restricted to a single array
element by expressing the .member argument as "member[index]".
Furthermore, these embedded member specifications may extend beyond
one level deep, for example, by expressing the member argument as
"member.member.member", or "member[index].member".
(Alexandr_Terekhov@epam.com, anderson@redhat.com)
data of specified cpus. It can be used in conjunction with all runq
command options. The cpus must be specified in a comma- and/or
dash-separated list; for examples, "3", "1,8,9", "1-23", or "1,8-15".
(anderson@redhat.com)
chains during session initialization. If a chain has been corrupted,
the patch prevents the sequence from entering an infinite loop, and
the error messages associated with corrupt/invalid chains have been
updated to report the pid_hash[] index number.
(anderson@redhat.com)
banner and by the "sys" command. Without the patch, many panic types
are categorized under the same generic message of the form:
PANIC: "Oops: 0000 [#1] SMP " (check log for details)
or in other types of crashes, no message is displayed at all. With
this patch, a more comprehensive search is made of the kernel log for
a more informative panic message.
(drc@yahoo-inc.com, anderson@redhat.com)
CONFIG_RT_GROUP_SCHED=n. Without the patch, real-time tasks queued
on a per-cpu rt_rq.rt_prio_array will not be displayed under the
"RT PRIO_ARRAY" header.
(mty.shibata@gmail.com)
the active tasks in x86_64 ELF or compressed dumpfiles created by the
KVM "virsh dump --memory-only" facility. Without the patch, the
backtraces of active tasks may show an invalid starting frame that
indicates "__schedule". The fix displays the exception RIP and dumps
the register contents that are stored in the dumpfile header. If the
active task was operating in the kernel, the backtrace continues from
there; if the task was operating in user-space, the backtrace is
complete at that point.
(anderson@redhat.com)
"virsh dump --memory-only --format <compression-type>" command,
where the compression-type is either "kdump-zlib", "kdump-lzo" or
"kdump-snappy". Without the patch, if an x86_64 guest kernel was loaded
with a non-zero "phys_base", the "--machdep phys_base=<offset>" command
line option was required as a workaround or the crash session would fail
with the warning message "WARNING: cannot read linux_banner string"
followed by the fatal error message "crash: vmlinux and <dumpfile name>
do not match!".
(anderson@redhat.com)
either "show" (the default) or "hide". When set to "hide", certain
command output associated with offline cpus will be hidden from view,
and the output will indicate that the cpu is "[OFFLINE]". The new
variable can be set during invocation on the crash command line via
the option "--offline [show|hide]". During runtime, or in a .crashrc
or other crash input file, the variable can be set by entering
"set offline [show|hide]". The commands or options that are affected
when the variable is set to "hide" are as follows:
o On X86_64 machines, the "bt -E" option will not search exception
stacks associated with offline cpus.
o On X86_64 machines, the "mach" command will append "[OFFLINE]"
to the addresses of IRQ and exception stacks associated with
offline cpus.
o On X86_64 machines, the "mach -c" command will not display the
cpuinfo_x86 data structure associated with offline cpus.
o The "help -r" option has been fixed so as to not attempt to
display register sets of offline cpus from ELF kdump vmcores,
compressed kdump vmcores, and ELF kdump clones created by
"virsh dump --memory-only".
o The "bt -c" option will not accept an offline cpu number.
o The "set -c" option will not accept an offline cpu number.
o The "irq -s" option will not display statistics associated with
offline cpus.
o The "timer" command will not display hrtimer data associated
with offline cpus.
o The "timer -r" option will not display hrtimer data associated
with offline cpus.
o The "ptov" command will append "[OFFLINE]" when translating a
per-cpu address offset to a virtal address of an offline cpu.
o The "kmem -o" option will append "[OFFLINE]" to the base per-cpu
virtual address of an offline cpu.
o The "kmem -S" option in CONFIG_SLUB kernels will not display
per-cpu data associated with offline cpus.
o When a per-cpu address reference is passed to the "struct"
command, the data structure will not be displayed for offline
cpus.
o When a per-cpu symbol and cpu reference is passed to the "p"
command, the data will not be displayed for offline cpus.
o When the "ps -[l|m]" option is passed the optional "-C [cpus]"
option, the tasks queued on offline cpus are not shown.
o The "runq" command and the "runq [-t/-m/-g/-d]" options will not
display runqueue data for offline cpus.
o The "ps" command will replace the ">" active task indicator to
a "-" for offline cpus.
The initial system information banner and the "sys" command will
display the total number of cpus as before, but will append the count
of offline cpus. Lastly, a fix has been made for the initialization
time determination of the maximum number of per-cpu objects queued
in a CONFIG_SLAB kmem_cache so as to continue checking all cpus
higher than the first offline cpu. These changes in behavior are not
dependent upon the setting of the crash "offline" variable.
(qiaonuohan@cn.fujitsu.com)
TASK_PARKED state in Linux 3.9 and later kernels. Without the patch,
the command's "ST" column entry for parked tasks shows "??". The
state column will now show "PA", and the foreach command will accept
"PA" as a "state" argument.
(anderson@redhat.com)
gathering of tasks from the kernel pid_hash[] in 2.6.24 and later
kernels. Without the patch, if an entry in a pid_hash[] chain is
not related to the "init_pid_ns" pid_namespace structure, any
remaining entries in the hlist chain are skipped.
(vvs@parallels.com)
(1) task.c: initialize the "curr" and "curr_my_q" variables in the
dump_tasks_in_task_group_cfs_rq() function.
(2) ramdump.c: make the "rd" and "len" return values from read()
and write() calls in write_elf() to be ssize_t types.
(3) cmdline.c: make the parsed PATH string buffer equal to the size
of the PATH string + 1 to prevent a possible buffer overflow
when a command line starts with a "!".
(anderson@redhat.com)
a crash-7.0.4 patch which added per-thread task_struct.rss_stat page
counts to the task's mm_struct.rss_stat page counts in order to show
an accurate/synchronized RSS value. Without the patch, the "ps"
command performance would degrade as the number of tasks increased,
most notably when there were thousands of tasks.
(panfy.fnst@cn.fujitsu.com, anderson@redhat.com)
the cgroup_name() function now utilizes kernfs_name(). Without the
patch, the command fails with the error message "runq: invalid
structure member offset: cgroup_dentry".
(anderson@redhat.com)
option to "runq -t", but which displays the amount of time that the
active task on each cpu has been running, expressed in a format
consisting of days, hours, minutes, seconds and milliseconds.
(anderson@redhat.com)
option to "ps -l", but which translates the task timestamp value from
a decimal or hexadecimal nanoseconds value into a more human-readable
string consisting of the number of days, hours, minutes, seconds and
milliseconds that have elapsed since the task started executing on a
cpu. More accurately described, it is the time difference between
the timestamp copied from the per-cpu runqueue clock when the task
last started executing compared to the most current value of the
per-cpu runqueue clock.
(anderson@redhat.com, bud.brown@redhat.com)
In addition, a new "ps -C <cpu-specifier>" option has been added
that can only be used with "ps -l" and "ps -m", which sorts the
global task list into per-cpu blocks; the cpu-specifier uses the
standard comma or dash separated list, expressed as "-C 1,3,5",
"-C 1-3", "-C 1,3,5-7,10", or "-Call" or "-Ca" for all cpus.
(anderson@redhat.com)
a slab object, consisting of the slab cache name and the address
value, separated by a colon, and encompassed in brackets:
[slab-cache-name:address]
Enhanced the "bt -F" option such that if "-F" is entered twice,
and if the stack frame contents reference a slab cache object, both
the slab cache name and the stack contents will be displayed within
brackets.
Enhanced the "rd -S" option such that if "-S" is entered twice,
and if the memory contents reference a slab cache object, both the
slab cache name and the memory contents will be displayed within
brackets.
(anderson@redhat.com)
CONFIG_FAIR_GROUP_SCHED, but not CONFIG_CFS_BANDWIDTH. Without the
patch, the command fails with the message "runq: invalid structure
member offset: cfs_rq_throttled".
(vinayakm.list@gmail.com)
