Terry : Core Dump

Core Dump

What is Core Dump?

A core dump is a technical term for a file containing detailed information about the state of an application at a given time, in a form which developers can make use of. They are particularly helpful for tracking down the reasons for application crashes, which is why many systems can be told to create one automatically when an application crash occurs.

The default action of certain signals is to cause a process to terminate and produce a core dump file, a disk file containing an image of the process's memory at the time of termination. This image can be used in a debugger (e.g., gdb) to inspect the state of the program at the time that it terminated. A list of the signals which cause a process to dump core can be found in signal.

How to create a core dump on Linux

A core dump is very helpful for helping us tracking down crashes of VirtualBox.

To create a core dump, start VirtualBox from a command line (e.g. xterm):

$ ulimit -c unlimited
$ echo -n 1 | sudo tee  /proc/sys/kernel/core_uses_pid
$ echo -n 1 | sudo tee /proc/sys/fs/suid_dumpable
$ VirtualBox

NOTE> and >> are Bash built-in redirection commands, after sudo only echo has root privilege so sudo echo -n 1 > /proc/sys/kernel/core_uses_pid will result in permission denied error.

If possible, start the virtual machine directly:

$ ulimit -c unlimited
$ echo -n 1 | sudo tee /proc/sys/kernel/core_uses_pid
$ echo -n 1 | sudo tee  /proc/sys/fs/suid_dumpable
$ /usr/lib/virtualbox/VirtualBox -startvm VM_NAME

IMPORTANT: Ensure that NO startup script (~/.bashrc, ~/.bash_profile, ~/.profile) contains an instruction like ulimit -c 0 as the limit cannot be increased once it was set to zero.

NOTE: Starting with version 2.0.0, the VirtualBox processes are started suid root, that is, with permissions to do things that "normal" applications cannot.

VirtualBox is (as of version 2.0.0) a system application which can be started from a user account without special privileges - a so-called "setuid" application. As core dumps are usually not allowed for these sensitive applications you have to explicitly allow them after starting your computer but before the crash which you want a core dump from. This can be done by typing the following command at the command line.

$ echo -n 1 | sudo tee /proc/sys/fs/suid_dumpable

When VirtualBox or one of its processes crashes, a file core.<pid> is created in the current directory. Be aware that core dumps can be very huge. Please compress the file before submitting it to a bug report. Or better don't attach the file to a report. Note that this core dump can contain a memory dump of your guest which can include sensitive information.

If several core files are created, you can check which process created them using the command

file core.<pid>

to be sure of the right one to send.

Core dump on Ubuntu

The procedure for actually creating a core dump is similar on most Linux systems, but on Ubuntu it is slightly different due to the Ubuntu crash reporting tool (Apport).

Core Pattern for Ubuntu kernels

cat /proc/sys/kernel/core_pattern 
|/usr/share/apport/apport %p %s %c

To create a core dump on an Ubuntu system, create a file called ~/.config/apport/settings in your home directory with the contents

[main]
unpackaged=true

and run VirtualBox. Any crashes of VirtualBox after you have created the file will cause a crash report to be created in /var/crash with a name like "_usr_lib_virtualbox_VirtualBox.1000.crash".

Extract the core file using the apport-unpack tool

apport-unpack <report > <target_directory>

The core dump file can be found in target directory, named CoreDump.

To stop crash files being created for non-system applications delete the file that you created again.

How to create dumps on Mac OS X

To create a core dump on Mac OS X, start VirtualBox from a command line:

ulimit -c unlimited
VirtualBox

or better start the VM directly:

ulimit -c unlimited
/Applications/VirtualBox.app/Contents/MacOS/VirtualBox -startvm VM_NAME

Ensure that NO startup script (~/.bashrc, ~/.bash_profile, ~/.profile) contains an instruction like ulimit -c 0 as the limit cannot be increased once it was set to zero.

Alternatively (for Finder & launchpad)

$ launchctl limit core unlimited

or for permanent effect the same could be specified in $HOME/.launchd.conf or /etc/launchd.conf.

The core files can be found in /cores directory.

How to create dumps on Solaris

To get core on Solaris host run the following command as root

coreadm -g /var/cores/core.%f.%p -i core.%f.%p -e global -e process -e global-setid -e proc-setid -e log

The cores will now be placed in /var/cores/ folder with the global dumps will go into /var/crash/<hostname>

System core dumps need to be enabled via dumpadm. The important thing is to have "Savecore enabled" to "yes" (use dumpadm -y). The config. should look something like this:

dumpadm
Dump content: kernel pages
Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/myhostname
Savecore enabled: yes
Save compressed: on

Forcing VirtualBox to terminate with a core dump

Sometimes it is required to force a VirtualBox process to terminate, for example, a VM hangs for some unknown reason. This can be done as follows:

On Linux

$ ulimit -c unlimited
$ echo -n 1 | sudo tee /proc/sys/fs/suid_dumpable
$ /usr/lib/virtualbox/VirtualBox -startvm VM_NAME &
$ pidof VirtualBox
7145
$ kill -4 7145

As an alternative to kill you can do

$ pidof VirtualBox
7145
$ gcore 7145

NOTE: gcore is a shell script calling gdb. Some Linux distributions do NOT have it in the GDB package.

or in GDB

# Attach to the virtualbox process using PID
(gdb) attach 641
Attaching to process 641
# Generate core dump
(gdb) gcore
Saved corefile core.641
# detach the virtualbox process previously attached
(gdb) detach
Detaching from program: /usr/lib/virtualbox/VirtualBox, process 641

On Mac OS X

$ ulimit -c unlimited
$ /Applications/VirtualBox.app/Contents/MacOS/VirtualBox -startvm VM_NAME &
$ ps aux | grep VirtualBox
... 7145 ... VirtualBox ...
$ kill -4 7145

On Solaris

# ulimit -c unlimited
# /opt/VirtualBox/amd64/bin/VirtualBox -startvm VM_NAME &
# ps -ef|grep VirtualBox
... 7145 ... VirtualBox ...
# kill -4 7145

You can find result core file according location specified in coreadm

# coreadm

NOTE: Passing the signal number 4 (SIGILL) is essential! The same applies to the alternative frontends VBoxHeadless and VBoxSDL.

Linux with systemd

For Linux distributions with systemd (Arch Linux, Fedora, OpenSUSE etc...), core dumps are stored in systemd journal.

cat /proc/sys/kernelcore_pattern
|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e 

The core dumps can be accessed using systemd-coredumpctl

For example

systemd-coredumpctl list
TIME                                         PID   UID   GID SIG EXE
              Thu 2013-02-21 20:54:30 EST    283  1000  1000  11 /usr/bin/gsetti
              Sat 2013-02-23 16:11:49 EST    249  1000  1000  11 /usr/bin/gsetti
              Thu 2013-03-14 11:40:08 EST   1706     0     0   4 /usr/sbin/httpd
              Thu 2013-03-14 13:32:01 EST   2105     0     0  11 /tmp/segfault
(END)

Send core dump for PID 2105 to GDB

systemd-coredumpctl gdb 2105

This behaviour can be disabled with a simple "hack":

$ ln -s /dev/null /etc/sysctl.d/coredump.conf
$ /lib/systemd/systemd-sysctl # or possibly a reboot

As always, the size of core dumps has to be equal or higher than the size of the core that is being dumped, as done by for example ulimit -c unlimited.

More on core dumps for Linux

/proc/sys/kernel/core_pattern

By default, a core dump file is named core, but the /proc/sys/kernel/core_pattern file (since Linux 2.6 and 2.4.21) can be set to define a template that is used to name core dump files.

The template can contain % specifiers which are substituted by the following values when a core file is created:

  • %% a single % character
  • %p PID of dumped process
  • %u (numeric) real UID of dumped process
  • %g (numeric) real GID of dumped process
  • %s number of signal causing dump
  • %t time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC)
  • %h hostname (same as nodename returned by uname(2))
  • %e executable filename (without path prefix)
  • %E pathname of executable, with slashes ('/') replaced by exclama‐tion marks ('!').
  • %c core file size soft resource limit of crashing process (since Linux 2.6.24)
Icon

A single % at the end of the template is dropped from the core filename, as is the combination of a % followed by any character other than those listed above. All other characters in the template become a literal part of the core filename. The template may include '/' characters, which are interpreted as delimiters for directory names. The maximum size of the resulting core filename is 128 bytes (64 bytes in kernels before 2.6.19). The default value in this file is "core". For backward  compatibility, if /proc/sys/kernel/core_pattern does not include "%p" and /proc/sys/kernel/core_uses_pid (see below) is nonzero, then .PID will be appended to the core filename.

/proc/sys/kernel/core_uses_pid

Since version 2.4, Linux has also provided a more primitive method of controlling the name of the core dump file. If the /proc/sys/kernel/core_uses_pid file contains the value 0, then a core dump file is simply named core.

sudo echo 1 > /proc/sys/kernel/core_uses_pid

If this file contains a nonzero value, then the core dump file includes the process ID in a name of the form core.PID

/proc/sys/fs/suid_dumpable

The value in this file determines whether core dump files are produced for set-user-ID or otherwise protected/tainted binaries. Three different integer values can be specified:

  • 0 (default) This provides the traditional (pre-Linux 2.6.13) behavior. A core dump will not be produced for a process which has changed credentials (by calling seteuid(2), setgid(2), or similar, or by executing a set-user-ID or set-group-ID program) or whose binary does not have read permission enabled.
  • 1 ("debug") All processes dump core when possible. The core dump is owned by the file system user ID of the dumping process and no security is applied. This is intended for system debugging situations only. Ptrace is unchecked.
  • 2 ("suidsafe") Any binary which normally would not be dumped (see "0" above) is dumped readable by root only. This allows the user to remove the core dump file but not to read it. For security reasons core dumps in this mode will not overwrite one another or other files. This mode is appropriate when administrators are attempting to debug problems in a normal environment.

Reference

VirtualBox - Core Dump