KVM Hypervisor (Optional)#

This document demonstrates the steps for setting up an environment that implements the Robotics KVM virtualization solution. The examples below are provided in both QEMU command-line format and libvirt XML parameter format.

Note

The parameter values in the examples below are not fixed. Please modify them based on your hardware resources, firmware/image paths, and business requirements.

Scenario#

The following diagram shows the overall architecture of the Robotics KVM solution on Intel® Core™ Ultra Series 3, organized into three layers: the Intel® Core™ Ultra Series 3 hardware, the Host OS, and the Guest OS. Each of the hardware and Host OS layers is further partitioned into a real-time (RT) domain and a non-real-time (non-RT) domain.

kvm_solution

(Hardware, Intel® Core™ Ultra Series 3 platform) — Using Intel® Core™ Ultra Series 3 X7 358H as an example: dedicated to the RT domain (4x LPE cores + a NIC), shared by the non-RT domain and Guest OS (4x P cores, 8x E cores, NIC, USB, and NVMe), and passed through to the Guest OS (Intel® Arc™ graphics, Intel NPU, USB controller)
(Host OS) — Ubuntu 24.04 with a 6.17 RT kernel. Its RT domain maps to the L1 RT hardware and runs motion control, using the RT NIC to drive external motors over EtherCAT. Its non-RT domain maps to the L1 non-RT hardware and provides KVM + QEMU virtualization services.
(Guest OS) — Ubuntu 24.04 with a 6.17 kernel. It receives Intel® Arc™ graphics, Intel NPU, and USB controller via KVM passthrough from the Host, and shares net and NVMe with the Host through virtio.

Basic Environment Setup#

Install Ubuntu 24.04 as the host OS#

Install and set up Ubuntu 24.04 on the main storage. Follow the steps to set up the OS and APT repository: OS Setup.
Set up real-time Linux. Follow the steps to set up real-time Linux: RT Linux. Follow the steps to complete real-time tuning: RT Tuning.
Install host packages for the KVM/QEMU system.

sudo apt update
sudo apt install qemu-system-x86 qemu-utils ovmf bridge-utils pciutils \
        libvirt-daemon libvirt-daemon-driver-qemu libvirt-daemon-system libvirt-clients

Set up virtualization related setting. Make sure Intel (VMX) Virtualization and VT-d are enabled in BIOS setting. Huge pages are also needed to boot the Guest OS. Edit boot parameters in host environment, add:

default_hugepagesz=1G hugepages=[size allocated for Guest OS in GB] hugepagesz=1G

Build the Guest OS image#

Build a QEMU qcow2 image that runs Ubuntu 24.04 OS. Refer to Build Image.
Set up the OS and install packages for the VM. Follow the steps to set up the OS and APT repository: OS Setup.

Build the OVMF and iGPU ROM file#

Refer to Build OVMF for KVM to build a specific OVMF image and an iGPU ROM file.

Launch Ubuntu Virtual Machines#

Ensure KVM and virtualization are available on the host.

lsmod | grep kvm
sudo kvm-ok

Check qemu is ready and version.

qemu-system-x86_64 --version

expected result:

QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.17)
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Ensure the VM image and firmware files exist (adjust paths as needed).

ls -lh /usr/share/OVMF/OVMF_VARS_iGPU.fd
ls -lh /usr/share/OVMF/OVMF_CODE_iGPU.fd
ls -lh /var/lib/libvirt/images/ubuntu24.04.qcow2
ls -lh /var/lib/libvirt/roms/iGPU_GOP.rom

If the guest image is not ready, please refer to Build Image. If OVMF and iGPU ROM file are not ready, please refer to Build OVMF Romfile to prepare OVMF and the iGPU ROM file.

Ensure bridge networking is configured and active.

Verify bridge network status on the host:

ip a show kvm-br0

Please refer to bridge_setting if bridge networking is not ready.

Launch VM example.

You can launch the guest VM with either:

A direct QEMU command (good for quick iteration)
libvirt (virsh) domain management (good for lifecycle management)

QEMU

Bind the VFIO driver for the passthrough device.

sudo modprobe vfio_pci
sudo systemctl stop gdm

bind_vfio() {
  local bdf="$1"
  local dev_path="/sys/bus/pci/devices/$bdf"
  local cur_driver

  if [[ ! -e "$dev_path" ]]; then
     echo "No such PCIe device, $bdf"
     return
  fi

  echo vfio-pci | sudo tee "$dev_path/driver_override" >/dev/null 2>&1

  if [[ -e "$dev_path/driver/unbind" ]]; then
    echo "$bdf" | tee "$dev_path/driver/unbind" >/dev/null 2>&1
  fi

  echo "$bdf" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind >/dev/null 2>&1

  cur_driver=$(basename $(readlink /sys/bus/pci/devices/${bdf}/driver) 2>/dev/null)
  if [[ "$cur_driver" == "vfio-pci" ]]; then
     echo "Bound $bdf to vfio-pci successfully."
  fi
}

# 00:02.0 is Intel iGPU; 0000:00:14.0 and 0000:00:14.1 are USB controllers
bind_vfio 0000:00:02.0
bind_vfio 0000:00:14.0
bind_vfio 0000:00:14.1

Launch the VM

# host=00:02.0 is Intel iGPU
# host=0000:00:14.0 is USB controller
VM_NET_MAC="52:54:00:8d:85:0d"
sudo qemu-system-x86_64 \
  -name ubuntu-vm,debug-threads=on \
  -enable-kvm \
  -machine q35 \
  -cpu host \
  -smp cores=12,threads=1,sockets=1 \
  -m 24G \
  -object memory-backend-memfd,hugetlb=on,hugetlbsize=1G,id=mem1,size=24G \
  -machine memory-backend=mem1 \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_iGPU.fd \
  -drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS_iGPU.fd \
  -drive file=/var/lib/libvirt/images/ubuntu24.04.qcow2,format=qcow2,cache=none,if=virtio \
  -netdev tap,id=net0,ifname=tap0,script=/etc/qemu-ifup,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=${VM_NET_MAC} \
  -device vfio-pci,host=00:02.0,x-igd-gms=2,id=hostdev0,x-igd-opregion=on,romfile=/var/lib/libvirt/roms/iGPU_GOP.rom,addr=0x2 \
  -device vfio-pci,host=0000:00:14.0 \
  -serial mon:stdio \
  -nographic \
  -vga none

Libvirt

Prepare the domain XML.

Please refer to example.xml.

Define and start the VM.

virsh define ubuntu24.04_example.xml
virsh start ubuntu24.04
virsh list --all

Reboot and shut down the VM.

virsh reboot ubuntu24.04
virsh destroy ubuntu24.04

KVM Management Solutions#

The following sections provide examples for managing VMs using either the direct QEMU monitor or libvirt.

QEMU Monitor#

QEMU Monitor is a runtime management and debugging interface provided by QEMU. Even if the Guest OS is already unresponsive, as long as the QEMU process still exists, you can usually use the Monitor to check the Guest-side and QEMU-side status.

QEMU Monitor can be used to inspect:

vCPU status;
CPU registers;
PCI devices;
Memory map;
IRQ status;
QEMU threads;
Guest memory dump.

Using a Unix Socket Monitor#

QEMU startup parameter:

-monitor unix:/tmp/qemu-monitor.sock,server,nowait

Connection method:

nc -U /tmp/qemu-monitor.sock

Using a Telnet Monitor#

QEMU startup parameter:

-monitor telnet:127.0.0.1:9999,server,nowait

Connection method:

telnet 127.0.0.1 9999

Or:

nc 127.0.0.1 9999

Verify the port:

ss -lntp | grep 9999
lsof -i:9999

Common QEMU Monitor Commands#

Command	Description
`info cpus`	Show the status and execution state of all virtual CPUs in the guest.
`info registers`	Display the current CPU register values of the selected virtual CPU.
`info pci`	List all PCI devices currently visible to the virtual machine.
`info mtree`	Display the guest physical memory map and QEMU memory region hierarchy.
`info irq`	Show interrupt routing and current IRQ status of the virtual machine.
`stop`	Pause virtual machine execution while keeping its state intact.
`cont`	Resume execution of a previously paused virtual machine.
`system_reset`	Reset the virtual machine as if the hardware reset button had been pressed.
`dump-guest-memory /tmp/guest.core`	Create a Guest memory dump and save it to the specified file for crash analysis.
`help`	Show QEMU Monitor command information.

Libvirt#

libvirt is a commonly used management framework for KVM/QEMU, and virsh is the command-line management tool provided by libvirt. For VMs managed by libvirt, it is recommended to use virsh first for the following operations:

Check VM status, configuration, and vCPU information;
View or export the VM XML configuration;
Start, shut down, reboot, or forcibly stop a VM;
Trigger a Guest memory dump through the libvirt management interface;
Check libvirt and QEMU logs.

Common Management Commands#

Command	Description
`virsh list --all`	List all Guest VMs, including Guest VMs that are shut off.
`virsh dominfo <vm_name>`	Show basic information about a Guest VM.
`virsh dumpxml <vm_name>`	Export the Guest VM XML configuration. This is useful for troubleshooting device passthrough, CPU pinning, memory configuration, and similar issues.
`virsh vcpupin <vm_name>`	Check vCPU pinning status.
`virsh vcpuinfo <vm_name>`	Check vCPU binding information.
`virsh console <vm_name>`	Connect to the Guest VM console.
`virsh shutdown <vm_name>`	Shut down the Guest VM gracefully.
`virsh destroy <vm_name>`	Forcibly power off the Guest VM. Use this command with caution.
`virsh start <vm_name>`	Start a Guest VM that is currently in the shut off state.
`virsh reboot <vm_name>`	Reboot a Guest VM that is currently running.

Recommended VM Dump Method#

If the VM is managed by libvirt, it is recommended to use virsh dump to export Guest memory:

virsh dump <vm_name> /tmp/guest.core --memory-only

If you want to minimize interruption to the running Guest, you can try:

virsh dump <vm_name> /tmp/guest.core --memory-only --live

If you want the VM to be marked as crashed after the dump:

virsh dump <vm_name> /tmp/guest.core --memory-only --crash

Notes:

/tmp/guest.core is a path on the Host file system;
The file size is usually close to the Guest memory size;
For device passthrough scenarios, prefer --memory-only to avoid additional complexity introduced by device state.

Checking libvirt / QEMU Logs#

journalctl -u libvirtd -b
journalctl -u libvirtd -f
sudo cat /var/log/libvirt/qemu/<vm_name>.log

It is recommended to preserve the debugging context:

mkdir -p debug_logs
sudo cp /var/log/libvirt/qemu/<vm_name>.log debug_logs/
journalctl -u libvirtd -b > debug_logs/libvirtd.log
dmesg > debug_logs/dmesg.log

KVM Debugging Solutions#

Core debugging solutions for a KVM Guest OS can be divided into two categories:

Guest memory dump + crash: suitable for scenarios where the Guest hangs or becomes unresponsive but the QEMU process still exists;
kdump + crash: suitable for scenarios such as guest kernel panic, oops, or actively triggered panic.

Both approaches require a matching vmlinux file. vmlinux must come from the kernel build that the Guest OS was running when the issue occurred.

Debug Preparation#

Crash Tool#

The crash tool version provided by Ubuntu 22.04/24.04 may fail to parse dump files, so a newer version of crash needs to be built.

Install Build Dependencies#

sudo apt update
sudo apt install -y \
 build-essential \
 bison \
 flex \
 texinfo \
 zlib1g-dev \
 liblzma-dev \
 libexpat1-dev \
 libgmp-dev \
 libmpfr-dev \
 libmpc-dev \
 wget \
 binutils \
 git

Get the crash Tool Source Code#

Download crash 9.0.2 source code from crash_9.0.2, extract it:

tar xf 9.0.2.tar.gz
cd crash-9.0.2

Prepare the GDB 16.2 Source Package#

crash 9.0.2 requires gdb-16.2.tar.gz.

Download link: GDB-16.2

Place it in the crash source root directory:

crash/
 ├── Makefile
 ├── gdb-16.2.tar.gz
 ├── gdb-16.2.patch
 └── ...

If automatic download fails due to a network proxy, manually download it and copy it into this directory.

You can also extract it manually:

tar xf gdb-16.2.tar.gz

Build crash 9.0.2#

make -j$(nproc)

The build process usually performs the following steps:

Extracts gdb-16.2;
Applies gdb-16.2.patch;
Configures and builds GDB;
Merges GDB into crash;
Generates ./crash.

If a previous build failed, it is recommended to clean and rebuild:

make clean
make -j$(nproc)

If clearer error output is needed:

make -j1

Common Build Errors#

Missing GMP / MPFR

Error:

configure: error: Building GDB requires GMP 4.2+, and MPFR 3.1.0+.

Solution:

sudo apt install -y libgmp-dev libmpfr-dev libmpc-dev
make clean
make -j$(nproc)

vmlinux#

Whether the dump file is guest.core or vmcore, crash analysis requires:

sudo crash vmlinux guest.core

Or:

sudo crash vmlinux /var/crash/<timestamp>/vmcore

The vmlinux file must be the uncompressed ELF image with symbols that correspond to the kernel running in the Guest OS when the dump was generated.

Check the currently running kernel in the Guest:

uname -r

If the debug symbol package is installed, a common path is:

/usr/lib/debug/boot/vmlinux-$(uname -r)

If vmlinux cannot be found, obtain it in the Guest OS by installing the -dbg Debian package through apt, or compile the kernel source package to generate and install a Debian package with debug symbols. For example, if the kernel is 6.17.11-intel-ese-experimental-lts-rt, install:

linux-image-6.17.11-intel-ese-experimental-lts-rt-dbg.deb

After obtaining vmlinux, check whether vmlinux and the dump file match.

Check the Linux version string in the dump:

sudo strings guest.core | grep "Linux version" | head

Check the Linux version string in vmlinux:

strings vmlinux | grep "Linux version" | head

Notes:

guest.core is a full memory image and may contain multiple historical kernel banners, dmesg ring buffers, or module strings;
strings guest.core can only be used as a reference and should not be the only basis for judgment;
The most reliable approach is to confirm the exact kernel build running in the Guest when the dump occurred, and use the corresponding vmlinux from that build.

Check file formats:

sudo file guest.core
file vmlinux

Expected output:

guest.core: ELF 64-bit LSB core file
vmlinux: ELF 64-bit LSB executable/shared object, x86-64

kdump#

Prerequisites#

If you want an ECI kernel to support kdump, the kernel needs to enable kexec/kdump-related configuration options. At minimum, the following kernel configuration options should be enabled:

CONFIG_CRASH_DUMP=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_PROC_VMCORE=y

If these options are missing, you may see symptoms such as:

The kdump-tools service fails to start;
/sys/kernel/kexec_crash_loaded is not 1;
After panic, the system cannot enter the second kernel;
/proc/vmcore does not exist or the dump cannot be saved.

For ECI kernel compilation, refer to: Linux BSP

Installation#

Install inside the Guest OS:

sudo apt update
sudo apt install -y kdump-tools crash

Enable the service:

sudo systemctl enable kdump-tools
sudo systemctl restart kdump-tools
systemctl status kdump-tools

Check whether the crash kernel is loaded:

cat /sys/kernel/kexec_crash_loaded

Expected output:

Configure crashkernel#

Edit grub:

sudo vim /etc/default/grub

Add or modify the kernel command line, for example:

GRUB_CMDLINE_LINUX="crashkernel=1G console=tty0 console=ttyS0,115200n8"

Update grub and reboot:

sudo update-grub
sudo reboot

Verify crashkernel:

dmesg | grep -i crashkernel
cat /sys/kernel/kexec_crash_loaded

Expected output:

Manually Trigger Panic to Validate kdump#

Enable SysRq:

echo 1 | sudo tee /proc/sys/kernel/sysrq

Trigger panic:

echo c | sudo tee /proc/sysrq-trigger

After the system reboots, check:

ls -lah /var/crash/

Analyze vmcore:

sudo crash vmlinux /var/crash/<timestamp>/vmcore

Guest Memory Dump + crash#

This solution is suitable for:

Guest OS hangs;
Guest is unresponsive but the QEMU process still exists;
Exporting a Guest physical memory image from the Host side;
Analyzing Guest kernel stack, task state, interrupt state, or memory state.

Purpose of dump-guest-memory#

dump-guest-memory is a QEMU Monitor command used to export a Guest physical memory image. It usually contains:

Guest physical RAM;
CPU state;
ELF header;
Memory layout.

It does not directly display:

kernel stack;
call trace;
backtrace.

To view the guest kernel stack, use crash or gdb together with the matching vmlinux to analyze guest.core.

Exporting guest.core Through QEMU Monitor#

dump-guest-memory /tmp/guest.core

Forward the QEMU Monitor command through virsh:

virsh qemu-monitor-command <vm_name> --hmp \
    "dump-guest-memory /tmp/guest.core"

If the VM is managed by libvirt, you can also use:

virsh dump <vm_name> /tmp/guest.core --memory-only

Analyzing guest.core with crash#

sudo crash vmlinux /tmp/guest.core

After entering the crash> prompt, you can use the following commands to inspect Guest OS information:

Crash Command	Purpose
`sys`	Show system information.
`log`	Show kernel log or panic information.
`bt`	Show the backtrace of the current context.
`ps`	Show the task list.
`foreach bt`	Show backtraces for all tasks.
`runq`	Show runqueue information.
`irq`	Show interrupt information.
`kmem -i`	Show memory overview.
`kmem -s`	Show slab information.
`mod`	Show module information.
`q`	Quit crash.

kdump + crash#

This solution is suitable for:

Guest kernel panic;
Guest kernel oops;
Automatically saving vmcore inside the Guest when a panic occurs;
Analyzing panic-scene kernel stack, task table, runqueue, IRQ, modules, and similar information.

Basic Principle of kdump#

kdump relies on the kexec/kdump mechanism. When the first kernel panics, the system jumps to the reserved crash kernel, and the second kernel saves /proc/vmcore to /var/crash/<timestamp>/.

Typical output directory:

/var/crash/<timestamp>/
├── dmesg.<timestamp>
└── vmcore

In Ubuntu/Debian kdump-tools, the following may also be generated:

/var/crash/<timestamp>/
├── dmesg.<timestamp>
└── dump.<timestamp>

dump.<timestamp> may be a flattened format generated by makedumpfile -F, and it must be converted to a standard ELF vmcore before it can be analyzed by crash.

kdump + crash Debugging Flow#

Guest kernel panic
  ↓
kdump second kernel starts
  ↓
Save vmcore or dump.<timestamp>
  ↓
If needed, convert dump.<timestamp> to vmcore
  ↓
Analyze with crash 9.0.2 + matching vmlinux

Common Debugging Issues#

Dump Conversion Succeeds but crash Analysis Fails#

If makedumpfile -R successfully generates vmcore, but running crash exits at the following stage:

please wait... (gathering task table data)

And you have already confirmed that:

vmlinux exactly matches the first kernel that panicked;
vmcore is already an ELF core file;
The crash tool version is new enough;

Then it is very likely that the filtering level of makedumpfile is too aggressive, causing key memory pages required by crash analysis to be filtered out.

Current setting:

MAKEDUMP_ARGS="-c -d 31"

Recommended change to a more conservative filtering level:

sudo vim /etc/default/kdump-tools

Change:

MAKEDUMP_ARGS="-c -d 31"

To:

MAKEDUMP_ARGS="-c -d 1"

If this configuration does not exist in the file, you can add it directly.

Restart kdump-tools:

sudo systemctl restart kdump-tools

Confirm that the crash kernel is reloaded:

cat /sys/kernel/kexec_crash_loaded

Expected output:

Then collect or trigger a new kdump file, and run:

cd /var/crash/<new_timestamp>
makedumpfile -R vmcore < dump.<new_timestamp>
sudo crash /path/to/matching-vmlinux vmcore

If you can then enter:

crash>

It indicates that the root cause was that -d 31 filtered out key memory pages needed by crash analysis.

Recommendation:

In ECI/ESE RT Kernel or Guest VM panic debugging scenarios, prefer:

MAKEDUMP_ARGS="-c -d 1"

After confirming that the dump is analyzable, evaluate whether to increase the filtering level according to disk space and dump size.

KVM Hypervisor (Optional)#

Scenario#

Basic Environment Setup#

Install Ubuntu 24.04 as the host OS#

Build the Guest OS image#

Build the OVMF and iGPU ROM file#

KVM Example Parameters#

CPU#

CPU Affinity#

Memory#

Disk#

Bridge Network#

Passthrough PCI Device#

Passthrough Intel® iGPU#

OVMF#

Launch Ubuntu Virtual Machines#

KVM Management Solutions#

QEMU Monitor#

Using a Unix Socket Monitor#

Using a Telnet Monitor#

Common QEMU Monitor Commands#

Libvirt#

Common Management Commands#

Recommended VM Dump Method#

Checking libvirt / QEMU Logs#

KVM Debugging Solutions#

Debug Preparation#

Crash Tool#

Install Build Dependencies#

Get the crash Tool Source Code#

Prepare the GDB 16.2 Source Package#

Build crash 9.0.2#

Common Build Errors#

vmlinux#

kdump#

Prerequisites#

Installation#

Configure crashkernel#

Manually Trigger Panic to Validate kdump#

Guest Memory Dump + crash#

Purpose of dump-guest-memory#

Exporting guest.core Through QEMU Monitor#

Analyzing guest.core with crash#

kdump + crash#

Basic Principle of kdump#

kdump + crash Debugging Flow#

Common Debugging Issues#

Dump Conversion Succeeds but crash Analysis Fails#