How to Read Kernel Logs for Hardware Issues

This guide shows you how to locate, interpret, and act on kernel log messages to diagnose faulty hardware components like memory, storage, and USB devices on Linux systems.

Difficulty: Intermediate

Time: 15-30 minutes

Best for: Linux users & system administrators

Steps may vary by device and OS version. Make a backup before major changes.

Quick Answer

Open a terminal and run sudo dmesg or journalctl -k to view kernel logs.
Look for lines containing keywords like ERROR, WARNING, fail, timeout, or specific hardware terms (e.g., ATA, usb, iommu).
Search for known error patterns: "I/O error" for storage, "EDAC" for memory, "irq" for interrupt conflicts.
Filter logs to a specific time frame with dmesg --since "10 minutes ago" or journalctl -k --since -1h.
Check persistent logs in /var/log/kern.log or via journalctl -k.

If this didn't work, follow the detailed checklist and symptom guide below to systematically diagnose the hardware problem.

Before You Start

What You Need

Terminal access (often via SSH or local console).
sudo privileges or root access.
Note of when the problem started or last occurred.
A general idea of your system hardware (e.g., disk model, USB device).

What Not to Do

Don't ignore repeated error messages; they indicate persistent failure.
Avoid physically reseating components without first checking for driver/logical issues.
Don't delete log files as a troubleshooting step.
Avoid making multiple hardware/software changes at once; it complicates diagnosis.

Fast Checklist

Check for kernel messages with dmesg -T | tail -50.
Confirm the log isn't full of old errors by clearing ring buffer with sudo dmesg -c (then reproduce issue).
Search for "error", "failed", "unable", "timeout", "corrupted".
Identify the hardware subsystem (USB, SATA, PCIe, Memory).
Look up specific error codes or messages online (include kernel version).
Test hardware with built-in tools (smartctl, memtester).
Check driver status with lsmod and dmesg | grep driver_name.
Update your kernel and firmware packages.
Disable problematic kernel modules temporarily (if safe).
Restart the affected service or the entire system to see if errors persist.
Inspect /var/log/kern.log for historical data.
Monitor logs in real-time with sudo dmesg -w while reproducing the issue.
Verify cable connections and try a different port for external devices.
Check system temperature and fan speeds for overheating signs.
Reduce system load to see if errors are load-dependent.

Mind Map: Kernel Log Troubleshooting Flow

Kernel Log Analysis (dmesg/journalctl)
|
├── 1. LOCATE ERROR
│   ├── Filter by time: dmesg --since "1 hour ago"
│   ├── Follow live: dmesg -w
│   └── Check persistent: journalctl -k -b
|
├── 2. CLASSIFY ERROR
│   ├── Storage (SATA/USB/NVMe)
│   │   ├── Symptoms: I/O error, timeout, revalidation failed
│   │   └── Action → Check cables, run smartctl, test on another port
│   │
│   ├── Memory (RAM)
│   │   ├── Symptoms: EDAC errors, page fault, MCE (Machine Check Exception)
│   │   └── Action → Run memtest86+, reseat RAM, replace module
│   │
│   ├── CPU/Overheat
│   │   ├── Symptoms: Thermal throttling, clock/watchdog timeout
│   │   └── Action → Check cooling, clean fans, repaste CPU
│   │
│   ├── USB/Peripheral
│   │   ├── Symptoms: disconnect/reset, enumeration failed
│   │   └── Action → Try different port/cable, check power, update firmware
│   │
│   └── PCI/Graphics
│       ├── Symptoms: AER (Advanced Error Reporting), IRQ conflict
│       └── Action → Reseat card, update kernel/drivers, check BIOS settings
|
└── 3. ACT & VERIFY
    ├── Apply fix (update, replace, reconfigure)
    ├── Clear log buffer (dmesg -c)
    ├── Re-trigger the event
    └── Verify error is gone from new logs

Step-by-Step Guide

Step 1: Access the Kernel Logs

What to do: Open a terminal and use the primary command to view kernel ring buffer messages.

sudo dmesg or journalctl -k

Why it helps: dmesg reads from the kernel's ring buffer, showing messages from the current boot. journalctl -k shows kernel messages from the systemd journal, which may include previous boots.

Expected outcome: You see a chronological list of kernel events. If the system is stable, most lines will be informational.

Step 2: Filter and Search for Errors

What to do: Pipe the output into grep to search for critical keywords and filter by time.

dmesg -T --level=err,warn | tail -20
journalctl -k --since "yesterday" | grep -i "error\|failed\|timeout"

Why it helps: Isolating errors and warnings from the noise makes problem patterns visible. The -T flag adds human-readable timestamps.

Expected outcome: A shorter list of potentially problematic entries. If the list is empty, the issue might not be kernel-level or may be cleared.

Note: Some systems use dmesg --human for timestamps. If -T fails, check your dmesg version with dmesg --help.

Step 3: Interpret the Error Message

What to do: Examine a specific error line. Note the component (e.g., sd 0:0:0:0: for SCSI disk), error code, and description.

Example: [ 1234.567890] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen

Why it helps: The format often reveals the subsystem (ata1 = first SATA controller), the nature of the error (exception, frozen), and sometimes a suggested action.

Expected outcome: You can pinpoint the failing hardware (e.g., "this error is about the first SATA port"). Copy the full line for further research.

Step 4: Correlate with System Events

What to do: Note the timestamp of the error and check what you or the system was doing at that time (e.g., plugging in a USB device, starting a heavy process).

Why it helps: Hardware errors are often triggered by specific actions. This correlation can confirm the faulty component.

Expected outcome: You establish a pattern, like "every time I copy a large file to this disk, an I/O error appears in the log."

Step 5: Take Corrective Action

What to do: Based on the identified subsystem and error, apply the initial fix. For example, for a USB error, try a different port and cable.

Why it helps: Simple physical fixes resolve many transient hardware issues.

Expected outcome: After the fix and a system restart or reconnection, the error should stop appearing in new log entries.

Common Causes (Explain Like I’m Busy)

Failing Storage (HDD/SSD): Bad sectors or a dying controller. Symptom: Repeated "I/O error", "SATA link down", or "revalidation failed" messages.
Faulty RAM: Corrupt memory cells. Symptom: "EDAC" errors, "MCE", "page fault" in kernel log, often causing system crashes.
Overheating: Dusty fans or failing thermal paste. Symptom: "CPU throttling", "thermal trip" messages, system slowing down.
Loose or Bad Cables: Poor physical connection for SATA or power. Symptom: Intermittent "link down" or device disconnection errors.
Power Supply Issues: Insufficient or unstable power delivery. Symptom: Random resets, "under-voltage" warnings (on some SBCs like Raspberry Pi).
Kernel Driver Bugs: Incompatible or buggy driver for new hardware. Symptom: "kernel panic" or "failed to probe" messages after an update.
USB Power/Protocol Issues: Device drawing too much power or protocol mismatch. Symptom: "reset high-speed USB device", "enumeration failed".
Firmware Bugs: Outdated or buggy device/UEFI firmware. Symptom: ACPI errors, boot hangs, or specific PCIe initialization failures.

Diagnostics (Safe, Text-First)

Linux (General)

Use dmesg -H for human-readable output with pager.
Check hardware-specific logs: sudo cat /sys/class/block/sda/device/state (replace sda).
Monitor interrupts and hardware stats: cat /proc/interrupts and cat /proc/meminfo.
Test disk health: sudo smartctl -a /dev/sda (install smartmontools first).

Systemd-based Distros (e.g., Ubuntu, Fedora)

View all kernel messages from the current boot: journalctl -k -b
Show kernel messages with priority 'err' or higher: journalctl -k -p err
Follow new messages live: journalctl -k -f
Export logs to a file for sharing: journalctl -k --since="2024-01-01" > kernel_errors.txt

Fixes by Symptom

Symptom: "I/O error", "buffer I/O error on device sdX"

Likely cause: Failing hard drive or SSD, bad cable connection.

Best first fix: Check SATA/USB cable connections, try a different port/cable.

Next fix: Run smartctl -a /dev/sdX to check SMART status; back up data immediately if failing.

Symptom: "EDAC MC0: UE" or "Hardware Error" with MCE

Likely cause: Uncorrectable memory error (bad RAM).

Best first fix: Run a memory test with memtester or boot from a memtest86+ USB.

Next fix: Reseat RAM modules, clean contacts. If errors persist, replace the faulty DIMM.

Symptom: "usb 2-1: reset high-speed USB device number 2 using xhci_hcd"

Likely cause: USB device drawing too much power, faulty cable, or driver issue.

Best first fix: Disconnect and reconnect the device, try a different USB port (preferably on a different controller).

Next fix: Replace the USB cable. Check dmesg | grep xhci_hcd for more details.

Symptom: "ata1: softreset failed" or "link down"

Likely cause: SATA controller or drive communication failure.

Best first fix: Power down, check both SATA data and power cables, reconnect securely.

Next fix: Try a different SATA port on the motherboard. Update motherboard BIOS/UEFI.

Symptom: "thermal throttling", "CPU package temperature above threshold"

Likely cause: CPU overheating due to poor cooling.

Best first fix: Check that all fans are spinning. Clean dust from heatsinks and vents.

Next fix: Reapply thermal paste between CPU and heatsink. Improve case airflow.

Symptom: "PCIe Bus Error: severity=Corrected" (AER errors)

Likely cause: PCIe device (GPU, NVMe) encountering correctable errors, possibly a bad connection.

Best first fix: Power down, reseat the PCIe card or NVMe drive firmly.

Next fix: Update the device's firmware (if available) and kernel PCIe drivers. Try a different PCIe slot.

Advanced Details

Understanding dmesg Timestamp Formats

Raw dmesg timestamps are in seconds since kernel boot. Use dmesg -T for human-readable local time (requires CONFIG_PRINTK_TIME). For precise microsecond resolution and monotonic timestamps, use dmesg --time-format ctime or journalctl -k which provides more reliable timestamps.

Persistent Journal Storage and Size Limits

Systemd journals are stored in /var/log/journal/. Size is managed by /etc/systemd/journald.conf (parameters: SystemMaxUse=, SystemKeepFree=). If logs are missing after reboot, ensure Storage=persistent is set.

Triggering a Kernel Oops for Driver Testing

You can force a non-fatal kernel oops to test logging: echo c > /proc/sysrq-trigger (requires sysrq enabled). This will create a crash log in dmesg. Use only on a test system.

Interpreting ACPI Errors in Boot Logs

ACPI (Advanced Configuration and Power Interface) errors are common with older hardware or buggy UEFI. They often appear as "ACPI Error: [xxxx]". While sometimes harmless, they can cause sleep/resume issues. A common workaround is adding kernel parameters like acpi=off or acpi=noirq in the bootloader, but this disables power management.

Using Ftrace for Dynamic Kernel Tracing

For deep driver/IRQ issues, the kernel's Ftrace can trace function calls. Enable via /sys/kernel/debug/tracing. Example: echo function > /sys/kernel/debug/tracing/current_tracer. This is a powerful debugging tool but requires in-depth kernel knowledge.

Prevention & Best Practices

Regularly update your kernel and system firmware to benefit from hardware bug fixes.
Monitor SMART attributes of hard drives periodically to predict failures.
Keep your system clean and ensure proper ventilation to prevent overheating.
Use high-quality, shielded cables for storage and peripherals.
Employ a UPS (Uninterruptible Power Supply) to protect against power surges and brownouts.
Configure log rotation (logrotate) and monitoring for /var/log/kern.log.
Before installing new hardware, check kernel module compatibility (look for DKMS support).
Set up alerting for critical kernel messages using tools like logwatch or systemd's journal alerts.
Test new kernel versions on a non-critical system or VM before deploying widely.
Document any persistent non-critical kernel errors and their workarounds for future reference.
Ensure your system's BIOS/UEFI is updated to the latest stable version from the manufacturer.
Use ECC (Error-Correcting Code) memory for servers and critical workstations to catch memory errors early.

Frequently Asked Questions

What's the difference between dmesg and /var/log/kern.log?

dmesg reads directly from the kernel ring buffer in memory (current boot). /var/log/kern.log is a file where the rsyslog or journald service writes kernel messages persistently, potentially across reboots.

My dmesg output is empty. What's wrong?

The ring buffer might have been cleared, or you may lack read permissions. Use sudo dmesg. Also, check if kernel logging is disabled via sysctl kernel.printk (should not be all zeros).

Are all kernel errors serious?

No. Some are corrected by the kernel (e.g., corrected PCIe AER errors) and are informational. Repeated uncorrected errors, hardware failures (I/O, memory), and panics are serious.

How can I save dmesg output to a file?

Use redirection: dmesg > dmesg_dump.txt. For human-readable timestamps: dmesg -T > dmesg_timed.txt.

What does "kernel: tainted" mean?

A "tainted" kernel means a proprietary or out-of-tree module has been loaded, or the system experienced a critical error. This flags the kernel for support purposes but usually doesn't affect functionality.

Can I clear the kernel ring buffer?

Yes, with sudo dmesg -c. This clears the buffer in RAM but not persistent log files. Useful for isolating new errors after a change.

How do I search for messages from a specific driver?

Use dmesg | grep -i driver_name. For module-related messages, also check lsmod to see the exact module name.

When should I stop and ask a professional?

If you see recurring, severe hardware errors (memory, disk) that persist after basic fixes, or if the system is unstable and you lack backup hardware for testing. Also, if you are dealing with mission-critical production servers, involve a system administrator or hardware technician early.

References / Sources

The Linux kernel documentation: Linux Kernel User's and Administrator's Guide (admin-guide).
Systemd documentation: journalctl man pages and systemd-journald.service details.
Ubuntu and Debian documentation: Kernel Logs and Debugging Hardware sections.
Red Hat Enterprise Linux documentation: System Logging and Viewing Kernel Ring Buffer.
The dmesg(1), journalctl(1), and syslog(2) man pages.
SMART (Self-Monitoring, Analysis and Reporting Technology) standards documentation.
ACPI Specification (from the UEFI Forum) for understanding ACPI error messages.
EDAC (Error Detection and Correction) kernel subsystem documentation.

Menu

How to read kernel logs for hardware issues

AI Assistant

Was this tool helpful?