SOFTIRQSLOWER(8)

NAME
softirqslower - Trace softirq handlers with latency exceeding a threshold

SYNOPSIS
softirqslower.py [min_us]
softirqslower.py [-c CPU] [min_us]

DESCRIPTION
The softirqslower tool traces softirq events that exceed a specified latency threshold. This tool helps diagnose interrupt handling bottlenecks by measuring two critical latency dimensions:

•   irq(hard) to softirq: Time delay between hardware interrupt completion and softirq execution start

•   softirq runtime: Actual execution duration of softirq handlers

Developed as part of the BCC (BPF Compiler Collection) tools, it's particularly useful for network stack optimization, scheduler tuning, and real-time systems analysis.

OPTIONS

min_us
    Minimum latency threshold to trace (in microseconds). Default: 10,000 μs (10 ms)

-c, --cpu CPU
    Trace only events on the specified CPU core

OUTPUT FORMAT
The tool outputs events in the following table format:

TIME STAGE SOFTIRQ LAT(us) CPU COMM

With fields defined as:

TIME
    Event timestamp (HH:MM:SS format)

STAGE
    Latency measurement type:

    irq(hard) to softirq
        Time from hardware interrupt completion to softirq dispatch

    softirq runtime
        SoftIRQ handler execution duration

SOFTIRQ
    SoftIRQ category (case-sensitive). Common values:

    NET_RX
        Network reception processing

    NET_TX
        Network transmission processing

    TIMER
        Timer callbacks

    SCHED
        Scheduler operations

    RCU Read-Copy-Update synchronization

    TASKLET
        Deferred task execution

    HRTIMER
        High-resolution timers

    BLOCK
        Block device operations

LAT(us)
    Measured latency in microseconds

CPU CPU core where softirq was handled (0-based numbering)

COMM
    Process context handling the softirq:

    swapper/N
        Idle thread for CPU N

    ksoftirqd/N
        Softirq daemon for CPU N

    <process>
        User-space process

EXAMPLES

Trace softirqs exceeding 10μs latency:
    softirqslower 10

Monitor only CPU core 1:
    softirqslower -c 1

Trace network-related softirq delays (>50μs):
    softirqslower 50 | grep -E 'net_rx|net_tx'

Capture RCU delays longer than 100μs to file:
    softirqslower 100 | grep rcu > rcu_latency.log

SIGNALS
Ctrl+C Stop tracing and exit gracefully

AUTHOR
Chenyue Zhou

SEE ALSO
runqslower(8), hardirqs(8), softirqs(8), trace(8), funclatency(8),

REPORTING BUGS
BCC Tools Issue Tracker: https://github.com/iovisor/bcc/issues