WaitChannels - FreeBSD Wiki

What is a Wait Channel?

Threads in the kernel can wait for resources using various APIs such as tsleep() or cv_wait() . The resource being waited on is associated with a pointer in the kernel's virtual address space. These pointers are known as wait channels. In addition to the raw pointer itself, each thread stores a reference to a short string describing the wait channel. This string is known as a wait message.

Wait Channels of Active Threads

There are various ways of determining which wait channel a thread is waiting on. The raw pointer can be displayed via the "nwchan" keyword for ps(1) :

> ps O nwchan
  PID NWCHAN           TT  STAT    TIME COMMAND
 6515 -                v0  Z    0:00.19 <defunct>
 6517 fffff8000deffa18 v0  IW   0:00.00 -tcsh (tcsh)
 6545 fffff8000dbea970 v0  IW+  0:00.00 /bin/sh /usr/local/bin/startx
...

The wait message is available via the "wchan" keyword:

> ps O wchan
  PID WCHAN  TT  STAT    TIME COMMAND
 6515 -      v0  Z    0:00.19 <defunct>
 6517 pause  v0  IW   0:00.00 -tcsh (tcsh)
 6545 wait   v0  IW+  0:00.00 /bin/sh /usr/local/bin/startx
...

The ps(1) command also includes another keyword called "mwchan". This keyword displays the wait message for a thread blocked on a wait channel. For a thread blocked on a mutex , read-mostly lock or read/write lock , the name of the lock is displayed after a "*" prefix. Note that other locks such as lockmanager and shared/exclusive locks are implemented using wait channels, so threads blocked on one of those locks display the lock name as the wait message without a "*" prefix.

In addition, the STATE column in top(1) displays the same data as the "mwchan" keyword for threads blocked on either a wait channel or a lock. Wait messages generally use lower-case letters in contrast to the non-waiting status messages displayed in this column such as CPU x , RUN, or WAIT.

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root       155 ki31     0K    64K CPU3    3 275.1H  92.09% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     2 273.6H  89.45% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     0 273.6H  87.89% idle{idle: cpu
   11 root       155 ki31     0K    64K RUN     1 273.6H  86.77% idle{idle: cpu
 6692 john        39    0   445M   119M CPU1    1 267:50  22.07% akonadi_imap_r
 6637 john        33    0   217M 73660K CPU2    2 158:27  15.19% akonadiserver{
 6638 john        27    0   242M   153M sbwait  0 204:18   7.86% mysqld{mysqld}
 6692 john        24    0   445M   119M select  0   3:22   4.49% akonadi_imap_r
 6559 john        21    0  3330M   101M select  3 127:07   0.88% Xorg
 6622 john        20    0   613M 93880K select  3  85:02   0.29% kwin{kwin}
 6629 john        20    0   849M   154M select  3  49:13   0.10% kdeinit4{kdein
 6638 john        20    0   242M   153M sbwait  2 219:54   0.00% mysqld{mysqld}
 7546 john        20    0   505M 86916K select  3  48:41   0.00% kdeinit4{kdein
   12 root       -96    -     0K   320K WAIT    3  42:18   0.00% intr{irq264: v
 1765 root        20    0 28876K  4180K select  1  37:53   0.00% hald-addon-mou
    8 root        16    -     0K    16K syncer  3  32:32   0.00% syncer
 6692 john        20    0   445M   119M select  3  17:17   0.00% akonadi_imap_r

When a process is sent SIGINFO, the kernel outputs the wait channel of the "most interesting" thread in the process on the terminal. Shells send SIGINFO to the running process when Ctrl+T is pressed:

> cat
load: 0.07  cmd: cat 44115 [ttyin] 0.67r 0.00u 0.00s 0% 1532k

The procstat(1) utility can also be used to query the wait channel of processes either via its default output or via the -t flag:

> procstat -t `pgrep firefox`
  PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN    
16864 100646 firefox          -                  1  120 sleep   select    
16864 100723 firefox          Gecko_IOThread     0  124 sleep   kqread    
16864 100724 firefox          Socket Thread      1  120 sleep   select    
16864 100725 firefox          JS GC Helper       0  120 sleep   uwait     
16864 100726 firefox          JS Watchdog        0  120 sleep   uwait     
16864 100727 firefox          Hang Monitor       3  122 sleep   uwait     
16864 100728 firefox          Analysis Helper    2  120 sleep   uwait     
...

In the in-kernel debugger, DDB , wait channels for threads are displayed via the ps , show proc , and show thread commands.

In kgdb(1) , the wait message of a given thread can be found by the string pointed to by the td_wmesg field of the associated thread structure. The wait channel pointer can be found via the td_wchan field.

Finding Wait Channels

Wait channels are associated with wait messages in the kernel source code. To determine the meaning of an unfamiliar wait channel, use a tool like grep(1) to search for the wait message string in the kernel sources. Once you have found the relevant string, examine the nearby code to determine what the thread is waiting for. For example, the "biord" message is used here:

> grep -4Hr biord /sys
/sys/kern/vfs_bio.c-int
/sys/kern/vfs_bio.c-bufwait(struct buf *bp)
/sys/kern/vfs_bio.c-{
/sys/kern/vfs_bio.c-    if (bp->b_iocmd == BIO_READ)
/sys/kern/vfs_bio.c:            bwait(bp, PRIBIO, "biord");
/sys/kern/vfs_bio.c-    else
/sys/kern/vfs_bio.c-            bwait(bp, PRIBIO, "biowr");
/sys/kern/vfs_bio.c-    if (bp->b_flags & B_EINTR) {
/sys/kern/vfs_bio.c-            bp->b_flags &= ~B_EINTR;

Common Wait Channels

This is a table of some of the common wait channel messages one might see in a system and a brief description of what state the associated thread is in (for example, which resource a thread is waiting on). Note that this list is not exhaustive.

Wait Channel	Description
biord	Waiting for a disk read operation to complete
biowr	Waiting for a disk write operation to complete
nfsreq	Waiting for a reply to an NFS request
wdrain	Waiting for pending writes to a file to drain to disk
flushbuf	Waiting for pending writes to a file to drain to disk so the pages can be flushed from RAM via posix_fadvise(2)
sbwait	Waiting for data to arrive on a socket, e.g. a blocking read(2) on a socket
kqread	Waiting for an event to arrive on a kqueue
select	Waiting for an event to trigger on a file descriptor via select(2) or poll(2) , usually a socket
piperd	Waiting for data to arrive on a pipe, i.e. a blocking read(2)
pipewr	Waiting for room to write data into a pipe, i.e. a blocking write(2)
fifoor	Waiting in a blocking read-only open of a FIFO (O_RDONLY) for another process to open a writable descriptor on this FIFO (O_RDWR or O_WRONLY)
fifoow	Waiting in a blocking write-only open of a FIFO (O_WRONLY) for another process to open a readable descriptor on this FIFO (O_RDWR or O_RDONLY)	Waiting for a child process to exit, e.g. a shell waiting for a command to complete
ttyin	Waiting to read data from a terminal, e.g. idle shell for an ssh session or xterm
ttyout	Waiting to write data to a terminal
pause	Waiting for a signal (tcsh uses this instead of "wait")
nanslp	Waiting for a specified amount of time to pass, e.g. "sleep 1"
ucond	Waiting for a userland condition variable (pthread_cond_wait())
getblk	Waiting to access file data that is being accessed by another thread	Waiting to read from a file or directory on a UFS filesystem that is being written to by another thread, or waiting to write to a local file or directory that is being read or written by another thread	Same as "ufs" but for an NFS file or directory. Often if you see this it's because the other thread is blocked in "nfsreq".	Same as "ufs" but for a ZFS file or directory.

WaitChannels (last edited 2014-08-29T15:14:40+0000 by JohnBaldwin )