Today's Outline

interrupt handling in Linux
bottom halves
linux kernel time

Windows System Implementation

a window system provides an API for programs to perform graphic operations within an area of the screen
the results of the operations may be partially visible or invisible
in (the original) Microsoft Windows, a process is connected to a window (or an icon), hence the window system is part of the operating system
in Unix, processes can run independently of windows, so the windowing system is a process of its own, the X server

Interrupt Handlers

when a device needs attention
it causes the execution of an interrupt handler or interrupt service routine (ISR)
the operating system assures that the interrupt handler called is from the driver for the interrupting device

Identifying handlers and devices: IRQ

on the x86 system, each device is assigned a unique Interrupt ReQuest (IRQ) number, either statically, or at boot time
see /proc/interrupts for a typical list
the IRQ may be shared among multiple handlers
if an IRQ is shared, all handlers are called, and each handler must decide whether it needs to do something
this is done by checking its device to see if it interrupted
the interrupt handler can check the device ID (passed in as parameter) so a single function can be used to handle multiple interrupts

Interrupt Context

a reentrant function can be called by several threads at the same time without error
Linux interrupt handlers need not be re-entrant -- they will be called one at a time, with their interrupt line disabled (for a fast interrupt handler, with all interrupts disabled but only on the current processor)
the current task is not relevant to the interrupt handler: the handler CANNOT suspend
the handler should not take too long
the handler should not have too many local variable (limited stack)

Interrupt Enabling

Code may need to disable interrupts, do something (critical region), then re-enable interrupts
but if interrupts were already disabled, re-enabling them might be incorrect

use:

local_irq_save (flags);

... /* critical region */

local_irq_restore (flags);

Bottom Half Processing

interrupt handler may have to initiate an action that it cannot afford to complete -- because it may take too long, or require too many resources, e.g. locks
instead, interrupt handler schedules execution of a bottom half
the bottom half runs with interrupts enabled, and may sleep
linux bottom halves:
- softirqs: fixed at kernel compile time, limited number, more time critical
- tasklets: more flexible, created statically or dynamically
- work queues: run in process context, may sleep
softirqs and work queues must be reentrant, tasklets are serialized

Bottom Halves in Linux

softirqs: fixed at kernel compile time, limited number, more time critical
tasklets: more flexible, created statically or dynamically
kernel timers: execute at a given time (or later)
work queues: run in process context, may sleep

SoftIRQs

currently 6:
1. high priority tasklets
2. timer
3. network transmit
4. network receive
5. SCSI
6. regular tasklets
raise_soft_irq(SOFTIRQ_ID) called by interrupt handler (or any other code, including the soft IRQ itself) marks the soft IRQ pending
pending softIRQs executed:
- after a hardware interrupt
- by the kernel thread softirqd
- by any code (e.g. networking code) that calls do_softirq()
only interrupt handlers can pre-empt a soft IRQ, but multiple soft IRQ handlers can run concurrently on separate processors
softirqd runs with low priority (nice 19), so gives fast response on a lightly-loaded system, but does not starve user processes on a heavily loaded system

Tasklets

tasklet_schedule(&tasklet) called by interrupt handler (or any other code) marks the tasklet as pending
pending tasklets executed whenever softIRQs are executed
like soft IRQs, a tasklet executes at most once for each call to do_softirq()
cannot sleep, because called by a soft IRQ handler
unlike soft IRQs, a tasklet never runs concurrently with itself: not reentrant, so less synchronization may be needed

Kernel Timers

also executed by a soft IRQ handler, so similar to a tasklet
TIMER_SOFTIRQ raised by the timer interrupt handler
add_timer() to start a timer
mod_timer() to change the expiration date of a timer
del_timer() to delete a timer
del_timer_sync() to delete a timer and wait until any currently executing timer handler for that timer has completed
kernel keeps timers approximately sorted (in 5 bins) to make it easier to determine which timer(s) has/have expired

Work Queues

potentially many work queue kernel threads, including the default events thread
work is created by specifying a function and data for that function to work on
work is then assigned to a kernel thread (via schedule_work() for the events thread or queue_work for an arbitrary thread)
these functions execute their work queue in order (and sleep when idle -- see the Sleeping Barber model in Tanenbaum)
schedule_delayed_work () and queue_delayed_work () can also specify a (minimum) delay (in timer ticks)
kernel thread: has a current process, but has no user space, can sleep (but probably unwise to make the events thread sleep too long)

Hardware Timers

most architectures have two timers:
- a real time clock, RTC, which keeps track of time even when the computer is off
- a system timer, which interrupts on a periodic basis. The exact period can be programmed in software
on a PC there are other timers, but these two are the main timers used in Linux

Jiffies and `HZ`

the system timer interrupts HZ times per second: for example, if HZ == 1000, the system timer interrupts once every millisecond, whereas if HZ == 100, the system timer interrupts once every 10ms
the time between system timer interrupts is 1 jiffy
the variable jiffies keeps track of time since boot:
- 64 bits are maintained
- most programs only read the lower 32 bits
- time_before, time_after, time_before_eq, and time_after_eq provide time comparisons that work with the lower 32 bits and correclty handle wrap-around
if HZ == 1000, jiffies wraps around in about 49 days

Timer Interrupt Handler

hardware dependent part, e.g. timer_interrupt in arch/i386/kernel/time.c:
1. acquire the xtime_lock (may spin on SMP systems)
2. ack or reset the system timer
3. save the system time to the RTC (only once every n calls)
4. call the hardware independent part, do_timer()
5. release the xtime_lock
hardware independent part, do_timer():
1. increment jiffies_64, which also increments jiffies
2. update resource usage for the current process (system time if in a system call, user time otherwise) -- note this is a very rough sampling way of accounting for process execution
3. run expired kernel timers
4. call scheduler_tick (), which decrements the timeslice and may pre-empt the current process
5. update the wall time
6. compute the load average

Timer Code Synchronization

the system time, variable xtime, has many readers and few writers
a seq_lock combines a lock and a counter:
- lock and increment the counter before writing
- increment the counter and unlock after writing
- check counter before and after reading: if they are the same and even, the value read is OK
- if counters are odd or have changed during reading, read again
synchronization is needed because reading multiple shared variables (e.g. the seconds and nanoseconds of xtime) is only reasonable if none of them has changed

Example system clock race condition

this is what might happen if access to the clock were not synchronized:

clock interrupt handler:

  xtime.tv_nsec += 1000000000 / HZ;      /* increment by one tick */
  while (xtime.tv_nsec >= 1000000000) {  /* roll over any ns > 10^9 */
    xtime.tv_sec++;
    xtime.tv_nsec -= 1000000000;
  }

gettimeofday system call:

  return_value->tv_sec = xtime.tv_sec;
  /* timer interrupt here will lead to incorrect return value.
     for example, successive reads may return such times as
     10.999s, 10.001s, 11.001s, .... */
  return_value->tv_usec = xtime.tv_nsec / 1000;

similar things (e.g. the clock incrementing from 10.999s to 11.999s to 11.001s) can happen if gettimeofday reads the values in the opposite order
since time should never go back, it is important to use the seq_lock to repeat the reading until the time can be read in a stable manner
see Love, p. 153, which also compensates for the case where jiffies has been updated but wall_jiffies has not yet been updated

Today's Outline

Windows System Implementation

Interrupt Handlers

Identifying handlers and devices: IRQ

Interrupt Context

Interrupt Enabling

Bottom Half Processing

Bottom Halves in Linux

SoftIRQs

Tasklets

Kernel Timers

Work Queues

Hardware Timers

Jiffies and HZ

Timer Interrupt Handler

Timer Code Synchronization

Example system clock race condition

Jiffies and `HZ`