Today's Outline

• bottom halves
• linux kernel time

Bottom Halves in Linux

• softirqs: fixed at kernel compile time, limited number, more time critical
• tasklets: more flexible, created statically or dynamically
• kernel timers: execute at a given time (or later)
• work queues: run in process context, may sleep

SoftIRQs

• currently 6:
  1. high priority tasklets
  2. timer
  3. network transmit
  4. network receive
  5. SCSI
  6. regular tasklets
• raise_soft_irq(SOFTIRQ_ID) called by interrupt handler (or any other code, including the soft IRQ itself) marks the soft IRQ pending
• pending softIRQs executed:
  • after a hardware interrupt
  • by the kernel thread softirqd
  • by any code (e.g. networking code) that calls do_softirq()
• only interrupt handlers can pre-empt a soft IRQ, but multiple soft IRQ handlers can run concurrently on separate processors
• softirqd runs with low priority (nice 19), so gives fast response on a lightly-loaded system, but does not starve user processes on a heavily loaded system

Tasklets

• tasklet_schedule(&tasklet) called by interrupt handler (or any other code) marks the tasklet as pending
• pending tasklets executed whenever softIRQs are executed
• like soft IRQs, a tasklet executes at most once for each call to do_softirq()
• cannot sleep, because called by a soft IRQ handler
• unlike soft IRQs, a tasklet never runs concurrently with itself: not reentrant, so less synchronization may be needed

Kernel Timers

• also executed by a soft IRQ handler, so similar to a tasklet
• TIMER_SOFTIRQ raised by the timer interrupt handler
• add_timer() to start a timer
• mod_timer() to change the expiration date of a timer
• del_timer() to delete a timer
• del_timer_sync() to delete a timer and wait until any currently executing timer handler for that timer has completed
• kernel keeps timers approximately sorted (in 5 bins) to make it easier to determine which timer(s) has/have expired

Work Queues

• potentially many work queue kernel threads, including the default events thread
• work is created by specifying a function and data for that function to work on
• work is then assigned to a kernel thread (via schedule_work() for the events thread or queue_work for an arbitrary thread)
• these functions execute their work queue in order (and sleep when idle -- see the Sleeping Barber model in Tanenbaum)
• schedule_delayed_work () and queue_delayed_work () can also specify a (minimum) delay (in timer ticks)
• kernel thread: has a current process, but has no user space, can sleep (but probably unwise to make the events thread sleep too long)

Hardware Timers

• most architectures have two timers:
  • a real time clock, RTC, which keeps track of time even when the computer is off
  • a system timer, which interrupts on a periodic basis. The exact period can be programmed in software
• on a PC there are other timers, but these two are the main timers used in Linux

Jiffies and HZ

• the system timer interrupts HZ times per second: for example, if HZ == 1000, the system timer interrupts once every millisecond, whereas if HZ == 100, the system timer interrupts once every 10ms
• the time between system timer interrupts is 1 jiffy
• the variable jiffies keeps track of time since boot:
  • 64 bits are maintained
  • most programs only read the lower 32 bits
  • time_before, time_after, time_before_eq, and time_after_eq provide time comparisons that work accurately with the lower 32 bits
  if HZ == 1000, jiffies wraps around in about 49 days

Timer Interrupt Handler

• hardware dependent part:
  1. acquire the xtime_lock
  2. ack or reset the system timer
  3. save the system time to the RTC (only once every n calls)
  4. call the hardware independent part, do_timer()
  5. release the xtime_lock
• hardware independent part, do_timer():
  1. increment jiffies_64, which also increments jiffies
  2. update resource usage for the current process (system time if in a system call, user time otherwise) -- note this is a very rough sampling way of accounting for process execution
  3. run expired kernel timers
  4. call scheduler_tick (), which decrements the timeslice and may pre-empt the current process
  5. update the wall time
  6. compute the load average

Timer Code Synchronization

• the system time, variable xtime, has many readers and few writers
• a seq_lock combines a lock and a counter:
  • lock and increment the counter before writing
  • increment the counter and unlock after writing
  • check counter before and after reading: if they are the same and even, the value read is OK
  • if counters are odd or have changed during reading, read again
• synchronization is needed because reading multiple shared variables (e.g. the seconds and nanoseconds of xtime) is only reasonable if none of them has changed

Example system clock race condition

• this is what might happen if access to the clock were not synchronized:
• clock interrupt handler:

    xtime.tv_nsec += 1000000000 / HZ;
    while (xtime.tv_nsec >= 1000000000) {
      xtime.tv_sec++;
      xtime.tv_nsec -= 1000000000;
    }
  

• gettimeofday system call:

    return_value->tv_sec = xtime.tv_sec;
    /* timer interrupt here will lead to incorrect return value.
       for example, successive reads may return such times as
       10.999s, 10.001s, 11.001s, .... */
    return_value->tv_usec = xtime.tv_nsec / 1000;
  

• similar things (e.g. the clock incrementing from 10.999s to 11.999s to 11.001s) can happen if gettimeofday reads the values in the opposite order
• since time should never go back, it is important to use the seq_lock to repeat the reading until the time can be read in a stable manner
• see Love, p. 153, which also compensates for the case where jiffies has been updated but wall_jiffies has not yet been updated