Outline

• heaps
• heap storage in arrays
• heap insertion and removal
• priority queues
• priority queues implementation
• Huffman coding
• Huffman trees
• Implementation of Huffman coding

heap reminders

• a heap is a binary tree with two properties:
  • the heap property: each node is less (min heap) or greater (max heap) than both its children
  • a heap is a complete binary tree
• because a heap is a complete binary tree, it can be stored in an array, without any gaps.
• a heap node at position i in the array has:
  • its parent in array element (i - 1) / 2
  • its left child (if any) in array element 2i + 1
  • its right child (if any) in array element 2i + 2

in-class exercises

• which of the following arrays store max heaps, min heaps, or neither?

  index     0   1   2   3   4   5   6   7   8   9
  array 1   0   1   2   0   4   5   6   7   8   9
  array 2   9   8   7   6   5   4   3   2   1   0
  array 3   5   5   5   6   6   6   6   7   7   1
  array 4   9   3   9   2   1   6   7   1   2   1
  array 5   8   7   6   1   2   3   4   2   1   2
  

Heap insertion

• the two heap requirements must be maintained when adding to a heap
• to maintain the complete binary tree property, the new node must be added to the right of all nodes at level d_max
• or, if there already are 2^(d_max) nodes at that level, the new node should be inserted all the way to the left, making the tree deeper by one level
• either way, the new value is inserted in the array just after all elements already in the array
• now the tree is complete, but may not have the heap property
• to check, compare the node with the parent, and swap the two if needed to maintain the heap property
• continue with the parent's parent, all the way to the root if necessary
• now the complete binary tree also obeys the heap property
• in-class exercise: add the value 17 to each of the following two max-heaps, and show the resulting array:

  index     0   1   2   3   4   5   6   7   8   9
  array 1   9   7   8   3   4   5   6   2   1   0
  array 2   17  3   17  2   2   8   5   1   0   3
  

Max heap deletion

• the largest node is at the root of the heap
• that node is deleted, and replaced with the bottommost, rightmost node, which is the last value in the array
• the remaining tree is complete, but may not have the tree property
• if the root node is less than either of its children, it is swapped with the largest of its children
• the operation continues with the new node
• now the complete binary tree also obeys the heap property
• similarly for min heaps
• in-class exercise: remove the top value (9 and 17, respectively) from each of the following two max-heaps, and show the resulting array:

  index     0   1   2   3   4   5   6   7   8   9
  array 1   9   7   8   3   4   5   6   2   1   0
  array 2   17  3   17  2   2   8   5   1   0   3
  

in-class exercises

• insert into a heap the following values:
  1 9 77 14 3 55 66 7 8
• remove from the heap each of the elements in turn

Runtime of heap insertion and removal

• a complete binary tree with i levels has between 2^i-1 and 2ⁱ nodes
• given that n = 2^{log₂ n}, if the complete binary tree has n nodes, the height of the tree is O(log n)
• adding starts at the bottom of the heap, and re-heapifies upward
• removing starts at the top of the heap, and re-heapifies downward
• each re-heapification step involves replacing a node with the largest of its children, so takes constant time: one swap and up to two comparisons
• so, adding a value takes at most O(log n) steps, and so does removing a value
• adding or removing n values takes at most O(nlog n) steps
• values are always removed in sorted order
• so, the heap sort algorithm, where we add n values, then remove them in order, takes time O(nlog n)

priority queues

• the queues studied so far were strictly FIFO
• that means objects were returned in the order inserted
• in the real world, queues often have priorities, e.g.:
  • at airport check-in, there is a special line for first-class passengers
  • if any first-class passengers are waiting, they are handled first
  • only if no first-class passengers are waiting, only then the other passengers can check-in
• likewise in the emergency room at a hospital or after a disaster
• the process of determining which patients should be seen first is called triage (a medical term, not a computer science term)

priority queue implementations

• linked list: objects are inserted in the proper place in the list (linear time), objects are returned from the front of the list (constant time)
• array: objects are inserted in the proper place in the array, with all other objects shifted to make room (linear time), objects are returned from the front of the array, with all other objects shifted down (linear time), or maintained as a circular array (constant time)
• binary search tree: objects are inserted into the binary search tree, using the priority as the key (log or linear), objects are removed from the rightmost node of the tree (log or linear)
• heap: objects are inserted into the heap using the priority as the key (log time), and removed from the top of the heap (log time)

Huffman Trees

• trees seen so far:
  • binary trees
  • binary search trees
  in these trees, each node has data and zero, one, or two children
• Huffman trees are trees where each node has zero or two children, and only leaf nodes have data
• the data in each leaf node is unique

Huffman coding

• suppose you wanted to compress a text file, that is, represent it using the least possible number of bits
• suppose also you want to use a unique string of bits for each character -- that is, we are not going to encode entire words
• it would be good to use fewer bits for characters that occur frequently, and as many bits as needed for other characters
• for example, in the string "hello world", I could use the following encoding:

  letter	bit string
  h	11000
  e	11001
  l	0
  o	10
  ' ' (space)	11010
  w	11011
  r	1110
  d	1111

• in-class exercise: what does the bit string (25 bits) "1100 0100 1111 1101 0111 1101 0111 0" represent?
• note that the above string takes 45 bits if each character is represented with a constant 5 bits/character, which is the smallest number of bits to represent all 26 characters and space
• so Huffman coding can save space!
• it does this by using shorter bit strings to encode more commonly-used characters, and longer bit strings to encode less frequently used characters

algorithm for building a Huffman coding tree

1. make a list of all symbols with their frequencies
2. sort the list so the symbols with the least frequency are in front
3. if the list only has one element, the element is the root of the tree and we are done
4. remove the first two elements from the list and put them into a binary tree
5. add the frequencies of the two subtrees to give the frequency of this binary tree
6. insert this tree in the right place in the sorted list
7. return to step 3

• in-class exercise: do this for the characters in "hello world"

  letter	frequency
  h	1
  e	1
  l	3
  o	2
  ' ' (space)	1
  w	1
  r	1
  d	1

using a Huffman coding tree

• to encode:
  • find the leaf with the next character to encode
  • starting from the root to this leaf, place a 0 bit for each time you take the left subtree, and a 1 bit for each time you take the right subtree
• in-class exercise: use this algorithm and the tree from the last exercise to encode the string "wow"
• can do the above search once for each character, put the results in a table (e.g. a hash table), then use the table to do the actual encoding quickly
• to decode:
  • use each bit to go left (0 bit) or right (1 bit) in the tree
  • if you have reached a leaf, put this character into the result string, then
  • start at the root of the tree with the next bit
• in-class exercise: use this algorithm and the above tree to decode as many bits as possible of the bit string "1100 0100 1101 0111 1101 0111 0"

compression using Huffman coding

• an encoding based on letter frequencies in one string (or a large sample) can be used for encoding many different strings
• if so, a single copy of the table (tree) can be kept, and huffman coding is guaranteed to do no worse than fixed-length encoding
• otherwise, a separate table (tree) is needed for each compression, and the table has to be included in the count of bytes to be stored or transmitted after compression
• in such cases, Huffman coding might actually give a somewhat larger size than the original
• in practice, even including the table, Huffman coding is usually worthwhile for sufficiently long strings in natural languages, which have lots of redundancy and different letter frequencies

an implementation for Huffman trees

• many possible implementations, this is one (textbook, chapter 6.6/8.6)
• use several data structures:
  • a priority queue implemented by a min heap to hold the sorted data
  • each heap element refers to a tree node
  • an interior tree node has no value, but has two children
  • a leaf node has a value, but has no children
  • each node has a frequency of occurrence, which is used as a priority in the queue (low priority returned first)
• begin by computing the frequency of each value, perhaps by using a hash table (can use a perfect hash if the number of values is known in advance, e.g. 128 or 65,536 characters)
• once the frequency of values is known, insert each value into the priority queue, using its frequency as a priority
• remove the front two elements from the queue, create an interior node to refer to these two elements, and insert the interior node back into the queue with, as priority, the sum of the priorities of the two nodes
• once there is only one element left in the queue, this is the huffman tree
• a further step would be to build an encoding table from the tree
• in-class exercise: do this given the string "there is no place like home"