The recursive nature of Divide & Conquer leads to recurrences, or functions defined in terms of:
Reviewing from Topic #2, a common (but not the only) form of recurrence is as follows. Let T(n) be the running time on a problem of size n.
Then the total time to solve a problem of size n can be expressed as:
Some technical points should be made:
Today we cover three approaches to solving such relations: substitution, recursion tree, and the master method. But first, we look at two examples, one of which we have already seen ...
Sort an array A[p .. r] of comparable elements recursivly by divide and conquer:
We have seen in Topic 2 that this has the following recurrence (please review Topic 2 if you don't see why):
Suppose you have an array of numbers and need to find the subarray with the maximum sum of elements in the subarray. (The problem is trival unless there are negative numbers involved.)
The book provides an application to a simplified investment problem, with the unrealistic assumption that we can predict stock prices. There are real applications to graphics (the 2D version is finding the brightest spot in an image).
The following algorithm is not the fastest known (a linear solution exists), but it illustrates divide and conquer. The solution strategy, given an array A[low .. high], is:
The strategy works because any subarray must lie in one of these three positions:
Recursion will handle the lower and upper halves. The algorithm relies on a helper to find the crossing subarray. Any maximum subarray crossing the midpoint must include arrays ending at A[mid] and starting at A[mid+1]:
Therefore the pseudocode finds the maximum array on each side and adds them up:
Find-Max-Crossing-Subarray returns multiple values (using the syntax return(x,y,z)), and these are recorded with multiple value assignment (using the syntax (x, y, x) = ... ). It should be clear that Find-Max-Crossing-Subarray is Θ(n). The recursive solution follows.
Check your understanding: Where is the work done? What adds up the values in the left and right subarrays?
The analysis relies on the simplifying assumption that the problem size is a power of 2 (the same assumption for merge sort). Let T(n) denote the running time of FIND-MAXIMUM-SUBARRAY on a subarray of n elements.
The resulting recurrence is the same as for merge sort:
So how do we solve these? We have three methods: Substitution, Recursion Trees, and the Master Method.
Don't you love it when a "solution method" starts with ...
Recursion trees (next section) are one way to guess solutions. Experience helps too. For example, if a problem is divided in half we may expect to see lg n behavior.
As an example, let's solve the recurrence for merge sort and maximum subarray. We'll start with an exact rather than asymptotic version:
Induction would require that we show our solution holds for the boundary conditions. This is discussed in the textbook.
Normally we use asymptotic notation rather than exact forms:
If we want Θ, sometimes we can prove big-O and Ω separately "squeezing" the Θ result.
But be careful when using asymptotic notation. For example, suppose you have the case where a=4 and b=4 and want to prove T(n) = O(n) by guessing that T(n) ≤ cn and writing:
One must prove the exact form of the inductive hypothesis, T(n) ≤ cn.
These web notes don't adequately cover the nuances of substitution proofs of asymptotic complexity. See the text for other strategies and pitfalls.
Problems 4.3-1 and 4.3-2 are good practice problems.
Although recursion trees can be considered a proof format, they are normally used to generate guesses that are verified by substitution.
We have already seen recursion trees when analyzing the recurrence relations for Merge Sort:
The subproblems are of size n/20, n/21, n/22, .... The tree ends when n/2p = n/n = 1, the trivial subproblem of size 1.
Thus the height of the tree is the power p to which we have to raise 2 before it becomes n, i.e., p = lg n. Since we start at 20 there are lg n + 1 levels. Multiplying by the work cn at each level, we get cn lg n + cn for the total time.
A more complex example is developed in the textbook for
T(n) = 3T(n/4) + Θ(n2)
which is rewritten (making the implied constant explicit) as
T(n) = 3T(n/4)+ cn2
We can develop the recursion tree in steps, as follows. First, we begin the tree with its root node T(n):
Now let's branch the tree for the three recursive terms 3T(n/4). There are three children nodes with T(n/4) as their cost, and we leave the cost cn2 behind at the root node.
We repeat this for the subtrees rooted at each of the nodes for T(n/4): Since each of these costs 3T((n/4)/4) +c(n/4)2, we make three branches, each costing T((n/4)/4) = T(n/16), and leave the c(n/4)2 terms behind at their roots.
Continuing this way until we reach the leaf nodes where the recursion ends at trivial subproblems T(1), the tree looks like this:
Subproblem size for a node at depth i is n/4i, so
the subproblem size reaches n = 1 when (assuming n a power of 4)
n/4i = 1, or when i = log4n.
Including i = 0, there are log4n + 1 levels. Each level has 3i nodes.
Substituting i = log4n into 3i, there are 3log4n nodes in the bottom level.
Using alogbc = clogba, there are nlog43 in the bottom level (not n, as in the previous problem).
Adding up the levels, we get:
It is easier to solve this summation if we change the equation to an inequality and let the
summation go to infinity (the terms are decreasing geometrically), allowing us to apply equation
A.6 (∑k=0,∞xk = 1/1-x):
Additional observation: since the root contributes cn2, the root dominates the cost of the tree, and the recurrence must also be Ω(n2), so we have Θ(n2).
Please see the text for an example involving unequal subtrees. For practice, exercises 4.4-6 and 4.4-9 have solutions posted on the book's web site.
If we have a divide and conquer recurrence of the form
T(n) = aT(n/b) + f(n)
where a ≥ 1, b > 1, and f(n) > 0 is asymptotically positive,
then we can apply the master method, which is based on the master theorem. We compare f(n) to nlogba under asymptotic (in)equality:Case 1: f(n) = O(nlogba - ε) for some constant ε > 0.
Important: there are functions that fall between the cases!
T(n) = 5T(n/2) + Θ(n2)
T(n) = 27T(n/3) + Θ(n3 lg n)
T(n) = 5T(n/2) + Θ(n3)
T(n) = 27T(n/3) + Θ(n3 / lg n)
Chapter 12, Binary Search Trees (entire chapter), to which we can apply divide & conquer and use recurrence relations.