ICS 311 Topic #3: Growth of Functions and Asymptotic Concepts


Prerequisites review

Outline

  1. Intro to Asymptotic Analysis
  2. Big-O
  3. Ω (Omega)
  4. Θ (Theta)
  5. Asymptotic Notation in Equations
  6. Asymptotic Inequality
  7. Properties of Asymptotic Sets
  8. Common Functions

Readings and Screencasts


Intro to Asymptotic Analysis

The notations discussed today are ways to describe behaviors of functions, particularly in the limit, or asymptotic behavior.

The functions need not necessarily be about algorithms, and indeed asymptotic analysis is used for many other applications.

Asymptotic analysis of algorithms requires:

  1. Identifying what aspect of an algorithm we care about, such as:
  2. Identifying a function that characterizes that aspect; and

  3. Identifying the asymptotic class of functions that this function belongs to, where classes are defined in terms of bounds on growth rate.

The different asymptotic bounds we use are analogous to equality and inequality relations:

In practice, most of our analyses will be concerned with run time. Analyses may examine:


Big-O (asymptotic ≤)

Our first question about an algorithm's run time is often "how bad can it get?" We want a guarantee that a given algorithm will complete within a reasonable amount of time for typical n expected. This requires an asymptotic upper bound: the "worst case".

Big-O is commonly used for worst case analyses, because it gives an upper bound on growth rate. Its definition in terms of set notation is:

O(g(n)) = {f(n) : ∃ positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) ∀ nn0}.

This definition means that as n increases, afer a point f(n) grows no faster than g(n) (as illustrated in the figure): g(n) is an asymptotic upper bound for f(n).

Since O(g(n)) is a set, it would be natural to write f(n) ∈ O(g(n)) for any given f(n) and g(n) meeting the definition above, for example, f ∈ O(n2).

But the algorithms literature has adopted the convention of using = instead of ∈, for example, writing f(n) = O(g(n)). This "abuse of notation" makes some manipulations possible that would be more tedious if done strictly in terms of set notation. (We do not write O(g(n))=f(n); will return to this point).

Using the = notation, we often see definitions of big-O in in terms of truth conditions as follows:

f(n) = O(g(n)) iff ∃ positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) ∀ nn0.

We assume that all functions involved are asymptotically non-negative. Other authors don't make this assumption, so may use |f(n)| etc. to cover negative values. This assumption is reflected in the condition 0 ≤ f(n).

Examples

Show that 2n2 is O(n2).

To do this we need to show that there exists some c and n0 such that (letting 2n2 play the role of f(n) and n2 play the role of g(n) in the definition):

0 ≤ 2n2cn2 for all nn0.

It works with c = 2, since this makes the f and g terms equivalent for all nn0 = 0. (We'll do a harder example under Θ.)

What's in and what's out

These are all O(n2): These are not:
  • n2
  • n2 + 1000n
  • 1000n2 + 1000n
  • n1.99999
  • n
  • n3
  • n2.00001
  • n2 lg n

Insertion Sort Example

Recall that we did a tedious analysis of the worst case of insertion sort, ending with this formula:

T(n) can be expressed as pn2 + qn - r for suitable p, q, r (p = (c5/2 + c6/2 + c7/2), etc.).

The textbook (page 46) sketches a proof that f(n) = an2 + bn + c is Θ(n2), and we'll see shortly that Θ(n2) → O(n2). This is generalized to all polynomials in Problem 3-1. So, any polynomial with highest order term and (i.e., a polynomial in n of degree d) will be O(nd).

This suggests that the worst case for insertion sort T(n) ∈ O(n2). An upper bound on the worst case is also an upper bound on all other cases, so we have already covered those cases.

Notice that the definition of big-O would also work for g(n) = n3, g(n) = 2n, etc., so we can also say that T(n) (the worst case for insertion sort) is O(n3), O(2n), etc. However, these loose bounds are not very useful! We'll deal with this when we get to Θ (Theta).


Ω (Omega, asymptotic ≥)

We might also want to know what the best we can expect is. In the last lecture we derived this formula for insertion sort:

We could prove that this best-case version of T(n) is big-O of something, but that would only tell us that the best case is no worse than that something. What if we want to know what is "as good as it gets": a lower bound below which the algorithm will never be any faster?

We must both pick an appropriate function to measure the property of interest, and pick an appropriate asymptotic class or comparison to match it to. We've done the former with T(n), but what should it be compared to?

It makes more sense to determine the asymptotic lower bound of growth for a function describing the best case run-time. In other words, what's the fastest we can ever expect, in the best case?

Ω (Omega) provides what we are looking for. Its set and truth condition definitions are simple revisions of those for big-O:

Ω(g(n)) = {f(n) : ∃ positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) ∀ nn0}.
[The f(n) and cg(n) have swapped places.]
f(n) = Ω(g(n)) iff ∃ positive constants c and n0 such that f(n) ≥ cg(n) ∀ nn0.
[≤ has been replaced with ≥.]

The semantics of Ω is: as n increases after a point, f(n) grows no slower than g(n) (see illustration).

Examples

Sqrt(n) is Ω(lg n) with c=1 and n0=16.
(At n=16 the two functions are equal; try at n=64 to see the growth, or graph it.)

What's In and What's Out

These are all Ω(n2): These are not:
  • n2
  • n2 + 1000n   (It's also O(n2)!)
  • 1000n2 + 1000n
  • 1000n2 - 1000n
  • n3
  • n2.00001
  • n1.99999
  • n
  • lg n

Insertion Sort Example

We can show that insertion will take at least Ω(n) time in the best case (i.e., it won't get any better than this) using the above formula and definition.

T(n) can be expressed as pn - q for suitable p, q (e.g., q = c2 + c4 + c5 + c8, etc.). (In this case, p and q are positive.) This suggests that T(n) ∈ Ω(n), that is, ∃ c, n0 s.t. pn - q ≥ cn,n ≥ n0. This follows from the generalized proof for polynomials.


Θ (Theta, asymptotic =)

We noted that there are loose bounds, such as f(n) = n2 is O(n3), etc., but this is an overly pessimistic assessment. It is more useful to have an asymptotically tight bound on the growth of a function. In terms of algorithms, we would like to be able to say (when it's true) that a given characteristic such as run time grows no better and no worse than a given function. That is, we want to simultaneoulsy bound from above and below. Combining the definitions for O and Ω:

Θ(g(n)) = {f(n) : ∃ positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n), ∀ nn0}.

As illustrated, g(n) is an asymptotically tight bound for f(n): after a point, f(n) grows no faster and no slower than g(n).

The book suggests the proof of this theorem as an easy exercise (just combine the two definitions):

f(n) = Θ(g(n)) iff f(n) = Ω(g(n)) ∧ f(n) = O(g(n)).

You may have noticed that some of the functions in the list of examples for big-O are also in the list for Ω. This indicates that the set Θ is not empty.

Examples

Reminder: f(n) = Θ(g(n)) iff ∃ positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n)∀ nn0.

n2 - 2n is Θ(n2),   with c1 = 1/2; c2 = 1, and n0 = 4,   since:

n2/2   ≤   n2 - 2n   ≤   n2   for nn0 = 4.

Find an asymptotically tight bound (Θ) for

Please try these before you find solutions here.

What's in and what's out

These are all Θ(n2): These are not
  • n2
  • n2 + 1000n
  • 1000n2 + 1000n + 32,700
  • 1000n2 - 1000n - 1,048,315
  • n3
  • n2.00001
  • n1.99999
  • n lg n


Asymptotic Inequality

The O and Ω bounds may or may not be asymptotically tight. The next two notations are for upper bounds that are strictly not asymptotically tight. There is an analogy to inequality relationships:

o-notation ("little o", asymptotic <)

o(g(n)) = {f(n) : ∀ constants c > 0, ∃ constant n0 > 0 such that 0 ≤ f(n) < cg(n) ∀ nn0}.

Alternatively, f(n) becomes insignificant relative to g(n) as n approaches infinity (see box):

We say that f(n) is asymptotically smaller than g(n) if f(n) = o(g(n))

ω-notation ("little omega", asymptotic >)

ω(g(n)) = {f(n) : ∀ constants c > 0, ∃ constant n0 > 0 such that 0 ≤ cg(n) < f(n) ∀ nn0}.

Alternatively, f(n) becomes arbitrarily large relative to g(n) as n approaches infinity (see box):

We say that f(n) is asymptotically larger than g(n) if f(n) = ω(g(n))

The two are related:   f(n) ∈ ω(g(n))   iff   g(n) ∈ o(f(n)).


Asymptotic Notation in Equations

We already noted that while asymptotic categories such as Θ(n2) are sets, we usually use "=" instead of "∈" and write (for example) f(n) = Θ(n2) to indicate that f is in this set.

Putting asymptotic notation in equations lets us do shorthand manipulations during analysis.

Asymptotic Notation on Right Hand Side: ∃

O(g(x)) on the right hand side stands for some anonymous function in the set O(g(x)).

2n2 + 3n + 1 = 2n2 + Θ(n)     means:
2n2 + 3n + 1 = 2n2 + f(n)       for some f(n) ∈ Θ(n)     (in particular, f(n) = 3n + 1).

Asymptotic Notation on Left Hand Side: ∀

The notation is only used on the left hand side when it is also on the right hand side.

Semantics: No matter how the anonymous functions are chosen on the left hand side, there is a way to choose the functions on the right hand side to make the equation valid.

2n2 + Θ(n) = Θ(n2)     means   for all f(n) ∈ Θ(n), there exists g(n) ∈ Θ(n2) such that
2n2 + f(n) = g(n).

Combining Terms

We can do basic algebra such as:

2n2 + 3n + 1   =   2n2 + Θ(n)   =   Θ(n2)

Properties

If we keep in mind the analogy to inequality, many of these make sense, but see the end for a caution concerning this analogy.

Relational Properties

Transitivity:
Reflexivity:
Symmetry:
Transpose Symmetry:

Incomparability

Here is where the analogy to numeric (in)equality breaks down: There is no trichotomy. Unlike with constant numbers, we can't assume that one of <, =, > hold. Some functions may be incomparable.

Example: n1 + sin n is incomparable to n since sin n oscillates between -1 and 1, so 1 + sin n oscillates between 0 and 2. (Try graphing it.)


Common Functions and Useful Facts

Various classes of functions and their associated notations and identities are reviewed in the end of the chapter: please review the chapter and refer to ICS 241 if needed. Here we highlight some useful facts:

Monotonicity

Polynomials

Exponentials

Logarithms

Factorials

Iterated Functions

Fibonacci Numbers


Nodari Sitchinava (based on material by Dan Suthers)
Images are from the instructor's manual for Cormen et al.