- Amortized Analysis: The General Idea
- Multipop Example
- Aggregate Analysis
- Accounting Method
- Potential Method -
*Optional for 2017, but it's the most powerful method* - Dynamic Table Example (first look)
- Other Examples

- Chapter 17 of Cormen et al. (2015: through 17.2, plus 17.4.1)
- Screencasts 15A, 15B - (also in Laulima)

We have already used *aggregate* analysis several times in this course. For example, when
analyzing the BFS and DFS procedures, instead of trying to figure out how many times their inner
loops

for eachv∈ G.Adj[u]

execute (which depends on the degree of the vertex being processed), we realized that no matter
how the edges are distributed, there are at most |*E*| edges, so in aggregate across all calls
the loops will execute |*E*| times.

But that analysis concerned itself only with the complexity of a single operation. In practice a given data structure will have associated with it several operations, and they may be applied many times with varying frequency.

Sometimes a given operation is designed to pay a larger cost than would otherwise be necessary to enable other operations to be lower cost.

*Example:* Red-black tree insertion. We pay a greater cost for balancing so that future searches
will remain O(lg *n*).

*Another example:* Java Hashtable.

- These grow dynamically when a specified load factor is exceeded.
- Copying into a new table is expensive, but copying is infrequent and table growth makes access operations faster.

It is "fairer" to average the cost of the more expensive operations across the entire mix of operations, as all operations benefit from this cost.

Here, "average" means average cost in the worst case (no probability is involved, which greatly simplifies the analysis).

We will look at three methods. The notes below use Stacks with Multipop to illustrate the methods. See the text for binary counter examples.

(We have already seen examples of aggregate analysis throughout the semseter. We will see examples of amortized analysis later in the semester.)

We already have the stack operations:

`Push(`: O(1) each call, so O(*S, x*)*n*) for any sequence of*n*operations.`Pop(`: O(1) each call, so O(*S*)*n*) for any sequence of*n*operations.

Suppose we add a `Multipop` (this is a generalization of `ClearStack`, which
empties the stack):

The example shows a `Multipop(S,4)` followed by another where *k* ≥ 2.

Running time of `Multipop`:

- Linear in number of
`Pop`operations (one per loop iteration) - Number of iterations of
`while`loop is min(*s*,*k*), where*s*= number of items on the stack - Therefore, total cost = min(
*s*,*k*).

What is the worst case of a sequence of *n* `Push`, `Pop` and
`Multipop` operations?

Using our existing methods of analysis we can identify a *loose bound:*:

- The most expensive operation is
`Multipop`, potentially O(*n*). - Therefore, potentially O(
*n*) over^{2}*n*operations.

We can tighten this loose bound by aggregating the analysis over all *n* operations:

- Each object can only be popped once per time that it is pushed.
- There are at most
*n*`Push`es, so at most n`Pop`s, including those in`Multipop` - Therefore, total cost = O(
*n*) - Averaging over the
*n*operations we get O(1) per operation.

This analysis shows O(1) per operation on average in a sequence of *n* operations without
using any probabilities of operations.

See text for example of aggregate analysis of binary counting. An example of aggregate analysis of dynamic tables is at the end of these notes.

Some of our previous analyses with indicator random variables have been a form of aggregate analysis, e.g., our analysis of the expected number of inversions in sorting, Topic 5, and our analysis of Depth-First Search, Topic 14.

Aggregate analysis treats all operations equally. The next two methods let us give different operations different costs, so are more flexible.

- View the computer as a coin operated appliance that requires one
*cyber-dollar*(CY$) per basic operation. - The banks are wary of making loans these days, so when an operation is to be performed we must have enough cyber-dollars available to pay for it.
- We are permitted to charge some operations more than they actually cost so we can save enough to pay for the more expensive operations.

**Amortized cost** = amount we charge each operation.

This differs from aggregate analysis:

- In aggregate analysis, all operations have the same cost.
- In the accounting method, different operations can have different costs.

When an operation is overcharged (amortized cost > actual cost), the difference is associated
with *specific objects* in the data structure as *credit*.

We use this credit to pay for operations whose actual cost > amortized cost.

The credit must never be negative. Otherwise the amortized cost may not be an upper bound on actual cost for some sequences.

Let

*c*= actual cost of_{i}*i*th operation.*ĉ*= amortized cost of_{i}*i*th operation*(notice the 'hat')*.

Require ∑_{i=1,n}* ĉ _{i}* ≥
∑

Whenever we `Push` an object (at actual cost of 1 cyberdollar), we potentially have to
pay CY$1 in the future to `Pop` it, whether directly or in `Multipop`.

To make sure we have enough money to pay for the `Pops`, we charge `Push` CY$2.

- CY$1 pays for the push
- CY$1 is prepayment for the object being popped (metaphorically, this CY$1 is stored "on" the object).

Since each object has CY$1 credit, the credit can never go negative.

The total amortized cost *ĉ* = ∑_{i=1,n}
*ĉ _{i}* for

Since *ĉ* = O(*n*), also *c* = O(*n*).

Note: we don't actually store cyberdollars in any data structures. This is just a metaphor to enable us to compute an amortized upper bound on costs.

Does it seem strange that we charge `Pop` and `Multipop` 0 when we know they cost
something?

- Remember that this is just a way of counting the total cost over a sequence of operations more precisely.
- It is not a claim about the actual cost of a specific procedural call.
- We are guaranteeing that we have just enough credit on hand to pay for the operations when they happen.
- The accounting method and the potential method below give a tight bound on amortized cost, but with much easier counting than if we had to reason about probability distributions, etc.

Instead of credit associated with objects in the data structure, this method uses the metaphor of
*potential* associated with the *entire data structure.*

(I like to think of this as potential *energy,* but the text continues to use the monetary
metaphor.)

This is the most flexible of the amortized analysis methods, as it does not require maintaining an object-to-credit correspondence.

Let

- D
_{0}= initial data structure - D
_{i}= data structure after*i*th operation *c*= actual cost of_{i}*i*th operation.*ĉ*= amortized cost of_{i}*i*th operation*(again, notice that this is "c-hat")*.

Potential Function **Φ**: D_{i} → ℜ, and we say that
Φ(D_{i}) is the **potential** associated with the data structure in state
D_{i}.

We define the amortized cost *ĉ _{i}* to be the actual cost

ĉ=_{i}c+ Φ(D_{i}_{i}) − Φ(D_{i-1})

- If at the
*i*th operation, Φ(D_{i}) − Φ(D_{i-1}) is positive, then the amortized cost*ĉ*is an_{i}*overcharge*and the potential of the data structure increases. - On the other hand, if Φ(D
_{i}) − Φ(D_{i-1}) is negative then*ĉ*is an undercharge, and the decrease of the potential of the data structure pays for the difference (as long as it does not go negative)._{i}

The total amortized cost across *n* operations is:

∑_{i=1,n}ĉ= ∑_{i}_{i=1,n}(c+ Φ(D_{i}_{i}) - Φ(D_{i-1})) = (∑_{i=1,n}c) + (Φ(D_{i}_{n}) - Φ(D_{0}))

The last step is taken because the middle expression involves a __telescoping sum__: every
term other than D_{n} and D_{0} is added once and subtracted once.

If we require that Φ(D_{i}) ≥ Φ(D_{0}) for all *i* then
the amortized cost will always be an upper bound on the actual cost no matter which *i*th step
we are on.

This is usually accomplished by defining Φ(D_{0}) = 0 and then showing that
Φ(D_{i}) ≥ 0 for all *i*. (Note that this is a constraint on Φ, not
on *ĉ*. *ĉ* can go negative as long as Φ(D_{i}) never does.)

Define Φ(D_{i}) = number of objects in the stack.

Then Φ(D_{0}) = 0 and Φ(D_{i}) ≥ 0 for all *i*, since
there are never less than 0 objects on the stack.

Charge as follows (recalling that *ĉ _{i}* =

Since we charge 2 for each `Push` and there are O(n) Pushes in the worst case, the
amortized cost of a sequence of *n* operations is O(*n*).

There is often a tradeoff between time and space, for example, with hash tables. Bigger tables give faster access but take more space.

Dynamic tables, like the Java Hashtable, grow dynamically as needed to keep the load factor reasonable.

Reallocation of a table and copying of all elements from the old to the new table is expensive!

But this cost is amortized over all the table access costs in a manner analogous to the stack example: We arrange matters such that table-filling operations build up sufficient credit before we pay the large cost of copying the table; so the latter cost is averaged over many operations.

**Load factor α** = *num*/*size*, where *num* = # items stored and
*size* = the allocated size of the table.

For the boundary condition of *size* = *num* = 0, we will define α = 1.

We never allow α > 1 (no chaining).

We'll assume the following about our tables. (See Exercises 17.4-1 and 17.4-3 concerning different assumptions.):

When the table becomes full, we double its size and reinsert all existing items. This guarantees that α ≥ 1/2, so we are not wasting a lot of space.

Table-Insert (T,x) 1 if T.size == 0 2 allocate T.table with 1 slot 3 T.size = 1 4 if T.num == T.size 5 allocate newTable with 2*T.size slots 6 insert all items in T.table into newTable 7 T.table = newTable 8 T.size = 2*T.size 9 insert x into T.table 10 T.num = T.num + 1

Each *elementary insertion* has unit actual cost. Initially *T.num* = *T.size* =
0.

Charge 1 per elementary insertion (including copying items into a new table). Count only these insertions, since all other costs are constant per call.

** c_{i}** = actual cost of

- If the table is not full,
*c*= 1 (for lines 1, 4, 10, 11)._{i} - If full, there are
*i*- 1 items in the table at the start of the*i*th operation. Must copy all of them (line 6), and then insert the*i*th item. Therefore*c*=_{i}*i*- 1 + 1 =*i*.

A sloppy analysis: In a sequence of *n* operations where any operation can be
O(*n*), the sequence of *n* operations is O(*n*^{2}).

This is "correct", but inprecise: we rarely expand the table! A more precise account of
*c _{i}* is based on the fact that if we start with a table of size 1 and double its
size every time we expand it, we do the expansions when the number of items in the table is a power
of 2:

Then we can sum the total cost of all *c _{i}* for a sequence of

*Explanation of steps:*

- Why the
*n*? ....

It counts the "1" for "otherwise". It counts too many of them; hence the ≤. - What is the summation term counting? ....

The costs for copying at exact powers of 2. - Why does the summation start at
*j*= 0? Why does it end at*j*= lg*n*? ...

You can explain this! - How did we get rid of the summation? ...

See the formula in the box. - How did we get 2
*n*? ...

2^{x+1}= 2*2^{x}, and here*x*= lg*n*, so 2 to that power is by definition*n*.

The −1 was removed by using < and the denominator is simply 1.

Therefore, the amortized cost per operation = 3: we are only paying a small constant price for the expanding table.

We charge CY$3 for each insertion into the table, i.e., the amortized cost of the *i*-th
insertion as *ĉ _{i}* = 3. The actual cost of the

We must show that we never run out of credit. It's equivalent to proving the following Lemma.

**Lemma: For any sequences of n insertions
∑_{i=1,n} ĉ_{i} ≥
∑_{i=1,n} c_{i}.**

*Proof*: By using the same equation as above, we know that for any *n*:
∑_{i=1,n} *c _{i}* ≤ 3

That's it!

In case it helps to understand why this works:

We charge CY$3 for inserting some item *x* into the table. Obviously, we are overcharging
for each *simple* insertion which costs only CY$1. However, the overcharging will provide
enough credit to pay for future copying of items when the table becomes full:

- The first CY$1 pays for the actual cost of inserting
*x*. - The second CY$1 will pay for the cost of copying
*x*into a new table the next time table becomes full.

But the table might need to be expanded more than once after *x* is inserted, so *x*
might need to be copied more than once. Who will pay for future copying of *x*? That's where
the third CY$1 comes in:

- The third CY$1 will pay for the cost of copying some other item currently in the table that had been copied at least once before.

Let's see why CY$3 is enough to cover all possible future exansions and copying associated with them.

- Suppose the capacity of the table is
*m*immediately after an expansion. Then it holds*m*/2 items and no item in the table contains any credits. - For any insertion of an item
*x*we charge CY$3. CY$1 pays for the actual insertion of*x*; we place CY$1 credit on*x*to pay for the cost of copying it in the future, and we place CY$1 credit on some other item in the table that does not have any credit yet. - We will have to insert
*m*/2 items before the next expansion of the table. Therefore, by the time the table will get expanded next time (and, consequently, items need to be copied), every item of the table will have CY$1 credit associated with it and this credit will pay for copying that item.

The text also provides the potential analysis of table expansion.

This analysis assumed that the table never shrinks. See section 17.4 for an analysis using the potential method that covers shrinking tables.

Here are some other algorithms for which amortized analysis is useful:

An amortized analysis of Red-Black Tree restructuring (Problem 17-4 of CLRS) improves upon our analysis earlier in the semester:

- Any sequence of
*m*`RB-Insert`and`RB-Delete`operations performs O(*m*) structural modifications (rotations), - Thus each operation does
**O(1) structural modifications on average**, regardless of the size of the tree! - An operation still may need to do O(lg
*n*) recolorings, but these are very simple operations.

- Self-organizing lists use a
: Immediately after searching for an element, it is moved to the front of the list.*move-to-front heuristic* - This makes frequently accessed items more readily available near the front of the list.
- An amortized analysis (Problem 17-5) shows that the heuristic is no more than 4 times worse than optimal.

- Splay trees are ordinary binary search trees (no colors, no height labels, etc.)
- After every access (every insertion, deletion, or search), the element operated on (or its parent in the case of deletion) is moved towards the top of the tree.
- This movement uses three
operations called "zig", "zig-zig" and "zig-zag".*splaying* - Although in the worst case a splay tree can degenerate into an O(
*n*) linked list, amortized analysis shows that the expected case is O(lg*n*) - Randomization can be used to make the worst case very unlikely.
- If a single element is accessed at least
*m*/4 times where*m*is the number of operations, then the amortized running time of each of these accesses is O(1). - Thus, splay-trees self-organize to provide fast access to frequently accessed items.
- This makes them good for locality of reference in memory, but multithreaded access must be implemented carefully.

Amortized analysis will be used in analyses of

- Graph search (Topic 14, Ch. 22)
- Disjoint set operations (Topic 16, Ch. 21)
- Dijkstra's Algorithm for Shortest Paths (Topic 18, Ch. 24)

Dan Suthers & Nodari Sitchinava Last modified: Sun Nov 5 22:46:24 HST 2017

Images are from the instructor's material for Cormen et al. Introduction to Algorithms, Third Edition.