fullscreen
timer
qrcode
plickers
selector
edit
reset

Analysis of Algorithms

COS 265 - Data Structures & Algorithms

Analysis of Algorithms

introduction

cast of characters


Although you all are currently students, someday you might play any or all of these roles

running time

“

As soon as an Analytical Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question will then arise—By what course of calculation can these results be arrived at by the machine in the shortest time?
–Charles Babbage (1864)

”
Difference Engine, Photo by geni, link

running time

“

Rare book containing the world’s first computer algorithm earns $125,000 at auction

”

Ada Lovelace's algorithm to compute Bernoulli numbers on Analytical Engine (1843)
Article | News Atlas

reasons to analyze algorithms

Reasons to analyze algorithms:

COS265 COS320 COS435
• Predict performance ✓
• Compare algorithms ✓ ✓
• Provide guarantees ✓ ✓ ✓
• Understand theoretical basis ✓ ✓

Primary practical reason: avoid performance bugs

Client gets poor performance because programmer did not understand performance characteristics



COS320 Algorithm Design
COS435 Theory of Computation

an algorithmic success story

N-body simulation

an algorithmic success story

Discrete Fourier transform

original (67.0KB), 90% (30.5KB), 10% (4.7KB)
original (67.0KB), 90% (30.5KB), 10% (4.7KB)
Test Card F, George Hersee, link

the challenge

Q: Will my program be able to solve a large practical input?

“

Why is my program so slow??

”

“

Why does it run out of memory??

”



Insight by Knuth (1970s): Use scientific method to understand performance

scientific method applied to alg. analysis

A framework for predicting performance and comparing algorithms

Scientific method:

Principles:

algorithm analysis

observations

example: 3-sum

3-Sum: Given \(N\) distinct integers, how many triples sum to exactly zero?

Context: Deeply related to problems in computational geometry

$ cat 8ints.txt
8
30 -40 -20 -10 40 0 10 5

$ java ThreeSum 8ints.txt
4
1. 30 -40 10
2. 30 -20 -10
3. -40 40 0
4. -10 0 10
Angry Birds, by Rovio Entertainment

3-sum brute-force algorithm

ThreeSum.java: source, 1Kints.txt, 2Kints.txt, 4Kints.txt, 8Kints.txt

Algs4 booksite, link
public class ThreeSum {
    public static int count(int[] a) {
        int N = a.length;
        int count = 0;

        // check each triple (ignore integer overflow for simplicity)
        for(int i = 0; i < N; i++)
            for(int j = i+1; j < N; j++)
                for(int k = j+1; k < N; k++)
                    if(a[i] + a[j] + a[k] == 0)
                        count++;
        return count;
    }

    public static void main(String[] args) {
        In in = new In(args[0]);
        int[] a = in.readAllInts();
        StdOut.println(count(a));
    }
}

quiz: 3-sum brute-force run-time estimation

live view; card view; qrcode

Based on the code listing to the right (same as previous slide), which of the following best estimates the run-time of running the brute force implementation of 3-sum?

int count(int[] a) {
    int N = a.length;
    int count = 0;

    // check each triple
    // ignore integer overflow for simplicity
    for(int i = 0; i < N; i++)
        for(int j = i+1; j < N; j++)
            for(int k = j+1; k < N; k++)
                if(a[i] + a[j] + a[k] == 0)
                    count++;
    return count;
}
  1. Constant
    (independent of \(N\))
  2. \(N\)
  3. \(N^2\)
  4. \(N^3\)

measuring the running time

Q:   How to time a program?

measuring the running time

Q:   How to time a program?
A1: (ノಠ益ಠ)ノ Manually with a stopwatch or wall clock

measuring the running time

Q:   How to time a program?
A1: (ノಠ益ಠ)ノ Manually with a stopwatch or wall clock
A2: (ಠ_ಠ) Unix time (ex: time java ThreeSum ...)

measuring the running time

Q:   How to time a program?
A1: (ノಠ益ಠ)ノ Manually with a stopwatch or wall clock
A2: (ಠ_ಠ) Unix time (ex: time java ThreeSum ...)
A3: (♥‿♥) Automatically using programming!

public static void main(String[] args) {
    // do the things that should not be timed first...
    In in = new In(args[0]);
    int[] a = in.readAllInts();

    // now, run the experiment and measure the running time...
    Stopwatch stopwatch = new Stopwatch();    // start stopwatch
    StdOut.println(ThreeSum.count(a));        // run experiment
    double time = stopwatch.elapsedTime();    // record elapsed time

    // finally, report the results
    StdOut.println("elapsed time = " + time);
}

Emperical analysis

Run the program for various input sizes and measure running time

\(N\) \(T(N)\)
250 0.0015
500 0.0139
1000 0.1038
2000 0.7989
4000 6.3868
8000 51.0588
16000
???

Can we predict \(T(N)\) when \(N = 16000\)?



Note

\(N\) is the size of the input (ex: count of integers for 3-sum) and \(T(N)\) is time in seconds on some particular machine

data analysis

Standard linear plot of \(N\) vs. \(T(N)\)

Difficult to predict \(T(N)\) when \(N = 16000\)...

data analysis

Log-log plot of \(\lg(N)\) vs. \(\lg(T(N))\)

Far easier to predict \(T(N)\) when \(N = 16000\) using log-log scale!

data analysis

Log-log plot of \(\lg(N)\) vs. \(\lg(T(N))\) (log-log scale)

Line Equation: \(y = m' x + b'\)

substitute for \(y\), \(x\); solve for \(m'\), \(b'\)

\[\underbrace{\lg(T(N))}_y = m' \cdot \underbrace{\lg N}_x + b'\]

Fit line to data using regression

\[ m' = 2.9929 \quad b' = -33.1214 \]

Power Law Equation: \(T(N) = a N^b\)

\[ a = 2^{b'} \quad b = m' \quad T(N) = 2^{b'} N^{m'} \]

Hypothesis: Running time is about \(1.0702 \cdot 10^{-10} \times N^{2.9929}\) secs

Warning

\(b\) from Power Law and \(b'\) from Line are different!

prediction and validation

Hypothesis: Running time is about \(1.0702 \cdot 10^{-10} \times N^{2.9929}\) secs
("order of growth" of running time is about \(N^3\))

Predictions:

\(N\) \(T(N)\)
8000 51.3904
16000 409.0955

Additional observations:

\(N\) \(T(N)\)
8000 51.1, 51.0, 51.1
16000 410.8

Observations match predictions ⇒ validates hypothesis!


Note

We already measured \(T(N=8000)\), but we predicted \(T(N=16000)\) before we ran and measured.

doubling hypothesis

Linear regression is relatively easy to implement, especially after taking MAT210, COS345, or MAT345. In fact, the algs4.jar file has LinearRegression class!

public static void main(String[] args) {
    double[] lg_N = { lg( 250), lg( 500), lg(1000),
                      lg(2000), lg(4000), lg(8000) };
    double[] lg_T = { lg(0.0015), lg(0.0139), lg( 0.1038),
                      lg(0.7989), lg(6.3868), lg(51.0588) };

    // solve for m and b of lg-lg line
    LinearRegression line = new LinearRegression(lg_N, lg_T);

    StdOut.println("m = " + line.slope());
    StdOut.println("b = " + line.intercept());
    StdOut.println("predict: " + line.predict(lg(16000)));
}

However linear regression requires a few samples to work well.

There is a quicker way to estimate \(b\) in a power-law relationship that is even easier to implement...

doubling hypothesis

Doubling hypothesis: estimate \(b\) in a power-law relationship by doubling the size of the input.

The ratio of \(T(N)\) over \(T(N/2)\) is \(2^b\), so \(\lg\) both sides to get \(b\).

\[ \underbrace{\frac{T(N)}{T(N/2)}}_{\text{ratio}} = \frac{aN^b}{a \left(\frac{N}{2}\right)^b} = \frac{aN^b}{aN^b \left(\frac{1}{2}\right)^b} = 2^b \]

\[ b = \lg\left(\frac{T(N)}{T(N/2)}\right) \]

Hypothesis: Running time is about \(aN^b\) with \(b = \lg(\text{ratio})\)

Warning

Caveat: Cannot identify log factors (ex: \(N \lg N\)) or lower-order terms (ex: \(N^3 + N^2\)) with doubling hypothesis!

doubling hypothesis

Run program, doubling the size of the input, and plug into equation

\(N\) \(T(N)\) \(\text{ratio}\) \(\lg(\text{ratio})\)
250 0.0015 — —
500 0.0139 9.2667 3.2121
1000 0.1038 7.4676 2.9007
2000 0.7989 7.6965 2.9442
4000 6.3868 7.9945 2.9990
8000 51.0588 7.9944 2.9990

\[\begin{array}{rcl} b & \approx & \lg\left( \frac{51.0588}{6.3868} \right) \\ & \approx & \lg( 7.9944 ) \\ & \approx & 2.9990 \end{array}\]

seems to converge to \(b \approx 3\)


Hypothesis: Running time is about \(a N^{2.9990}\) for some \(a\)


Note

It is best to average \(T(N)\) and \(T(N/2)\) over few runs.

doubling hypothesis

Q: How to estimate \(a\) (assuming we know \(b\))?
A: Run the program (for sufficiently large value of \(N\)) and solve for \(a\)

\(N\) \(T(N)\)
8000 51.0588

\[\begin{array}{rcl} a N^b & = & T(N) \\ a & = & T(N) / N^b \\ & \approx & 51.0588 / 8000^{2.9990} \\ & \approx & 1.0062 \cdot 10^{-10} \end{array}\]

Hypothesis: Running time is about \(1.0062 \cdot 10^{-10} \times N^{2.9990}\) seconds


Note

This is almost identical hypothesis to one obtained via regression (\(1.0702 \cdot 10^{-10} \times N^{2.9929}\)), but required less work and very easy to write code to compute this!

quiz: estimate running time

live view; card view; qrcode

Estimate the running time (\(T(N)\), units:seconds) to solve a problem of size \(N=96{,}000\) based on the timing table on right.

\(N\)\(T(N)\)
1,000 0.02
2,000 0.05
4,000 0.20
8,000 0.81
16,000 3.25
32,000 13.01
96,000 ????

  1. 39 seconds
  2. 52 seconds
  3. 117 seconds
  4. 350 seconds

math tangent: change of base

Most math libraries do not have \(\lg = \log_2\) function

But you can compute the \(\log\) (ex: of \(A\)) with any base (ex: \(b\)) using the \(\log\) of any other base (ex: \(c\))

\[ \log_b A = \frac{\log_c A}{\log_c b} \]

So, if you have \(\log_{10}\) (Math.log10) or \(\ln = \log_e\) (Math.log), you can compute \(\lg\) as

\[ \log_2 A = \frac{\log_{10} A}{\log_{10} 2} = \frac{\ln A}{\ln 2} \]

quiz: change of base

live view; card view; qrcode

If we don't have \(\lg\), how can we calculate equation below?

\[ b = \lg\left( \frac{T(N)}{T(N/2)} \right) \]

  1. \(b = \log\left( \frac{T(N)}{T(N/2)} \right)\)
  2. \(b = \log\left( \frac{T(N)}{ T(N/2) } \right) / \log(2)\)
  3. \(b = \frac{\log( T(N) ) }{ \log( T(N / 2) )}\)
  4. \(b = \ln \left( \frac{T(N) }{ T(N/2) }\right) / \log(2)\)

experimental algorithmics

\[\text{power law: } \quad T(N) = a N^b\]

system independent effects, determines factor \(a\) and exponent \(b\)

system dependent effects, determines factor \(a\)


thumb_down

Sometimes difficult to get precise measurements.

thumb_up

Much easier and cheaper than other sciences

as an aside

Algorithmic experiments virtually free compared to other sciences

chemistry: 1 experiment
chemistry: 1 experiment
biology: 1 experiment
biology: 1 experiment
physics: 1 experiment
physics: 1 experiment
computer science: 1 million experiments
computer science: 1 million experiments

Bottom line: no excuse to not run experiments to understand costs!

chem, bio, phys, compsci

Analysis of Algorithms

mathematical models

mathematical models for running time

total running time: sum of cost × frequency for all operations

In principle, accurate mathematical models are available.

books, knuth

example: 1-sum

Q: How many instructions as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    if(a[i] == 0)               // <- N array accesses
        count++;
operation cost† frequency
variable declaration \(0.4\text{ ns}\) \(2\)
assignment statement \(0.2\text{ ns}\) \(2\)
less-than compare \(0.2\text{ ns}\) \(N+1\)
equal-to compare \(0.1\text{ ns}\) \(N\)
array access \(0.1\text{ ns}\) \(N\)
increment \(0.1\text{ ns}\) \(N\) to \(2N\)

\[ (1.4 + 0.5N) \text{ ns} \quad\textrm{to}\quad (1.4 + 0.6N) \text{ ns} \]

cost†: representative estimates (with some poetic license)

example: 2-Sum

Q: How many instructions as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)        // inner loop
            count++;

Q: How many times does inner loop body (i.e., if) repeat?

example: 2-Sum

Q: How many instructions as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)        // inner loop
            count++;

Q: How many times does inner loop body (i.e., if) repeat?

Pf. by Carl Friedrich Gauss

\[\begin{array}{cccccccccccc} & T(N) & = & 0 & + & 1 & + & \ldots & + & (N-2) & + & (N-1) \\ + & T(N) & = & (N-1) & + & (N-2) & + & \ldots & + & 1 & + & 0 \\ \hline & 2T(N) & = & (N-1) & + & (N-1) & + & \ldots & + & (N-1) & + & (N-1) \\ \\ \Rightarrow & T(N) & = & N (N-1) / 2 \end{array}\]

Note

\[0 + 1 + 2 + \ldots + (N-1) = \frac{N \cdot (N-1)}{2} = \binom{N}{2}\]

example: 2-Sum

Q: How many instructions as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)        // inner loop
            count++;
operation cost frequency
variable declaration \(0.4\text{ ns}\) \(N+2\)
assignment statement \(0.2\text{ ns}\) \(N+2\)
less-than compare \(0.2\text{ ns}\) \(\onehalf (N+1)(N+2)\)
equal-to compare \(0.1\text{ ns}\) \(\onehalf N(N-1)\)
array access \(0.1\text{ ns}\) \(N(N-1)\)
increment \(0.1\text{ ns}\) \(\small \onehalf(N^2{+}3N{+}2)\) to \(\small N^2{+}N{+}1\)

Timing is tedious to count exactly...

\[ \left( 0.30 N^2 + 0.90 N + 1.5 \right) \text{ns} \quad\textrm{to}\quad \left( 0.35 N^2 + 0.85 N + 1.5 \right) \text{ns} \]

simplifying the calculations




How do we simplify this work, especially so that we can analyze more complex algorithms?

“

It is convenient to have a measure of the amount of work involved in a computing process, even though it be a very crude one. We may count up the number of times that various elementary operations are applied in the whole process and then given them various weights. We might, for instance, count the number of additions, subtractions, multiplications, divisions, recording of numbers, and extractions of figures from tables. In the case of computing with matrices most of the work consists of multiplications and writing down numbers, and we shall therefore only attempt to count the number of multiplications and recordings.
–Alan Turing (1947)

”

simplification 1: cost model

Cost model: use some basic operation as a proxy for running time

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)        // inner loop
            count++;
operation cost frequency
variable declaration \(0.4\text{ ns}\) \(N+2\)
assignment statement \(0.2\text{ ns}\) \(N+2\)
less-than compare \(0.2\text{ ns}\) \(\onehalf (N+1)(N+2)\)
equal-to compare \(0.1\text{ ns}\) \(\onehalf N(N-1)\)
array access \(0.1\text{ ns}\) \(N(N-1)\)
increment \(0.1\text{ ns}\) \(N(N+1)\) to \(N^2\)

Focus on the "key" operation of the process

simplification 1: cost model

Cost model: use some basic operation as a proxy for running time

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)        // inner loop
            count++;
operation cost frequency
variable declaration \(0.4\text{ ns}\) \(N+2\)
assignment statement \(0.2\text{ ns}\) \(N+2\)
less-than compare \(0.2\text{ ns}\) \(\onehalf (N+1)(N+2)\)
equal-to compare \(0.1\text{ ns}\) \(\onehalf N(N-1)\)
array access \(0.1\text{ ns}\) \(N(N-1)\)
increment \(0.1\text{ ns}\) \(N(N+1)\) to \(N^2\)

Focus on the "key" operation of the process (ex: array access)

Note

Assuming compiler/JVM does not optimize away any array accesses

simplification 2: tilde notation

Tilde notation: keep highest order term; ignore lower order terms


original: \(f(N)\) tilde: \(g(N)\)
\(\onesixth N^3 + 20N + 16\) \(\sim\onesixth N^3\)
\(\onesixth N^3 + 100N^2 + 56\) \(\sim\onesixth N^3\)
\(\onesixth N^3 - \onehalf N^2 + \onethird N\) \(\sim\onesixth N^3\)



Note

Technical definition: \(f(N) \sim g(N)\) means

\[\lim_{N \rightarrow \infty} \frac{f(N)}{g(N)} = 1\]

simplification 2: tilde notation

Tilde notation: keep highest order term; ignore lower order terms

\[N = 1000\]

\[f(N) = 166.17 \text{million}\]

\[g(N) = 166.67 \text{million}\]

simplification 2: tilde notation

Tilde notation: keep highest order term; ignore lower order terms


operation frequency tilde notation
var declaration \(N+2\) \(\sim N\)
assign statement \(N+2\) \(\sim N\)
less-than compare \(\onehalf (N+1)(N+2)\) \(\sim \onehalf N^2\)
equal-to compare \(\onehalf N(N-1)\) \(\sim \onehalf N^2\)
array access \(N(N-1)\) \(\sim N^2\)
increment \(\onehalf N(N+1)\) to \(N^2\) \(\sim \onehalf N^2\) to \(\sim N^2\)

example: 2-Sum

Q: Approximately how many array accesses as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)                // inner loop
            count++;

example: 2-Sum

Q: Approximately how many array accesses as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        if(a[i] + a[j] == 0)                // inner loop
            count++;

A: \(\sim N^2\) array accesses


Bottom line: use cost model and tilde notation to simplify counts

example: 3-sum

Q: Approximately how many array accesses as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        for(int k = j+1; k < N; k++)
            if(a[i] + a[j] + a[k] == 0)     // inner loop
                count++;

example: 3-sum

Q: Approximately how many array accesses as a function of input size \(N\)?

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        for(int k = j+1; k < N; k++)
            if(a[i] + a[j] + a[k] == 0)     // inner loop
                count++;

\[\binom{N}{3} = \frac{N(N-1)(N-2)}{3!} \sim \frac{1}{6}N^3\]

A: \({\sim}\frac{1}{2} N^3\) array accesses


Bottom line: use cost model and tilde notation to simplify counts

Estimating a discrete sum

Q: How to estimate a discrete sum?

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)
A2: Replace the sum with an integral and use calculus


Ex1: \(1 + 2 + \ldots + N\)

\[\sum_{i=1}^N i \sim \int_{x=1}^N x\ dx \sim \frac{1}{2}N^2\]

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)
A2: Replace the sum with an integral and use calculus


Ex2: \(1 + 1/2 + 1/3 + \ldots + 1/N\)

\[\sum_{i=1}^N \frac{1}{i} \sim \int_{x=1}^N \frac{1}{x}\ dx \sim \ln N\]

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)
A2: Replace the sum with an integral and use calculus


Ex3: 3-Sum triple loop

\[\sum_{i=1}^N \sum_{j=i}^N \sum_{k=j}^N 1 \sim \int_{x=1}^N \int_{y=x}^N \int_{z=y}^N dz\ dy\ dx \sim \frac{1}{6} N^3\]

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)
A2: Replace the sum with an integral and use calculus


Ex4: \(1 + 1/2 + 1/4 + 1/8 + \ldots\)

\[\sum_{i=0}^\infty \left(\frac{1}{2}\right)^i = 2\]

\[\int_{x=0}^\infty \left(\frac{1}{2}\right)^x\ dx = \frac{1}{\ln 2} \approx 1.4427\]

Warning

Integral trick does not always work

Estimating a discrete sum

Q: How to estimate a discrete sum?
A1: Take a discrete mathematics course (MAT 215)
A2: Replace the sum with an integral and use calculus
A3: Use Maple or Wolfram Alpha

mathematical models for running time

In principle, accurate mathematical models are available.


In practice,

mathematical models for running time

\[\begin{array}{ccl} T_N & = & c_1 A + c_2 B + c_3 C + c_4 D + c_5 E \\ A & = & \text{array access} \\ B & = & \text{integer add} \\ C & = & \text{integer compare} \\ D & = & \text{increment} \\ E & = & \text{variable assignment} \\ A–E & = & \text{frequencies (depend on algorithm, input)} \\ c_i & = & \text{costs (depend on machine, compiler)} \end{array}\]

Bottom line: we use approximate models in this course

\[T(N) \sim a N^b\]

analysis of algorithms

order-of-growth classifications

common order-of-growth classifications

Definition: If \(f(N) \sim c g(N)\) for some constant \(c > 0\), then the order of growth of \(f(N)\) is \(g(N)\).

Ex: The order of growth of the running time of this code is \(N^3\)

int count = 0;
for(int i = 0; i < N; i++)
    for(int j = i+1; j < N; j++)
        for(int k = j+1; k < N; k++)
            if(a[i] + a[j] + a[k] == 0)
                count++;

Typical usage: mathematical analysis of running times, where leading coefficients depend on machine, compiler, JVM, ...

common order-of-growth classifications

Good news: the set of functions below suffices to describe the order of growth of most common algorithms.

constant logarithmic linear linearithmic quadratic cubic exponential
\(1\) \(\log N\) \(N\) \(N \log N\) \(N^2\) \(N^3\) \(2^N\)

common order-of-growth classifications

order name description example ratio
\(1\) constant statement add two numbers \(1\)
\(\log N\) logarithmic divide in half binary search \(\sim 1\)
\(N\) linear single loop find the max \(2\)
\(N \log N\) linearithmic divide and conquer mergesort \(\sim 2\)
\(N^2\) quadratic double loop check all pairs \(4\)
\(N^3\) cubic triple loop check all triples \(8\)
\(2^N\) exponential exhaustive search check all subsets \(T(N)\)

where ratio is \(\frac{T(N)}{T(N/2)}\)

binary search

Goal: given sorted array and key, find index of the key in the array

Binary search: compare middle entry against key
too big look in left side repeat
too small look in right side repeat
equal found it! done
else cannot find it! done

Example with 15 values.

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};

binary search
if a[mid]>key a[mid]==key a[mid]<key empty
then hi=mid-1 return mid lo=mid+1 return -1
go left found it! go right not in 
find(33);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//         |_                   ^^                   _|
//         lo                   mid                  hi <---
//                            53 > 33

a[mid] > key, so go left by updating hi to mid-1

find(33);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//         |_       ^^       _|
//    ---> lo       mid      hi
//                25 < 33

a[mid] < key, so go right by updating lo to mid+1

find(33);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//                     |_ ^^ _|
//                     lo md hi <---
//                      43 > 33

a[mid] > key, so go left

find(33);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//                     ^^
//                lo = mid = hi
//                  33 == 33

a[mid] == key, so found key!

find(34);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//         |_                   ^^                   _|
//         lo                   mid                  hi <---
//                            53 > 34

a[mid] > key, so go left

find(34);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//         |_       ^^       _|
//    ---> lo       mid      hi
//                25 < 34

a[mid] < key, so go right

find(34);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//                     |_ ^^ _|
//                     lo md hi <---
//                     43 > 34

a[mid] > key, so go left

find(34);

//          0  1  2  3  4  5  6  7  8  9 10 11 12 13 14
int[] a = {11,13,14,25,33,43,51,53,64,72,84,93,95,96,97};
//                     ^^
//                lo = mid = hi
//                  33 != 34

a[mid] != key and lo == hi (nothing left to search), so not found!

binary search: implementation

Trivial to implement? Easy to get right?

binary search: implementation

Trivial to implement? Easy to get right?

link

binary search: java implementation

Invariant: if key appears in array a[], then a[lo]<=key<=a[hi].

public static int binarySearch(int key, int[] a) {
    int lo = 0, hi = a.length - 1;
    while(lo <= hi) {
        int mid = lo + (hi - lo) / 2;         // why not mid = (lo + hi) / 2?

        if     (key < a[mid]) hi = mid - 1;   // |
        else if(key > a[mid]) lo = mid + 1;   // | one "3-way compare"
        else return mid;                      // |
    }
    return -1;
}

binary search: mathematical analysis

Proposition: binary search uses at most \(1 + \lg N\) key compares to search a sorted array of size \(N\).

Def: \(T(N) =\) number key compares to binary search a sorted subarray of size \(\leq N\)

Binary search recurrence:

binary search: mathematical analysis

Proposition: binary search uses at most \(1 + \lg N\) key compares to search a sorted array of size \(N\).

Pf sketch (assume \(N\) is a power of \(2\)) \[\begin{array}{rcll} T(N) & \leq & 1 + \quad\quad T(N/2) & \text{given} \\ & \leq & 1 + (1 + \quad T(N/4)) & \text{apply recurrence to 1st term} \\ & \leq & 1 + (1 + (1 + T(N/8))) & \text{apply recurrence to 1st term} \\ & \vdots & \vdots & \vdots \\ & \leq & \underbrace{1 + \ldots + 1}_{\lg N} + T(N/N) & \text{stop applying, } T(1)=1 \\ & = & \lg N + 1 & \end{array}\]

the 3-sum problem

3-Sum: Given \(N\) distinct integer, find three such that \(a + b + c = 0\)

version time space
0 \(N^3\) \(N\)
1 \(N^2 \lg N\) \(N\)
2 \(N^2\) \(N\)




Note

For full credit in COS265, running time should be worst case.

Comparing programs

Hypothesis: the sorting-based \(N^2 \log N\) algorithm for 3-Sum is significantly faster in practice than the brute-force \(N^3\) algorithm.

ThreeSum.java
\(N\) time (secs)
1000 0.1
2000 0.8
4000 6.4
8000 51.1
ThreeSumDeluxe.java
\(N\) time (secs)
1000 0.14
2000 0.18
4000 0.34
8000 0.96
16000 3.67
32000 14.88
64000 59.16

Guiding principle: Typically, better order of growth \(\Rightarrow\) faster in practice

Analysis of Algorithms

Memory

basics

name values/sizes base
Bit 0 or 1 binary
Byte 8 bits binary
Megabyte (MB) 10002 bytes decimal
Mebibyte (MiB) 220 bytes binary
Gigabyte (GB) 10003 bytes decimal
Gibibyte (GiB) 230 bytes binary

64-bit machine: We assume a 64-bit machine with 8-byte pointers

(some JVMs "compress" ordinary object pointers to 4 bytes to avoid this cost)


Old school floppy disks 💾: 1 MB = 1000 * 1024 bytes (3.5" 1.44MB)

Byte - Wikipedia

typical memory usage

typical memory usage for primitive types and one- and two-dimensional arrays

type bytes
boolean \(1\)
byte \(1\)
char \(2\)
int \(4\)
float \(4\)
long \(8\)
double \(8\)
type bytes
char [] \(2N + 24\)
int [] \(4N + 24\)
double[] \(8N + 24\)
char [][] \(\sim 2MN\)
int [][] \(\sim 4MN\)
double[][] \(\sim 8MN\)

Every Java array has 24 bytes overhead to store meta info (class, hash code, length, etc.), but could be less depending on JVM

typical memory for Java objects

Each Java object uses multiple of 8 bytes and has overhead
Overhead 16 bytes
Padding round up to 8 bytes

Ex: A Date object uses 32 bytes of memory

public class Date {
    private int day;
    private int month;
    private int year;
    // ...
}

Could be less depending on JVM

typical memory for java references

Every reference to Java object (even if null) uses 8 bytes


Ex:

Date date = null;   // date uses 8 bytes, even though it is null!

date = new Date(2026, 12, 25);   // now date (still 8 bytes) refs
                                 // a new Date object (32 bytes)
                                 // 8 + 32 = 40 bytes used

Date another = date;  // another uses 8 bytes, but refs same Date
                      // object that date refs.
                      // 8 + 8 + 32 = 48 bytes used

date = null;    // date and another still use 8 bytes each, but the
another = null; // Date object (32 bytes) will eventually be freed
                // by garbage collector
                // 8 + 8 = 16 bytes used

Could be less depending on JVM

typical memory usage summary

Total memory usage for a data type value:
Primitive \(4\) bytes for int, \(8\) bytes for double, ...
Object ref \(8\) bytes
Enum ref \(8\) bytes
Array \(24\) bytes + memory for each array entry
Object \(16\) bytes + memory for each instance variable
(add \(8\) extra bytes if inner class object for reference to enclosing object)
Padding round up to multiple of \(8\) bytes

Note

Depending on application, we may want to count memory for any referenced objects (recursively)

quiz: Analysis of algorithms

live view; card view; qrcode

How much memory does a WeightedQuickUnionUF use as a function of \(N\)?

A: \(\sim 4N\) bytes

B: \(\sim 8N\) bytes

C: \(\sim 4N^2\) bytes

D: \(\sim 8N^2\) bytes

 
public class WeightedQuickUnionUF {
    private int[] parent;
    private int[] size;
    private int count;

    public WeightedQuickUnionUF(int N)
    {
        parent = new int[N];
        size = new int[N];
        count = 0;
        // ...
    }

    // ...
}

turning the crank: summary

Empirical analysis

Mathematical analysis

Scientific method

turning the crank: summary

Empirical analysis

Mathematical analysis

Scientific method

turning the crank: summary

Empirical analysis

Mathematical analysis

Scientific method

turning the crank: summary

Empirical analysis

Mathematical analysis

Scientific method

×