Elementary Sorts

COS 265 - Data Structures & Algorithms

Elementary Sorts

rules of the game

sorting problem

Ex: Student records in a university

Item: a row in the table

Key: a specific entry of an item that may or may not be unique

sorting problem

Sort: Rearrange array of \(N\) items into ascending order

sorting applications

sorting applications

sorting applications

sample sort clients

Goal: Sort any type of data

public class Experiment {
    public static void main(String[] args) {
        int N = Integer.parseInt(args[0]);
        Double[] a = new Double[N];
        for(int i = 0; i < N; i++) a[i] = StdRandom.uniform();
        Insertion.sort(a);    // some sorting algorithm, for now...
        for(int i = 0; i < N; i++)
            StdOut.println(a[i]);
    }
}

$ java Experiment 6
0.08614716385210452
0.10708746304898642
0.21166190071646818
0.363292849257276
0.460954145685913
0.5340026311350087

public class StringSorter {
    public static void main(String[] args) {
        String[] a = StdIn.readAllStrings();


        Insertion.sort(a);    // some sorting algorithm, for now...
        for(int i = 0; i < a.length; i++)
            StdOut.println(a[i]);
    }
}

$ more words3.txt
bed bug dad yet zoo [...] all bad yes

$ java StringSorter < words3.txt
all bad bed bug dad [...] yes yet zoo
[supressing newlines]

import java.io.File;
public class FileSorter {
    public static void main(String[] args) {
        File directory = new File(args[0]);
        File[] files = directory.listFiles();
        Insertion.sort(files);  // some sorting algorithm, for now...
        for(int i = 0; i < files.length; i++)
            StdOut.println(files[i].getName());
    }
}

$ java FileSorter .
FileSorter.class
FileSorter.java
Insertion.class
Insertion.java
Selection.class
Selection.java

total order

Goal: Sort any type of data for which sorting is well defined

A total order is a binary relation \(\leq\) that satisfies

Ex:

• Standard order for natural and real numbers
• Chronological order for dates or times
• Alphabetical order for strings

total order violations

Is-Related-To violates antisymmetry

• if both \(v \leq w\) and \(w \leq v\), then \(v = w\)

total order violations

Rock-Paper-Scissors violates transitivity

• if both \(v \leq w\) and \(w \leq x\), then \(v \leq x\)

\[\begin{array}{c} \textrm{rock} < \textrm{paper} \quad\textit{and}\quad \textrm{paper} < \textrm{scissors} \\ \textit{but}\quad \textrm{rock} \cancel< \textrm{scissors} \end{array}\]

total order violations

CSE course prerequisites violate totality

• either \(v \leq w\) or \(w \leq v\) or both

\[\begin{array}{c} \textrm{cos120} \leq \textrm{cos121} \quad\textit{and}\quad \textrm{cos121} \leq \textrm{cos265} \quad\textit{and}\quad \ldots \\ \textit{but}\quad \textrm{cos320} \overset{?}{\leq} \textrm{cos350} \quad\textit{or}\quad \textrm{cos350} \overset{?}{\leq} \textrm{cos320} \end{array}\]

callbacks

Goal: Sort any type of data for which sorting is well defined

Q. How can sort() know how to compare data without any information about the type of an item's key?

• ex: Double, String, java.io.File, pancakes?
  
  

1. Use compareTo() callback (Reference to executable code)

   • Client passes array of objects to sort() function
   • sort() calls object's compareTo() method as needed

aside: callbacks in languages

Implementing callbacks

callbacks: roadmap

client code

public class StringSorter {
    public static void main(String[] args) {
        String[] a = StdIn.readAllStrings();
        Insertion.sort(a);  // <- defined soon...
        for(int i = 0; i < a.length; i++) StdOut.println(a[i]);
    }
}

callbacks: roadmap

Comparable interface (built into Java)

public interface Comparable<Item> {
    public int compareTo(Item that);
}

String data-type implementation (built into Java)

public class String implements Comparable<String> {
    /* ... */

    public int compareTo(String that) {
        /* if this <  that, return <0 */
        /* if this == that, return =0 */
        /* if this  > that, return >0 */
    }
}

callbacks: roadmap

sort implementation (discussed soon...)

public class Insertion {
    public static void sort(Comparable[] a) {
        int N = a.length;
        for(int i = 0; i < N; i++)
            for(int j = i; j > 0; j--)
                // vvvvvvvvvvvvvvvvvvvvvv
                if(a[j].compareTo(a[j-1]) < 0) exch(a, j, j-1);
                // ^^^^^^^^^^^^^^^^^^^^^^
                // key point: no dependence on String data type!
                else break;
    }
}

comparable api

Implement compareTo() so that v.compareTo(w)

• defines a total order
• returns (resp.)...
  • negative integer if v is less than w
  • zero if v is equal to w
  • positive integer if v is greater than w
• throws an exception if incompatible types (or either is null)

if(v <  w) return -1;  // or ANY negative number
if(v == w) return  0;
if(v  > w) return +1;  // or ANY positive number

Built-in comparable types: Integer, Double, String, Date, File, ...

User-defined comparable types: implement the Comparable interface

implementing the Comparable interface

Date data type (simplified version of java.util.Date)

public class Date implements Comparable<Date> {
    //                                 ^^^^^^
    // only compares dates to other dates

    private final int month, day, year;

    public Date(int m, int d, int y) {
        month = m;
        day = d;
        year = y;
    }

    public int compareTo(Date that) {
        if(this.year  < that.year ) return -100;
        if(this.year  > that.year ) return +100;
        if(this.month < that.month) return -10;
        if(this.month > that.month) return +10;
        if(this.day   < that.day  ) return -1;
        if(this.day   > that.day  ) return +1;
        return 0;
    }
}

Elementary Sorts

selection sort

selection sort demo

• In iteration i, find index min of smallest remaining entry
• Swap a[i] and a[min]

selection sort demo  #

‹ □ 0 play: 1fps play: 2fps play: 4fps play: 8fps ›

selection sort

Algorithm: ↑ scans from left to right

Invariants

• Entries at and left of ↑ are fixed and in non-descending order
• No entry to right of ↑ is smaller than any entry at or left of ↑

two useful sorting abstractions

Helper functions: Refer to data through compares and exchanges

Less: is item v less than w?

private static boolean less(Comparable v, Comparable w) {
    return v.compareTo(w) < 0;  // note: NOT checking if == -1
}

Exchange: swap item in array a[] at index i with one at index j

private static void exch(Comparable[] a, int i, int j) {
    Comparable swap = a[i];
    a[i] = a[j];
    a[j] = swap;
}

selection sort inner loop

To maintain algorithm invariants:

1. Move ↑ to the right
2. Identify index of minimum entry on right
3. Exchange into position

// 1
i++;

// 2
int min = i;
for(int j = i+1; i < N; j++)
    if(less(a[j], a[min])) min = j;

// 3
exch(a, i, min);

selection sort: java implementation  #

public class Selection {
    public static void sort(Comparable[] a) {
        int N = a.length;
        for(int i = 0; i < N; i++) {
            int min = i;
            for(int j = i+1; j < N; j++)
                if(less(a[j], a[min])) min = j;
            exch(a, i, min);
        }
    }

    private static boolean less(Comparable v, Comparable w)
    { /* as before */ }

    private static void exch(Comparable[] a, int i, int j)
    { /* as before */ }
}

selection sort: animations  #

Key:

• Black values are sorted
• Gray values are unsorted
• Red triangle marks algorithm position

selection sort: mathematical analysis

Proposition: Selection sort uses

• \((N-1)+(N-2)+\ldots+1+0 \sim N^2/2\) compares
• \(N\) exchanges

Running time insensitive to input: Quadratic time, even if input is sorted

Data movement is minimal: Linear number of exchanges

Elementary Sorts

insertion sort

insertion sort demo

• In iteration i, swap a[i] with each larger entry to its left

insertion sort demo

‹ □ 0 play: 1fps play: 2fps play: 4fps play: 8fps ›

insertion sort

Algorithm: ↑ scans from left to right

Invariants

• Entries at and left of ↑ are in non-descending order
• Entries to the right of ↑ have not yet been sorted

insertion sort inner loop

To maintain algorithm invariants:

1. Move the pointer to the right
2. Moving from right to left, exchange a[j] with each larger entry to its left

// 1
i++;

// 2
for(int j = i; j > 0; j--)
    if(less(a[j], a[j-1])) exch(a, j, j-1);
    else break;

insertion sort: java implementation  #

public class Insertion {
    public static void sort(Comparable[] a) {
        int N = a.length;
        for(int i = 0; i < N; i++)
            for(int j = i; j > 0; j--)
                if(less(a[j], a[j-1])) exch(a, j, j-1);
                else break;
    }

    private static boolean less(Comparable v, Comparable w)
    { /* as before */ }

    private static void exch(Comparable[] a, int i, int j)
    { /* as before */ }
}

insertion sort: animations

Key:

• Black values are sorted
• Gray values are unsorted
• Red triangle marks algorithm position

insertion sort: mathematical analysis

Proposition: To sort a randomly-ordered array with distinct keys, on average insertion sort uses

• \(\sim \frac{1}{4} N^2\) compares
• \(\sim \frac{1}{4} N^2\) exchanges

Pf: Expect each entry to move halfway back.

insertion sort: analysis

Best case: If the array is in ascending order, insertion sort makes \(N-1\) compares and \(0\) exchanges.

A E E L M O P R S T X

Worst case: if the array is in descending order (and no duplicates), insertion sort makes \(\sim \frac{1}{2} N^2\) compares and \(\sim \frac{1}{2} N^2\) exchanges.

X T S R P O M L F E A

insertion sort: partially-sorted arrays

Def: An inversion is a pair of keys that are out of order

A E E L M O T R X P S

Above has 6 inversions: T-R, T-P, T-S, R-P, X-P, X-S

Def: An array is partially sorted if the number of inversions is \(\leq cN\)

• Ex 1: A sorted array has \(0\) inversions
• Ex 2: A subarray of size \(10\) appended to a sorted subarray of size \(N\)

Proposition: For partially-sorted arrays, insertion sort runs in linear time

Pf: Number of exchanges equals the number of inversions (num of compares = exchanges + \((N-1)\))

insertion sort: practical improvements

Half exchanges: Shift items over (instead of exchanging)

• Eliminates unnecessary data movement
• No longer uses only less() and exch() to access data

                         v
A C H H I  M N N P Q X Y K  B I N A R Y
            \ \ \ \ \ \ \
A C H H I  M M N N P Q X Y  B I N A R Y
           v
A C H H I  K M N N P Q X Y  B I N A R Y

Binary insertion sort: Use binary search to find insertion point

• Number of compares \(\sim N \lg N\)
• But still quadratic number of array accesses

|                       | v
|A C H H I M N N P Q X Y| K B I N A R Y
|   binary search for   |
 \    first key > K    /

Elementary Sorts

shellsort

shellsort overview

Idea: Move entries more than one position at a time by \(h\)-sorting the array

An \(h\)-sorted array is \(h\) interleaved sorted subsequences

h = 4
input:  L E E A M H L E P S O L T S X R
group0: L ----- M ----- P ----- T
group1:   E ----- H ----- S ----- S
group2:     E ----- L ----- O ----- X
group3:       A ----- E ----- L ----- R

Shellsort [Shell 1959]: \(h\)-sort array for decreasing sequence of values of \(h\)

  input: S H E L L S O R T E X A M P L E
13-sort: P H E L L S O R T E X A M S L E
 4-sort: L E E A M H L E P S O L T S X R
 1-sort: A E E E H L L L M O P R S S T X

\(h\)-sorting demo

• In iteration i, swap a[i] with each larger entry h positions to its left
• \(h\)-sort, where \(h=3\)

\(h\)-sorting demo

‹ □ 0 play: 1fps play: 2fps play: 4fps play: 8fps ›

\(h\)-sorted, where \(h=3\)

\(h\)-sorting

Q. How to \(h\)-sort an array?

1. Insertion sort, with stride length \(h\)

Q. Why insertion sort?

• Big increments (big \(h\)) → small subarray
• Small increments (small \(h\)) → nearly in order (stay tuned!)

Shellsort example: increments 7, 3, 1

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  ???

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
        M             S
          .             P
            L             R
              E             T
        M O L E E X A S P R T

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
          ________________________/___________________/
         /                   /
3-sort: M O L E E X A S P R T  ->  ???

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
          ________________________/___________________/
         /                   /
3-sort: M O L E E X A S P R T  ->  A E L E O P M S X R T
        E     M
          E     O
            .     X
        A     E     M
          .     .     S
            .     P     X
        .     .     .     R
          .     .     .     T
        A E L E O P M S X R T

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
          ________________________/___________________/
         /                   /
3-sort: M O L E E X A S P R T  ->  A E L E O P M S X R T
          ________________________/___________________/
         /                   /
1-sort: A E L E O P M S X R T  ->  ???

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
          ________________________/___________________/
         /                   /
3-sort: M O L E E X A S P R T  ->  A E L E O P M S X R T
          ________________________/___________________/
         /                   /
1-sort: A E L E O P M S X R T  ->  A E E L M O P R S T X
          E
            L
            E L
                O
                  P
                M O P
                      S
                        X
                      R S X
                          T X
        A E E L M O P R S T X

input:  S O R T E X A M P L E

7-sort: S O R T E X A M P L E  ->  M O L E E X A S P R T
          ________________________/___________________/
         /                   /
3-sort: M O L E E X A S P R T  ->  A E L E O P M S X R T
          ________________________/___________________/
         /                   /
1-sort: A E L E O P M S X R T  ->  A E E L M O P R S T X
          ________________________/___________________/
         /                   /
sorted: A E E L M O P R S T X

shellsort: java implementation  #

public class Shell {
    public static void sort(Comparable[] a) {
        int N = a.length;

        int h = 1;
        while(h < N/3) h = 3*h + 1; // 1, 4, 13, 40, 121, 364, ...

        while(h >= 1) {
            for(int i = h; i < N; i++) {
                for(int j = i; j >= h && less(a[j], a[j-h]); j -= h)
                    exch(a, j, j-h);
            }

            h = h / 3;
        }
    }

    private static boolean less(Comparable v, Comparable w)
    { /* as before */ }

    private static void exch(Comparable[] a, int i, int j)
    { /* as before */ }
}

shellsort: animations

Key:

• Black values are sorted
• Gray values are unsorted
• Dark gray values show current sub-array that is being sorted
• Red triangle marks algorithm position

shellsort: sequence

Which is the best increment sequence to use?

Powers of two? 1, 2, 4, 8, 16, 32, ...

• No

shellsort: sequence

Which is the best increment sequence to use?

Powers of two? 1, 2, 4, 8, 16, 32, ...

• No

Powers of two minus one? 1, 3, 7, 15, 31, 63, ...

• Maybe

shellsort: sequence

Which is the best increment sequence to use?

Powers of two? 1, 2, 4, 8, 16, 32, ...

• No

Powers of two minus one? 1, 3, 7, 15, 31, 63, ...

• Maybe

\(3x + 1\)? 1, 4, 13, 40, 121, 364, ...

• OK; Easy to compute

shellsort: sequence

Which is the best increment sequence to use?

Powers of two? 1, 2, 4, 8, 16, 32, ...

• No

Powers of two minus one? 1, 3, 7, 15, 31, 63, ...

• Maybe

\(3x + 1\)? 1, 4, 13, 40, 121, 364, ...

• OK; Easy to compute

Sedgewick: 1, 5, 19, 41, 109, 209, 505, 929, 2161, 3905, ...
(merging of \((9\cdot4^i) - (9\cdot2^i) + 1\) and \(4^i - (3\cdot2^i)+1\))

• Good; tough to beat in empirical studies

shellsort: intuition

Proposition: An \(h\)-sorted array remains \(h\)-sorted after \(g\)-sorting it.

7-sort: S O R T E X A M P L E    -- 3-sort: M O L E E X A S P R T
        M             S         /           E     M
          .             P       |             E     O
            L             R     |               .     X
              E             T   /           A     E     M
        M O L E E X A S P R T --              .     .     S
            ^             ^                     .     P     X
            |             |                 .     .     .     R
                7-sorted                      .     .     .     T
                                            A E L E O P M S X R T
                                                ^             ^
                                                |             |
                                                still 7-sorted!

Challenge: Prove this fact—it's more subtle than you'd think!

shellsort: analysis

Proposition: The order of growth of the worst-case number of compares used by shellsort with the \(3x+1\) increments is \(N^{3/2}\).

Property: The expected number of compares to shellsort a randomly-ordered array using \(3x+1\) increment is...

shellsort: analysis

In 2023, Skean, Ehrenborg, and Jaromczyk showed that equation

\[ \lfloor 4.0816 \cdot 8.5714^\frac{k}{2.2449} \rfloor \]

generates the sequence 1, 4, 10, 27, 72, 187, 488, ..., which has worst-case runtime of \(O(N^\frac{3}{2})\).

In other words, there is still more work to be done!

why are we interested in shellsort?

Example of simple idea leading to substantial performance gains

Useful in practice

• Fast unless array size is huge (used for small subarrays, ex: R, bzip2, /linux/kernel/groups.c)
• Tiny, fixed footprint for code (used in some embedded systems, ex: uClibc)
• Hardware sort prototype

why are we interested in shellsort?

Simple algorithm, nontrivial performance, interesting questions

• Asymptotic growth rate?
• Best sequence of increments? (open problem: find a better increment sequence)
• Average-case performance?

Lesson: Some good algorithms are still awaiting discovery

Elementary sorts summary

This section: Elementary sorting algorithms

Order of growth of running time to sort an array of \(N\) items

Next section: \(N \lg N\) sorting algorithms (in worst case)

Elementary Sorts

shuffling

how to shuffle an array

Goal: Rearrange array so that result is a uniformly random permutation (uniformly → all permutations are equally likely)

shuffle sort

• Generate a random real number for each array entry
• Sort the array based on the generated number

shuffle sort

• Generate a random real number for each array entry
• Sort the array based on the generated number

Proposition: Shuffle sort produces a uniformly random permutation, assuming real numbers uniformly at random (and no ties)

war story (microsoft)

Microsoft antitrust probe by EU: Microsoft agreed to provide a randomized ballot screen for users to select browser in Windows 7.

However, IE8 appeared last 50% of the time!

war story (microsoft)

Microsoft antitrust probe by EU: Microsoft agreed to provide a randomized ballot screen for users to select browser in Windows 7.

Solution? Implement shuffle sort by making comparator always return a random answer

// browser comparator (should implement a total order!)
public int compareTo(Browser that) {
    double r = Math.random();
    if(r < 0.5) return -1;
    if(r > 0.5) return +1;
    return 0;
}

knuth shuffle demo

• In iteration i, pick integer r between 0 and i uniformly at random
• Swap a[i] and a[r]

knuth shuffle demo

• In iteration i, pick integer r between 0 and i uniformly at random
• Swap a[i] and a[r]

‹ □ 0 play: 1fps play: 2fps play: 4fps play: 8fps ›

knuth shuffle demo

• In iteration i, pick integer r between 0 and i uniformly at random
• Swap a[i] and a[r]

Proposition [Fisher-Yates 1938]: Knuth shuffling algorithm produces a uniformly random permutation of the input array in linear time, assuming integers uniformly at random.

knuth shuffle

• In iteration i, pick integer r between 0 and i uniformly at random
• Swap a[i] and a[r]

Common bug: picking r between 0 and N-1.
Correct variant: between i and N-1

public class StdRandom {
    /* ... */
    public static void shuffle(Object[] a) {
        int N = a.length;
        for(int i = 0; i < N; i++) {
            int r = StdRandom.uniform(i+1); // between 0 and i
            exch(a, i, r);
        }
    }
}

Broken knuth shuffle

Q. What happens if integer is chosen between 0 and N-1?

1. Not uniformly random!

Probability of each result when shuffling A B C

War story (online poker)

Texas hold'em poker: Software must shuffle electronic cards

War story (online poker)

// shuffling algorithm in FAQ at www.planetpoker.com
for i := 1 to 52 do begin
    r := random(51) + 1; // between 1 and 51
    swap := card[r];
    card[r] := card[i];
    card[i] := swap;
end;

Bug 1: Random number r never 52 ⇒ 52nd card cannot end up in 52nd place

Bug 2: Shuffle not uniform (should be between 1 and i)

Bug 3: random() uses 32-bit seed ⇒ \(2^{32}\) possible shuffles

Bug 4: Seed = milliseconds since midnight ⇒ 86.4 million shuffles

War story (online poker)

“

The generation of random numbers is too important to be left to chance.
—Robert R. Coveyou

”

War story (online poker)

Best practices for shuffling (if your business depends on it)

• Use a hardware random-number generator that has passed both the FIPS 140-2 and the NIST statistical test suites
• Continuously monitor statistic properties: hardware random-number generators are fragile and fail silently
• Use an unbiased shuffling algorithm

Bottom line: Shuffling a deck of cards is hard!

lavarand

Lavarand was a Silicon Graphics service that used a wall of lava lamps to generate random numbers.

As of 2017, Cloudfare maintains a similar system of lava lamps for securing approximately 10% of the Internet's traffic (wikipedia, Tom Scott: The Lava Lamps That Help Keep The Internet Secure)