Dynamic Programming (6)
COS 320 - Algorithm Design
Dynamic Programming (6)
segmented least squares
least squares
Least squares: Foundational problem in statistics
- Given \(n\) points in the plane: \((x_1,y_1), (x_2,y_2), \ldots, (x_n,y_n)\)
- Find a line \(y=ax+b\) that minimizes the sum of the squared error:
|
\[ \mathrm{SSE} = \sum_{i=1}^n (y_i - a x_i - b)^2 \] |
 |
Solution: Calculus ⇒ min error is achieved when
\[ a = \frac{n \sum_i x_i y_i - (\sum_i x_i) (\sum_i y_i)}{n \sum_i x_i^2 - (\sum_i x_i)^2}, b = \frac{\sum_i y_i - a \sum_i x_i}{n} \]
segmented least squares
Segmented least squares:
- Points lie roughly on a sequence of several line segments
- Given \(n\) points in plane: \((x_1,y_1), (x_2,y_2), \ldots, (x_n,y_n)\) with \(x_1 < x_2 < \ldots x_n\), find sequence of lines that minimizes \(f(x)\)
Q: What is a reasonable choice for \(f(x)\) to balance accuracy (goodness of fit) and parsimony (number of lines)?
segmented least squares
Given \(n\) points in the plane: \((x_1,y_1), (x_2,y_2), \ldots, (x_n,y_n)\) with \(x_1 < x_2 < \ldots < x_n\) and a constant \(c > 0\), find a sequence of lines that minimizes \(f(x) = E + c L\):
- \(E\) is the sum of the sums of the squared errors in each segment
- \(L\) is the number of lines
dynamic programming: multiway choice
Notation
- \(\mathrm{OPT}(j)\) is minimum cost for points \(p_1, p_2, \ldots, p_j\)
- \(e(i,j)\) is minimum sum of squares for points \(p_i, p_{i+1}, \ldots, p_j\)
To compute \(\mathrm{OPT}(j)\):
- Last segment uses points \(p_i, p_{i+1}, \ldots, p_j\) for some \(i\)
- Cost is \(e(i,j) + c + \mathrm{OPT}(i-1)\) (optimal substructure property; proof via exchange argument) \[ \mathrm{OPT}(j) = \begin{cases} 0 & \text{if } j=0 \\ \min\limits_{1 \leq i \leq j} \{ e(i,j) + c + \mathrm{OPT}(i-1) \} & \text{otherwise} \end{cases} \]
segmented least squares algorithm
Segmented-Least-Squares(n, p1, ..., pn, c):
For j = 1 to n
For i = 1 to j
Compute the least squares e(i,j) for the segment pi--pj
M[0] <- 0
For j = 1 to n
Find i in [1,j] that minimizes e(i,j) + c + M[i-1]
M[j] <- e(i,j) + c + M[i-1]
Return M[n]
segmented least squares analysis
Theorem: The dynamic programming algorithm solves the segmented least squares problem in \(O(n^3)\) time and \(O(n^2)\) space
\[ a = \frac{n \sum_i x_i y_i - (\sum_i x_i) (\sum_i y_i)}{n \sum_i x_i^2 - (\sum_i x_i)^2}, b = \frac{\sum_i y_i - a \sum_i x_i}{n} \]
Pf:
- Bottleneck is computing \(e(i,j)\) for \(O(n^2)\) pairs
- \(O(n)\) per pair using previous formula ∎
segmented least squares analysis
Theorem: The dynamic programming algorithm solves the segmented least squares problem in \(O(n^3)\) time and \(O(n^2)\) space
\[ a = \frac{n \sum_i x_i y_i - (\sum_i x_i) (\sum_i y_i)}{n \sum_i x_i^2 - (\sum_i x_i)^2}, b = \frac{\sum_i y_i - a \sum_i x_i}{n} \]
Remark: Can be improved to \(O(n^2)\) time and \(O(n)\) space
- For each \(i\): precompute cumulative sums
- \(\sum_{k=1}^i x_k\), \(\sum_{k=1}^i y_k\), \(\sum_{k=1}^i x_k^2\), \(\sum_{k=1}^i x_k y_k\)
- Using cumulative sums, can compute \(e_{ij}\) in \(O(1)\) time