[Notes] Introduction to The Design and Analysis of Algorithms from Anany Levitin

Exercise solutions for 2nd edition

2nd Edition solutions

1. Introduction

1.1 What is an Algorithm?

Important points:

1. The non-ambiguity requirement for each step of an algorithm cannot be comprimised.
2. The range of inputs for which an algorithm works has to be specified carefully.
3. The same algorithm can be represented in several different ways.
4. There must exist several algorithms for solving the same problem.
5. Algorithms for the same problem can be based on very different ideas and can solve the problem with dramatically different speeds.

Greatest Common Divisor

Method 1:

Euclid's algorithm: for computing gcd(m, n)
Step 1: If n=0, return the value of m as the answer and stop; otherwise, proceed to Step2.
Step 2: Divide m by n and assign the value of the reminder to r.
Step 3: Assign the value of n to m and the value of r to n. Go to Step1.

ALGORITHM Euclid(m, n)
//Computes gcd(m, n) by Euclid's algorithm
//Input: Two nonnegative, not-both-zero integers m and n
//Output: Greatest common divisor of m and n
while n ≠ 0 do
  r ← m mod n 
  m ← n
  n ← r
return m

Proof of Euclid’s algorithm: for computing gcd(m, n)

Method 2:

Consecutive integer checking algorithm for computing gcd(m, n)
Step 1: Assign the value of min{m, n} to t.
Step 2: Divide m by t. If the remainder of this division is 0. go to Step3; otherwise, go to Step4.
Step 3: Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step4.
Step 4: Decrease the value of t by 1. Go to Step2.

This algorithm in the form presented, doesn’t work correctly when one of its input numbers is zero. It also underlines that it’s important to specify the set of an algorithm’s inputs explicitly and carefully.

Method 3:

Middle-school procedure for computing gcd(m, n)
Step 1: Find the prime factors of m.
Step 2: Find the prime factors of n.
Step 3: Identify all the common factors in the two prime expansions found in Step 1 and 2. (If p is a common factor occurring q1 and q2 times in m and n, respectively, it should be  repeated min{q1, q2} times.) 
Step 4: Compute the product of all the common factors and return it as the greatest common divisor of the numbers given.

Prime factors in Step 1 and 2 is not unambiguous. Also, Step 3 is not straightforward which makes Method 3 an unqualified algorithm. This is why Sieve of Eratosthenes needs to be introduced (generate consecutive primes not exceeding any given integer n > 1).

Sieve of Eratosthenes (Overview and Pseudocode part)

ALGORITHM Sieve(n)
//Implements the sieve of Eratosthenes
//Input: A positive integer n > 1
//Ouput: Array L of all prime numbers less than or equal to n
for p <- 2 to n do A[p] <- p
for p <-2 to floor(sqrt(n)) do
	if A[p] not equal to 0 //p hasn't been eliminated on previous passes
		j <- p * p
		while j <= n do
			A[j] <- 0  //mark elements as eliminated
			j <- j + p
//copy the remaining elements of A to array L of the primes
i <- 0
for p <- 2 to n do
	if A[p] not equal to 0	
		L[i] <- A[p]
		i <- i + 1
return L

Q: What is the largest number p whose multiples can still remain on the list to make further iterations of the algorithm necessary?

A: If p is a number whose multiples are being eliminated on the current pass, then the first multiple we should consider is $p^2$ because all its smaller multiples $2p,\dots ,(p-1)p$ have been eliminated on earlier passes through the list. Obviously, $p^2$ should not be greater than $n$.

Special care needs to be exercised if one or both input numbers are equal to 1: because mathematicians do not consider 1 to be a prime number, strictly speaking, the method does not work for such inputs.

1.2 Fundamentals of Algorithmic Problem Solving

• Understanding the Problem:

  Read the problem’s description carefully and ask questions if you have any doubts about the problem, do a few small examples by hand, think about special cases, and ask questions again if needed.

  An input to an algorithm specifies an instance of the problem the algorithm solves. If you fail to do this step, your algorithm may work correctly for a majority of inputs but crash on some “boundary” value. Remember a correct algorithm is not one that works most of the time, but one that works correctly for all legitimate inputs.

• Ascertaining the Capabilities of the Computational Device:

  RAM (random-access machine): instructions are executed one after another, one operation at a time, use sequential algorithms.

  New computers: execute operations concurrently, use parallel algorithms.

  Also, consider the speed and amount of memory the algorithm would take for different situations.

• Choosing between Exact and Approximate Problem Solving

• Algorithm Design Techniques

• Designing an Algorithm and Data Structures

• Methods of Specifying an Algorithm: natural language vs pseudocode

• Proving an Algorithm’s Correctness: the algorithm yields a required result for every legitimate input in a finite amount of time.

  A common technique for proving correctness is to use mathematical induction because an algorithm’s iterations provide a natural sequence of steps needed for such proofs.

  For an approximation algorithm, we usually would like to show that the error produced by the algorithm does not exceed a predefined limit.

• Analyzing an Algorithm: time and space efficiency, simplicity, generality.

  “A designer knows he has arrived at perfection not when there is no longer anything to add, but when there is no longer anything to take away.” —— Antoine de Saint-Exupery

• Coding an algorithm

  As a rule, a good algorithm is a result of repeated effort and rework.

1.3 Important Problem Types

The important problem types are sorting, searching, string processing, graph problems, combinatorial problems, geometric problems, and numerical problems.

• Sorting
  Rearrange the items of a given list in nondecreasing order according to a key.

  Although some algorithms are indeed better than others, there is no algorithm that would be the best solution in all situations.

  A sorting algorithm is called stable if it preserves the relative order of any two equal elements in its input, in-place if it does not require extra memory.

1.4 Fundamental Data Structures

Algorithms operate on data. This makes the issue of data structuring critical for efficient algorithmic problem solving. The most important elementary data structures are the array and the linked list. They are used for representing more abstract data structures such as the list, the stack, the queue/ priority queue (better implementation is based on an ingenious data structure called the heap), the graph (via its adjacency matrix or adjacency lists), the binary tree, and the set.

• Graph
  A graph with every pair of its vertices connected by an edge is called complete. A graph with relatively few possible edges missing is called dense; a graph with few edges relative to the number of its vertices is called sparse.

• Trees
  A tree is a connected acyclic graph. A graph that has no cycles but is not necessarily connected is called a forest: each of its connected components is a tree.

  The number of edges in a tree is always one less than the number of its vertices:|E| = |V| - 1. This property is necessary but not sufficient for a graph to be a tree. However, for connected graphs it is sufficient and hence provides a convenient way of checking whether a connected graph has a cycle.

  Ordered trees, ex. binary search trees. The efficiency of most important algorithms for binary search trees and their extensions depends on the tree’s height. Therefore, the following inequalities for the height h of a binary tree with n nodes are especially important for analysis of such algorithms: $$\lfloor {\log }_{2}n\rfloor \le h\le n-1$$

• Sets
  Representation of sets can be: a bit vector and list structure.

An abstract collection of objects with several operations that can be performed on them is called an abstract data type (ADT). The list, the stack, the queue, the priority queue, and the dictionary are important examples of abstract data types. Modern object-oriented languages support implementation of ADTs by means of classes.

2. Fundamentals of the Analysis of Algorithm Efficiency

Running time and memory space.

2.1 The Analysis Framework

The research experience has shown that for most problems, we can achieve much more spectacular progress in speed than in space.

• Measuring an Input’s Size
  When measuring input size for algorithms solving problems such as checking primality of a positive integer n. Here, the input is just one number, and it is this number’s magnitude that determines the input size. In such situations, it is preferable to measure size by the number b of bits in the n’s binary representation: $$b=\lfloor {\log }_{2}n\rfloor +1$$

• Units for Measuring Running Time
  The thing to do is to identify the most important operation of the algorithm, called the basic operation, the operation contributing the most to the total running time, and compute the number of times the basic operation is executed.

  Algorithms for mathematical problems typically involve some or all of the four arithmetical operations: addition, subtraction, multiplication and division. Of the four, the most time-consuming operation is division, followed by multiplication and then addition and subtraction, with the last two usually considered together.

  The established framework for the analysis of an algorithm’s time efficiency suggests measuring it by counting the number of times the algorithm’s basic operation is executed on inputs of size n.

• Orders of Growth
  logaⁿ = loga^b * logbⁿ

  Algorithms that require an exponential number of operations are practical for solving only problems of very small sizes.

• Worst-Case, Best-Case, and Average-Case Efficiencies
  If the best-case efficiency of an algorithm is unsatisfactory, we can immediately discard it without further analysis.

  The direct approach for investigating average-case efficiency involves dividing all instances of size n into several classes so that for each instance of the class the number of times the algorithm’s basic operation is executed is the same. Then a probability distribution of inputs is obtained or assumed so that the expected value of the basic operation’s count can be found.

  Amortized efficiency.

  Space efficiency is measured by counting the number of extra memory units consumed by the algorithm.

  The efficiencies of some algorithms may differ significantly for inputs of the same size. For such algorithms, we need to distinguish between the worst-case, average-case, and best-case efficiencies.

2.2 Asymptotic Notations and Basic Efficiency Classes

• O-notation
  O(g(n)) is the set of all functions with a lower or same order of growth as g(n) (to within a constant multiple, as n goes to infinity).

  Definition: A function t(n) is said to be in O(g(n)), denoted t(n) ∈ O(g(n)), if t(n) is bounded above by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n₀ such that t(n) ≤ cg(n) for all n ≥ n₀

• Ω-notation
  Ω(g(n)) is the set of all functions with a higher or same order of growth as g(n) (to within a constant multiple, as n goes to infinity).

  Definition: A function t(n) is said to be in Ω(g(n)), denoted t(n) ∈ Ω(g(n)), if t(n) is bounded below by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n₀ such that t(n) ≥ cg(n) for all n ≥ n₀

• Θ-notation
  Θ(g(n)) is the set of all functions with the same order of growth as g(n) (to within a constant multiple, as n goes to infinity).

  Definition: A function t(n) is said to be in Θ(g(n)), denoted t(n) ∈ Θ(g(n)), if t(n) is bounded both above and below by some constant multiples of g(n) for all large n, i.e., if there exist some positive constant c₁ and c₂ and some nonnegative integer n₀ such that c₂g(n) ≤ t(n) ≤ c₁g(n) for all n ≥ n₀.

• Useful Property Involving the Asymptotic Notations
  If t₁(n) ∈ O(g₁(n)) and t₂(n) ∈ O(g₂(n)), then t₁(n) + t₂(n) ∈ O(max{g₁(n), g₂(n)}) (also true for other two notations).
  
  L’Hôpital’s rule:
  
  Stirling’s Formula:
  

• Basic asymptotic efficiency classes
  

2.3 Mathematical Analysis of Nonrecursive Algorithms

1. Decide on parameter (or parameters) n indicating an input’s size.

2. Identify the algorithm’s basic operation. (As a rule, it is located in the inner most loop.)

3. Check whether the number of times the basic operation is executed depends only on the size of an input. If it also depends on some additional property, the worst-case, average-case, and, if necessary, best-case efficiencies have to be investigated separately.

4. Set up a sum expressing the number of times the algorithm’s basic operation is executed.

5. Using standard formulas and rules of sum manipulation, either find a closed-form formula for the count or, at the very least, establish its order of growth.

   Mathematical Analysis of Nonrecursive Algorithms

2.4 Mathematical Analysis of Recursive Algorithms

• Method of backward substitutions
• General Plan for Analyzing the Time Efficiency of Recursive Algorithms

1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed can vary on different inputs of the same size; if it can, the worst-case, average-case, and best-case efficiencies must be investigated separately.
4. Set up a recurrence relation, with an appropriate initial condition, for the number of times the basic operation is executed.
5. Solve the recurrence or, at least, ascertain the order of growth of its solution.

• Tower of Hanoi

  To move n > 1 disks from peg 1 to peg 3 (with peg 2 as auxiliary), we first move recursively n − 1 disks from peg 1 to peg 2 (with peg 3 as auxiliary), then move the largest disk directly from peg 1 to peg 3, and, finally, move recursively n − 1 disks from peg 2 to peg 3 (using peg 1 as auxiliary). Of course, if n = 1, we simply move the single disk directly from the source peg to the destination peg.
  One should be careful with recursive algorithms because their succinctness may mask their inefficiency.

• BinRec(n): smoothness rule

ALGORITHM     BinRec(n)
//Input: A positive decimal integer n
//Output: The number of binary digits in n’s binary representation 
if n = 1 return 1
else return BinRec(floor(n/2)) + 1

2.6 Empirical Analysis of Algorithms

• General Plan for the Empirical Analysis of Algorithm Time Efficiency

1. Understand the experiment’s purpose.
2. Decide on the efficiency metric M to be measured and the measurement unit (an operation count vs. a time unit).
3. Decide on characteristics of the input sample (its range, size, and so on).
4. Prepare a program implementing the algorithm (or algorithms) for the experimentation.
5. Generate a sample of inputs.
6. Run the algorithm (or algorithms) on the sample’s inputs and record the data observed.
7. Analyze the data obtained.

• linear congruential method

ALGORITHM     Random(n, m, seed, a, b)
//Generates a sequence of n pseudorandom numbers according to the linear congruential method
//Input: A positive integer n and positive integer parameters m, seed, a, b 
//Output: A sequence r1,...,rn of n pseudorandom integers uniformly distributed among integer values between 0 and m − 1 //Note: Pseudorandom numbers between 0 and 1 can be obtained by treating the integers generated as digits after the decimal point

r0 ← seed
for i ← 1 to n do
	ri ← (a ∗ ri−1 + b) mod m

The simplicity of this pseudocode is misleading because the devil lies in the details of choosing the algorithm’s parameters. Here is a partial list of recommendations based on the results of a sophisticated mathematical analysis (see [KnuII, pp. 184–185] for details): seed may be chosen arbitrarily and is often set to the current date and time; m should be large and may be conveniently taken as 2^w, where w is the computer’s word size; a should be selected as an integer between 0.01m and 0.99m with no particular pattern in its digits but such that a mod 8 = 5; and the value of b can be chosen as 1.

3. Brute Force and Exhaustive Search

Brute force is a straightforward approach to solving a problem, usually directly based on the problem statement and definitions of the concepts involved.

3.1 Selection Sort and Bubble Sort

Selection Sort

We start selection sort by scanning the entire given list to find its smallest element and exchange it with the first element, putting the smallest element in its final position in the sorted list. Then we scan the list, starting with the second element, to find the smallest among the last $n-1$ elements and exchange it with the second element, putting the second smallest element in its final position. After $n-1$ passes, the list is sorted.

ALGORITHM SelectionSort(A[0..n − 1]) 
//Sorts a given array by selection sort
//Input: An array A[0..n − 1] of orderable elements 
//Output: Array A[0..n − 1] sorted in nondecreasing order 
for i ← 0 to n − 2 do
	min ← i
	for j ← i + 1 to n − 1 do
		if A[j] < A[min] 
			min ← j 
	swap A[i] and A[min]

The basic operation is the key comparison $A[j]<A[min]$. The number of times it is executed depends only on the array size $n$ and is given by the following sum:$$C(n)=\sum _{i=0}^{n-2}\sum _{j=i+1}^{n-1}1=\sum _{i=0}^{n-2}(n-1-i)=\frac{(n-1)n}{2}$$

Selection sort is a $\Theta (n^2)$ algorithm on all inputs. Note, however, that the number of key swaps is only $\Theta (n)$, or, more precisely, $n-1$ (one for each repetition of the $i$ loop). This property distinguishes selection sort positively from many other sorting algorithms.

Bubble Sort

Another brute-force application to the sorting problem is to compare adjacent elements of the list and exchange them if they are out of order. By doing it repeatedly, we end up “bubbling up” the largest element to the last position on the list. The next pass bubbles up the second largest element, and so on, until after $n-1$ passes the list is sorted.

ALGORITHM BubbleSort(A[0..n − 1])
//Sorts a given array by bubble sort
//Input: An array A[0..n − 1] of orderable elements
//Output: Array A[0..n − 1] sorted in nondecreasing order
for i ← 0 to n − 2 do
	for j ← 0 to n − 2 − i do
		if A[j + 1] < A[j] 
			swap A[j] and A[j + 1]

The number of key comparisons:
:$$C(n)=\sum _{i=0}^{n-2}\sum _{j=0}^{n-2-i}1=\sum _{i=0}^{n-2}(n-1-i)=\frac{(n-1)n}{2}\in \Theta (n^2)$$
The number of key swaps, however, depends on the input. In the worst case of decreasing arrays, it is the same as the number of key comparisons:
$$S_{worst}(n)=C(n)=1+\cdots +(n-1)=\frac{(n-1)n}{2}\in \Theta (n^2)$$
A little trick: if a pass through the list makes no exchanges, the list has been sorted and we can stop the algorithm.Though the new version runs faster on some inputs, it is still in $\Theta (n^2)$ in the worst and average cases.

Highlights:A first application of the brute-force approach often results in an algorithm that can be improved with a modest amount of effort.

3.2 Sequential search and Brute-Force String Matching

Sequential search

Trick1: if we append the search key to the end of the list, the search for the key will have to be successful, and therefore we can eliminate the end of list check altogether

ALGORITHM SequentialSearch2(A[0..n], K)
//Implements sequential search with a search key as a sentinel
//Input: An array A of n elements and a search key K
//Output: The index of the first element in A[0..n − 1] whose value is equal to K or −1 if no such element is found
A[n] ← K
i ← 0
while A[i] not equal to K do
	i ← i + 1 
	if i < n return i
else return −1

Trick2: if a given list is known to be sorted: searching in such a list can be stopped as soon as an element greater than or equal to the search key is encountered.

Brute-Force String Matching

ALGORITHM BruteForceStringMatch(T[0..n − 1], P[0..m − 1]) //Implements brute-force string matching
//Input: An array T[0..n − 1] of n characters representing a text and an array P[0..m − 1] of m characters representing a pattern 
//Output: The index of the first character in the text that starts a matching substring or −1 if the search is unsuccessful for i ← 0 to n − m do
	j ← 0
	while j < m and P[j] = T[i + j] do
		j ← j + 1 
	if j = m return i
return −1

The worst case is much worse: the algorithm may have to make all $m$ comparisons before shifting the pattern, and this can happen for each of the $n-m+1$ tries. Thus, in the worst case, the algorithm makes $m(n-m+1)$ character comparisons, which puts it in the $\mathrm{O}(nm)$ class.

For a typical word search in a natural language text, however, we should expect that most shifts would happen after very few comparisons (check the example again). Therefore, the average-case efficiency should be considerably better than the worst-case efficiency. Indeed it is: for searching in random texts, it has been shown to be linear, i.e., $\Theta (n)$.

3.3 Closest-Pair Problem and Convex-Hull Problem

Closest-Pair Problem

One of the important applications of the closest-pair problem is cluster analysis in statistics.

ALGORITHM BruteForceClosestPair(P)
//Finds distance between two closest points in the plane by brute force
//Input:AlistP of n(n ≥ 2)points p1(x1,y1),...,pn(xn,yn) //Output: The distance between the closest pair of points 
d ← ∞
for i ← 1 to n − 1 do
	for j ← i + 1 to n do
		d ← min(d, sqrt((xi − xj)^2 + (yi − yj)^2)) 
		//sqrt is square root
return d

Reason: even for most integers, square roots are irrational numbers that therefore can be found only approximately. Moreover, computing such approximations is not a trivial matter.

Solution: use square instead of square root.

The basic operation of the algorithm will be squaring a number. The number of times it will be executed can be computed as follows:
$$C(n)=\sum _{i=1}^{n-1}\sum _{j=i+1}^{n}2=2\sum _{i=1}^{n-1}(n-i)=n(n-1)\in \Theta (n^2)$$

Convex-Hull Problem

Applications:

• in computer an-imation, replacing objects by their convex hulls speeds up collision detection;
• used in computing accessibility maps produced from satellite images by Geographic Information Systems;
• used for detecting outliers by some statistical techniques;
• compute a diameter of a set of points, which is the largest distance between two of the points, needs the set’s convex hull to find the largest distance between two of its extreme points
• convex hulls are important for solving many optimization problems, because their extreme points provide a limited set of solution candidates

A line segment connecting two points $p_i$ and $p_j$ of a set of $n$ points is a part of the convex hull’s boundary if and only if all the other points of the set lie on the same side of the straight line through these two points (to check whether certain points lie on the same side of the line, we can simply check whether the expression $ax+by-c$ has the same sign for each of these points). Repeating this test for every pair of points yields a list of line segments that make up the convex hull’s boundary.

Time efficiency: $\mathrm{O}(n^3)$: for each of $\frac{n(n-1)}{2}$ pairs of distinct points, we may need to find the sign of $ax+by-c$ for each of the other $n-2$ points.

3.4 Exhaustive Search

Exhaustive search is simply a brute-force approach to combinatorial problems.

• Travelling Salesman Problem

  Find the shortest tour through a given set of $n$ cities that visits each city exactly once before returning to the city where it started.

  Weighted graph -> finding the shortest Hamiltonian circuit of the graph: a cycle that passes through all the vertices of the graph exactly once.

  Get all the tours by generating all the permutations of $n-1$ intermediate cities, compute the tour lengths, and find the shortest among them. The total number of permutations needed is $\frac{1}{2}(n-1)!$ if direction is implied.

• Knapsack Problem

  Given $n$ items of known weights $w_1,w_2,\dots ,w_n$ and values $v_1,v_2,\dots ,v_n$ and a knapsack of capacity $W$, find the most valuable subset of the items that fit into the knapsack.

  Generate all the subsets of the set of $n$ items given, computing the total weight of each subset in order to identify feasible subsets. Since the number of subsets of an $n$-element set is $2^n$, the exhaustive search leads to a $\Omega (2^n)$ algorithm (exponential time), no matter how efficiently individual subsets are generated.

These two types of problems are NP-hard problems. No polynomial-time algorithm is known for any NP- hard problem.

• Assignment Problem

  There are $n$ people who need to be assigned to execute $n$ jobs, one person per job. (That is, each person is assigned to exactly one job and each job is assigned to exactly one person.) The cost that would accrue if the $ith$ person is assigned to the $jth$ job is a known quantity $C[i,j]$ for each pair $i,j=1,2,\dots ,n$. The problem is to find an assignment with the minimum total cost.

  The number of permutations to be considered for the general case of the assignment problem is $n!$, there is a much more efficient algorithm for this problem called the Hungarian method.

3.5 Depth-First Search and Breadth-First Search

Depth-First Search

ALGORITHM DFS(G)
//Implements a depth-first search traversal of a given graph
//Input: Graph G = ⟨V , E⟩
//Output: Graph G with its vertices marked with consecutive integers in the order they are first encountered by the DFS traversal mark each vertex in V with 0 
//0 as a mark of being “unvisited”
count ← 0
for each vertex v in V do
	if v is marked with 0 
		dfs(v)

dfs(v)
//visits recursively all the unvisited vertices connected to vertex v 
//by a path and numbers them in the order they are encountered //via global variable count
count ← count + 1; mark v with count
for each vertex w in V adjacent to v do
	if w is marked with 0 
		dfs(w)

Adjacency matrix or adjacency lists. For the adjacency matrix representation, the traversal time is in $\Theta (|V{|}^2)$, and for the adjacency list representation, it is in $\Theta (|V|+|E|)$ where $|V|$ and $|E|$ are the number of the graph’s vertices and edges, respectively.

Important elementary applications of DFS include checking connectivity and checking acyclicity of a graph.

Checking connectivity:start a DFS traversal at an arbitrary vertex and check, after the algorithm halts, whether all the vertices of the graph will have been visited. If they have, the graph is connected; otherwise, it is not connected.

Breadth-First Search

It proceeds in a concentric manner by visiting first all the vertices that are adjacent to a starting vertex, then all unvisited vertices two edges apart from it, and so on, until all the vertices in the same connected component as the starting vertex are visited. If there still remain unvisited vertices, the algorithm has to be restarted at an arbitrary vertex of another connected component of the graph.

ALGORITHM BFS(G)
//Implements a breadth-first search traversal of a given graph
//Input: Graph G = ⟨V , E⟩
//Output: Graph G with its vertices marked with consecutive integers in the order they are visited by the BFS traversal
mark each vertex in V with 0 as a mark of being “unvisited”

count ← 0
for each vertex v in V do
	if v is marked with 0 
		bfs(v)

bfs(v)
//visits all the unvisited vertices connected to vertex v
//by a path and numbers them in the order they are visited
//via global variable count
count ← count + 1; 
mark v with count and initialize a queue with v
while the queue is not empty do
	for each vertex w in V adjacent to the front vertex do 
		if w is marked with 0
			count ← count + 1; mark w with count 
			add w to the queue
	remove the front vertex from the queue

Do notice that in the for loop of bfs(v) function, it says < for each vertex w in V adjacent to the front vertex do >. Also, we have < remove the front vertex from the queue >. Actually, those vertices two edges from the starting vertex are those one edge from starting vertex’s adjacent vertices.

4. Decrease-and-Conquer

It is based on exploiting the relationship between a solution to a given instance of a problem and a solution to its smaller instance.

Once such a relationship is established, it can be exploited either top down or bottom up. The former leads naturally to a recursive implementation, although, as one can see from several examples in this chapter, an ultimate implementation may well be non-recursive. The bottom-up variation is usually implemented iteratively, starting with a solution to the smallest instance of the problem; it is called sometimes the incremental approach.

• decrease by a constant

  The size of an instance is reduced by the same constant (typically, 1) on each iteration of the algorithm.

• decrease by a constant factor

  Reduce a problem instance by the same constant factor (typically, 2) on each iteration of the algorithm.

  Efficient but got few examples.

• variable size decrease

  The size-reduction pattern varies from one iteration of an algorithm to another. Euclid’s algorithm for computing the greatest common divisor provides a good example of such a situation.

4.1 Insertion Sort

Starting with $A[1]$ and ending with $A[n-1]$, $A[i]$ is inserted in its appropriate place among the first $i$ elements of the array that have been already sorted.

ALGORITHM Insertion Sort(A[0..n − 1]) 
//Sorts a given array by insertion sort
//Input: An array A[0..n − 1] of n orderable elements 
//Output: Array A[0..n − 1] sorted in nondecreasing order 

for i ← 1 to n − 1 do
	v ← A[i]
	j ← i − 1
	while j ≥ 0 and A[j] > v do
		A[j + 1] ← A[j]
		j ← j − 1 
	A[j + 1] ← v

The basic operation of the algorithm is the key comparison $A[j]>v$.
$$C_{worst}(n)=\sum _{i=1}^{n-1}\sum _{j=0}^{i-1}1=\sum _{i=1}^{n-1}i=\frac{(n-1)n}{2}\in \Theta (n^2)$$$$C_{best}(n)=\sum _{i=1}^{n-1}1=n-1\in \Theta (n)$$
A rigorous analysis of the algorithm’s average-case efficiency is based on investigating the number of element pairs that are out of order. It shows that on randomly ordered arrays, insertion sort makes on average half as many comparisons as on decreasing arrays:$$C_{avg}(n)\approx \frac{n^2}{4}\in \Theta (n^2)$$

This twice-as-fast average-case performance coupled with an excellent efficiency on almost-sorted arrays makes insertion sort stand out among its principal competitors among elementary sorting algorithms, selection sort and bubble sort.

Its extension named shellsort, after its inventor D. L. Shell [She59], gives us an even better algorithm for sorting moderately large files.

Exercises 4.1

Ferrying soldiers: A detachment of $n$ soldiers must cross a wide and deep river with no bridge in sight. They notice two 12-year-old boys playing in a rowboat by the shore. The boat is so tiny, however, that it can only hold two boys or one soldier. How can the soldiers get across the river and leave the boys in joint possession of the boat? How many times need the boat pass from shore to shore?

A:

The algorithm is:
1. Drop one of the boy on the other side of the shore
2. Let the other boy bring back the boat
3. Remove the boy and place a soldier to reach the other side.
4. Let the small boy bring back the boat.

Repeat the process for every soldier

Marking cells: Design an algorithm for the following task. For any even $n$, mark $n$ cells on an infinite sheet of graph paper so that each marked cell has an odd number of marked neighbours. Two cells are considered neighbors if they are next to each other either horizontally or vertically but not diagonally. The marked cells must form a contiguous region, i.e., a region in which there is a path between any pair of marked cells that goes through a sequence of marked neighbors. [Kor05]

A: refer to [Algorithmic Puzzles] for detailed solution and analysis.

4.2 Topological Sorting

Example: consider a set of five required courses {C1, C2, C3, C4, C5} a part-time student has to take in some degree program. The courses can be taken in any order as long as the following course prerequisites are met: C1 and C2 have no prerequisites, C3 requires C1 and C2, C4 requires C3, and C5 requires C3 and C4. The student can take only one course per term. In which order should the student take the courses?

The topological sorting problem has a solution if and only if it is a dag (directed acyclic graph, i.e., no back edges)

Algorithm 1:

The first algorithm is a simple application of depth-first search: perform a DFS traversal and note the order in which vertices become dead-ends (i.e., popped off the traversal stack). Reversing this order yields a solution to the topological sorting problem, provided, of course, no back edge has been encountered during the traversal. If a back edge has been encountered, the digraph is not a dag, and topological sorting of its vertices is impossible.

Q: Why does the algorithm work?

A: When a vertex $v$ is popped off a DFS stack, no vertex $u$ with an edge from $u$ to $v$ can be among the vertices popped off before $v$. (Otherwise, $(u,v)$ would have been a back edge.) Hence, any such vertex $u$ will be listed after $v$ in the popped-off order list, and before $v$ in the reversed list.

Algorithm 2:

Based on a direct implementation of the decrease-(by one)-and-conquer technique: repeatedly, identify in a remaining digraph a source, which is a vertex with no incoming edges, and delete it along with all the edges outgoing from it. (If there are several sources, break the tie arbitrarily. If there are none, stop because the problem cannot be solved) The order in which the vertices are deleted yields a solution to the topological sorting problem.

The topological sorting problem may have several alternative solutions.

Q: prove a non-empty dag must have at least one source.

A:

Exercises 4.2

Q: Can one use the order in which vertices are pushed onto the DFS stack (instead of the order they are popped off it) to solve the topological sorting problem?

A:

Topological Sorting from GeeksforGeeks

4.3 Algorithms for Generating Combinatorial Objects

The number of combinatorial objects typically grows exponentially or even faster as a function of the problem size.

Generating Permutations

Decrease-by-one technique: for the problem of generating all $n!$ permutations of $1,...,n$. The smaller-by-one problem is to generate all $(n-1)!$ permutations. Assuming that the smaller problem is solved, we can get a solution to the larger one by inserting $n$ in each of the $n$ possible positions among elements of every permutation of $n-1$ elements.

We can insert $n$ in the previously generated permutations either left to right or right to left. It turns out that it is beneficial to start with inserting $n$ into $12...(n-1)$ by moving right to left and then switch direction every time a new permutation of $1,...,n-1$ needs to be processed.

It satisfies the minimal-change requirement: each permutation can be obtained from its immediate predecessor by exchanging just two elements in it.

Algorithm 1: bottom-up

Algorithm 2:

Algorithm 3:

Algorithm 3: minimum-movement

ALGORITHM HeapPermute(n)
//Implements Heap’s algorithm for generating permutations
//Input: A positive integer n and a global array A[1..n] //Output: All permutations of elements of A
if n = 1
	write A
else
	for i ← 1 to n do 
		HeapPermute(n − 1) 
		if n is odd
			swap A[1] and A[n] 
		else 
			swap A[i] and A[n]

Generating Subsets

Implementation 1: squashed order

Implementation 2: lexicographic order

Implementation 3: binary reflected Gray code

A minimal-change algorithm for generating bit strings so that every one of them differs from its immediate predecessor by only a single bit.

Recursive version

Non-recursive version

Start with the $n$-bit string of all $0$’s. For $i=1,2,...,2n-1$, generate the $ith$ bit string by flipping bit $b$ in the previous bit string, where $b$ is the position of the least significant $1$ in the binary representation of $i$.

Take n=3 as an example:
initialise with $000$
$i=1\to 001$, $b=3th$, $000\to 001$
$i=2\to 010$, $b=2rd$, $001\to 011$
$i=3\to 011$, $b=3th$, $011\to 010$
$i=4\to 100$, $b=1st$, $010\to 110$
$i=5\to 101$, $b=3th$, $110\to 111$
$i=6\to 110$, $b=2rd$, $111\to 101$
$i=7\to 111$, $b=3th$, $101\to 100$

Exercises 4.3

Q: What simple trick would make the bit string-based algorithm generate subsets in squashed order?

A: Reverse each bit string, for example: 001 -> 100, 010 -> 010, 011 -> 110, etc.

Q: Fair attraction In olden days, one could encounter the following attraction at a fair. A light bulb was connected to several switches in such a way that it lighted up only when all the switches were closed. Each switch was controlled by a push button; pressing the button toggled the switch, but there was no way to know the state of the switch. The object was to turn the light bulb on. Design an algorithm to turn on the light bulb with the minimum number of button pushes needed in the worst case for n switches.

A: Use Gray code non-recursive solution. Because Gray code is cyclic, whatever the state it is for now, it will finally move to all 0 state.

Formula:$$e=1+\frac{1}{1!}+\frac{1}{2!}+\cdots +\frac{1}{n!}$$

4.4 Decrease-by-a-Constant-Factor Algorithms

Usually run in logarithmic time.

Binary Search

Search in a sorted array.

It works by comparing a search key $K$ with the array’s middle element $A[m]$. If they match, the algorithm stops; otherwise, the same operation is repeated recursively for the first half of the array if $K<A[m]$, and for the second half if $K>A[m]$.

The basic operation is the comparison between the search key and an element of the array. We consider three-way comparisons here.
$$C_{worst}(n)=C_{worst}(\lfloor n/2\rfloor )+1forn>1,C_{worst}(1)=1$$$$C_{worst}(n)=\lfloor {\log }_{2}n\rfloor )+1=\lceil {\log }_{2}n+1\rceil \in \Theta (\log n)$$The average number of key comparisons made by binary search is only slightly smaller than that in the worst case:$$C_{avg}(n)\approx {\log }_{2}n$$$$C_{avg}^{yes}(n)\approx {\log }_{2}n-1$$$$C_{avg}^{no}(n)\approx {\log }_2(n+1)$$

Josephus Problem

$$J(2k)=2J(k)-1$$$$J(2k+1)=2J(k)+1$$The most elegant form of the closed-form answer involves the binary representation of size $n$: $J(n)$ can be obtained by a 1-bit cyclic shift left of $n$ itself:$$J(6)=J(110_2)=101_2=5,J(7)=J(111_2)=111_2=7$$.

Exercises 4.4

Q: Cutting a Stick A stick $100$ units long needs to be cut into $100$ unit pieces. What is the minimum number of cuts required if you are allowed to cut several stick pieces at the same time? Also outline an algorithm that performs this task with the minimum number of cuts for a stick of $n$ units long.

A:


Q: An array $A[0..n-2]$ contains $n-1$ integers from $1$ to $n$ in increasing order. (Thus one integer in this range is missing.) Design the most efficient algorithm you can to find the missing integer and indicate its time efficiency.

A:

Find the Missing Number in a sorted array
Find the Missing Number

4.5 Variable-Size-Decrease Algorithms

Computing a Median and the Selection Problem

The selection problem is the problem of finding the $kth$ smallest element in a list of $n$ numbers. This number is called the $kth$ order statistic.

A more interesting case of this problem is for $k=\lceil n/2\rceil$, which asks to find an element that is not larger than one half of the list’s elements and not smaller than the other half which is the median.

Partitioning idea can give us an efficient solution instead of sorting the entire list and finding the $kth$ element in the non-decreasing list whose efficiency is determined by the sorting algorithm.

This is a rearrangement of the list’s elements so that the left part contains all the elements smaller than or equal to $p$, followed by the pivot $p$ itself, followed by all the elements greater than or equal to $p$.

Quickselect

Recursive version

Assume that the list is implemented as an array whose elements are indexed starting with a $0$, and $s$ is the partition’s split position.

If $s=k-1$, pivot $p$ itself is obviously the $kth$ smallest element, which solves the problem.

If $s>k-1$, the $kth$ smallest element in the entire array can be found as the $kth$ smallest element in the left part of the partitioned array.

If $s<k-1$, it can be found as the $(k-s-1)th$ smallest element in its right part.

*Should be if s=l+k-1 return A[s] Non-recursive version

The same idea can be implemented without recursion as well. For the non-recursive version, there is no need to adjust the value of $k$ but continue until $s=k-1$.

Interpolation Search

Search in a sorted array.

This algorithm assumes that the array values increase linearly, i.e., along the straight line through the points $(l,A[l])$ and $(r,A[r])$.

The accuracy of this assumption can influence the algorithm’s efficiency but not its correctness.$$x=l+\lfloor \frac{(v-A[l])(r-l)}{A[r]-A[l]}\rfloor$$

After comparing $v$ with $A[x]$, the algorithm stops if they are equal or proceeds by searching in the same manner among the elements indexed either between $l$ and $x-1$ or between $x+1$ and $r$, depending on whether $A[x]$ is smaller or larger than $v$. Thus, the size of the problem’s instance is reduced, but we cannot tell a priori by how much.

Searching and Insertion in a Binary Search Tree

In the worst case of the binary tree search, the tree is severely skewed. This happens, in particular, if a tree is constructed by successive insertions of an increasing or decreasing sequence of keys.

The Game of Nim

One-pile Nim: refer to the explanation in the book.
General solution: very interesting!

5. Divide-and-Conquer

1. A problem is divided into several subproblems of the same type, ideally of about equal size.
2. The subproblems are solved (typically recursively, though sometimes a different algorithm is employed, especially when subproblems become small enough).
3. If necessary, the solutions to the subproblems are combined to get a solution to the original problem.

Actually, the divide-and-conquer algorithm, called the pairwise summation, may substantially reduce the accumulated round-off error of the sum of numbers that can be represented only approximately in a digital computer [Hig93].

The divide-and-conquer technique is ideally suited for parallel computations, in which each subproblem can be solved simultaneously by its own processor.

More generally, an instance of size $n$ can be divided into $b$ instances of size $n/b$, with $a$ of them needing to be solved. (Here, $a$ and $b$ are constants; $a\ge 1$ and $b>1$.) Assuming that size $n$ is a power of $b$ to simplify our analysis, we get the following recurrence for the running time $T(n)$:$$T(n)=aT(n/b)+f(n)$$
where $f(n)$ is a function that accounts for the time spent on dividing an instance of size $n$ into instances of size $n/b$ and combining their solutions.

5.1 Mergesort

Explanation:

The merging of two sorted arrays can be done as follows. Two pointers (array indices) are initialized to point to the first elements of the arrays being merged. The elements pointed to are compared, and the smaller of them is added to a new array being constructed; after that, the index of the smaller element is incremented to point to its immediate successor in the array it was copied from. This operation is repeated until one of the two given arrays is exhausted, and then the remaining elements of the other array are copied to the end of the new array.

Effciency:

$C(n)=2C(n/2)+C_{merge}(n)forn>1,C(1)=0.$

$C_{worst}(n)=2C_{worst}(n/2)+n-1forn>1,C_{worst}(1)=0.$

Exact solution to the worst-case recurrence for $n=2^k:C_{worst}(n)=nlo{g}_{2}n-n+1.$

Advantages:

1. The number of key comparisons made by mergesort in the worst case comes very close to the theoretical minimum $\lceil lo{g}_{2}n!\rceil \approx \lceil nlo{g}_{2}n-1.44n\rceil$ that any general comparison-based sorting algorithm can have. For large $n$, the number of comparisons made by this algorithm in the average case turns out to be about $0.25n$ less (see [Gon91, p. 173]) and hence is also in $\Theta (nlogn)$.

2. A noteworthy advantage of mergesort over quicksort and heapsort—the two important advanced sorting algorithms to be discussed later—is its stability.

Shortcomings

1. The algorithm requires linear amount of extra storage.
2. Though merging can be done in-place, the resulting algorithm is quite complicated and of theoretical interest only.

Improvements:

1. The algorithm can be implemented bottom up by merging pairs of the array’s elements, then merging the sorted pairs, and so on.
2. Multiway mergesort: divide a list to be sorted in more than two parts, sort each recursively, and then merge them together.

Exercises 5.1

Problem 11:

5.2 Quicksort

The difference with mergesort: the entire work happens in the division stage, with no work required to combine the solutions to the subproblems (with no need to merge two subarrays into one).

Index $i$ can go out of the subarray’s bounds in the above pseudocode, we can append a “sentinel” that would prevent index $i$ from advancing beyond position $n$ to the end of the array.

Efficiency analysis:

1. Best case:

The number of key comparisons made before a partition is achieved is $n+1$ if the scanning indices cross over and $n$ if they coincide. If all the splits happen in the middle of corresponding subarrays, we will have the best case. The number of key comparisons in the best case satisfies the recurrence:
$$C_{best}(n)=2C_{best}(n/2)+nforn>1,C_{best}(1)=0.$$$$C_{best}(n)\in \Theta (nlo{g}_{2}n)$$$$C_{best}(n)=nlo{g}_{2}nforn=2k.$$
2. Worst case:

The worst case is when $A[0..n-1]$ is a strictly increasing array, the total number of key comparisons made will be equal to:

3. Average case:

A partition can happen in any position $s(0\le s\le n-1)$ after $n+1$ comparisons are made to achieve the partition. After the partition, the left and right subarrays will have $s$ and $n-1-s$ elements, respectively. Assuming that the partition split can happen in each position $s$ with the same probability $1/n$, we get the following recurrence relation:

On the average, quicksort makes only 39% more comparisons than in the best case. Moreover, its innermost loop is so efficient that it usually runs faster than mergesort and heapsort on randomly ordered arrays of non-trivial sizes.

Improvements:

Weaknesses: not stable.

5.3 Binary Tree Traversals and Related Properties

The height is defined as the length of the longest path from the root to a leaf. => the maximum of the heights of the root’s left and right subtrees plus 1. (We have to add 1 to account for the extra level of the root.)

Also note that it is convenient to define the height of the empty tree as −1.

Efficiency analysis:

Checking that the tree is not empty is the most frequently executed operation of this algorithm and this is very typical for binary tree algorithms.

Trick: Replace the empty subtrees by special nodes called external which is different from original nodes called internal.

Height algorithm makes exactly one addition for every internal node of the extended tree, and it makes one comparison to check whether the tree is empty for every internal and external node.

The number of additions is $A(n)=n$, comparison is $A(n)=2n+1$.

5.4 Multiplication of Large Integers and Strassen’s Matrix Multiplication

Multiplication of Large Integers

If additions and subtractions are included in the analysis of efficiency:$$A(n)=3A(n/2)+cnforn>1,A(1)=1.$$

Applying the Master Theorem, $A(n)\in \Theta (n^{lo{g}_{2}3})$

Strassen’s Matrix Multiplication

If only multiplication is included in the analysis:

To multiply two matrices of order $n>1$, the algorithm needs to multiply seven matrices of order $n/2$ and make $18$ additions/subtractions of matrices of size $n/2$ ; when $n=1$, no additions are made.$$A(n)=7A(n/2)+18(n/2{)}^{2}forn>1,A(1)=0.$$

According to the Master Theorem, $A(n)\in \Theta (n^{lo{g}_{2}7}).$

5.5 The Closest-Pair and Convex-Hull Problems by Divide-and-Conquer

The Closest-Pair Problem

Convex-Hull Problem

See the book.

6. Transform-and-Conquer

Instance simplification: Transformation to a simpler or more convenient instance of the same problem.
Representation change: Transformation to a different representation of the same instance.
Problem reduction: Transformation to an instance of a different problem for which an algorithm is already available.

6.1 Presorting

Three elementary sorting algorithms—selection sort, bubble sort, and insertion sort—that are quadratic in the worst and average cases, and two advanced algorithms—mergesort, which is always in $\Theta (nlogn)$, and quicksort, whose efficiency is also $\Theta (nlogn)$) in the average case but is quadratic in the worst case.

No general comparison-based sorting algorithm can have a better efficiency than $nlogn$ in the worst case, and the same result holds for the average-case efficiency.

Sorting part that will determine the overall efficiency of the algorithm. If a good sorting algorithm is used, such as mergesort, with worst-case efficiency in $\Theta (nlogn)$, the worst-case efficiency of the entire presorting-based algorithm will be also in $\Theta (nlogn)$: $$T(n)=T_{sort}(n)+T_{scan}(n)\in \Theta (nlogn)+\Theta (n)=\Theta (nlogn)$$
The running time of the algorithm will be dominated by the time spent on sorting since the remainder of the algorithm takes linear time.

Exercises 6.1

Solution reference

The from looking at the question we can say the numbers be sorted and if less than symbol appears next we have to insert the least number, if greater than symbol appears we have to insert max number and proceed as so.

#Python code
def less(syms, i, j):
    if i == j: return False
    s = '<' if i < j else '>'
    return all(c == s for c in syms[min(i,j):max(i,j)])

def order(boxes, syms):
    if not boxes:
        yield []
        return
    for x in [b for b in boxes if not any(less(syms, a, b) for a in boxes)]:
        for y in order(boxes - set([x]), syms):
            yield [x] + y

def solutions(syms):
    for idxes in order(set(range(len(syms)+1)), syms):
        yield [idxes.index(i) for i in range(len(syms)+1)]

print(list(solutions('<><<')))

All possible solutions:

[[0, 2, 1, 3, 4], 
[0, 3, 1, 2, 4], 
[0, 4, 1, 2, 3], 
[1, 2, 0, 3, 4], 
[1, 3, 0, 2, 4], 
[1, 4, 0, 2, 3], 
[2, 3, 0, 1, 4], 
[2, 4, 0, 1, 3], 
[3, 4, 0, 1, 2]]

LightsOutPuzzle

6.2 Guassian Elimination

Elementary operations to get an equivalent system with an upper-triangular coefficient matrix A:

1. exchanging two equations of the system
2. replacing an equation with its nonzero multiple
3. replacing an equation with a sum or difference of this equation and some multiple of another equation
   

Improvement:

1. First, it is not always correct: if $A[i,i]=0$, we cannot divide by it and hence cannot use the $ith$ row as a pivot for the $ith$ iteration of the algorithm

   ==> we should exchange the $ith$ row with some row below it that has a nonzero coefficient in the $ith$ column. (If the system has a unique solution, which is the normal case for systems under consideration, such a row must exist.)

2. he possibility that $A[i,i]$ is so small and consequently the scaling factor $A[j,i]/A[i,i]$ so large that the new value of $A[j,k]$ might become distorted by a round-off error caused by a subtraction of two numbers of greatly different magnitude

   ==> Partial pivoting: always look for a row with the largest absolute value of the coefficient in the $ith$ column, exchange it with the $ith$ row, and then use the new $A[i,i]$ as the $ith$ iteration’s pivot. (guarantee the magnitude of scaling factor will never exceed 1)
   
   

Applications:

1. LU Decomposition
2. Computing a Matrix Inverse
3. Computing a Determinant

6.3 Balanced Search Trees

AVL tree

Single right rotation, or R-rotation: rotate the edge connecting the root and its left child in the binary tree to the right.

Single left rotation, or L-rotation:

Double left-right rotation (LR- rotation): perform the L-rotation of the left subtree of root r followed by the R-rotation of the new tree rooted at r.

Double right-left rotation (RL-rotation)

Keep in mind that if there are several nodes with the ±2 balance, the rotation is done for the tree rooted at the unbalanced node that is the closest to the newly inserted leaf.

Efficiency analysis:

Height $h$ of any AVL tree with $n$ nodes satisfies the inequalities$$\lfloor log2n\rfloor \le h<1.4405lo{g}_2(n+2)-1.3277$$

2-3 Trees

All its leaves must be on the same level. In other words, a 2-3 tree is always perfectly height-balanced: the length of a path from the root to a leaf is the same for every leaf.

Efficiency analysis:
$$lo{g}_3(n+1)-1\le h\le lo{g}_2(n+1)-1$$

imply that the time efficiencies of searching, insertion, and deletion are all in $\Theta (logn)$ in both the worst and average case.

6.4 Heaps and Heapsort

Heap is a clever, partially ordered data structure that is especially suitable for implementing priority queues.

Operations:

1. finding an item with the highest (i.e., largest) priority
2. deleting an item with the highest priority
3. adding a new item to the multiset

Key values in a heap are ordered top down; i.e., a sequence of values on any path from the root to a leaf is decreasing (non-increasing, if equal keys are allowed).

Properties:

Bottom-up heap construction


Top-down heap construction

This insertion operation cannot require more key comparisons than the heap’s height. Since the height of a heap with $n$ nodes is about $lo{g}_{2}n$, the time efficiency of insertion is in $O(logn)$.

Deleting the root’s key from a heap:

The time efficiency of deletion is in $O(logn)$ as well.

Heapsort

6.5 Horner’s Rule and Binary Exponentiation

Horner’s Rule

The problem of computing the value of a polynomial at a given point x is important for fast Fourier transform (FFT) algorithm.

Except for its first entry, which is $a_n$, the second row is filled left to right as follows: the next entry is computed as the $x$’s value times the last entry in the second row plus the next coefficient from the first row. The final entry computed in this fashion is the value being sought.


Efficiency analysis:

Just computing this single term by the brute-force algorithm would require $n$ multiplications, whereas Horner’s rule computes, in addition to this term, $n-1$ other terms, and it still uses the same number of multiplications!

Synthetic division: Horner’s rule also has some useful byproducts. The intermediate numbers generated by the algorithm in the process of evaluating $p(x)$ at some point $x_0$ turn out to be the coefficients of the quotient of the division of $p(x)$ by $x-x_0$, and the final result, in addition to being $p(x_0)$, is equal to the remainder of this division. $$2x^4-x^3+3x^2+x-5=(x-3)\ast (2x^3+5x^2+18x+55)+160$$$$Whenx=3,2x^4-x^3+3x^2+x-5=160$$

Binary Exponentiation

Thus, after initializing the accumulator’s value to $a$, we can scan the bit string representing the exponent $n$ to always square the last value of the accumulator and, if the current binary digit is 1, also to multiply it by $a$. These observations lead to the following left-to-right binary exponentiation method of computing $a^n$.

Efficiency analysis:

$b$ is the length of the bit string representing the exponent $n$:
$$(b-1)\le M(n)\le 2(b-1),M:multiplications$$$$b-1=\lfloor log2n\rfloor$$
This algorithm is in a logarithm efficiency class, which is better than the brute-force exponentiation, which always requires $n-1$ multiplications.

Similar efficiency as the LeftRightBinaryExponentiation. But, the usefulness of both binary exponentiation algorithms is reduced somewhat by their reliance on availability of the explicit binary expansion of exponent $n$.

6.6 Problem Reduction

The practical difficulty in applying it lies, of course, in finding a problem to which the problem at hand should be reduced.

In fact, the entire idea of analytical geometry is based on reducing geometric problems to algebraic ones.

Computing the Least Common Multiple

Drawbacks for middle-school algorithm: inefficient and requires a list of consecutive primes (same as middle-school algorithm forn computing the greatest common divisor)

Counting Paths in a Graph

The number of different paths of length $k>0$ from the $ith$ vertex to the $jth$ vertex of a graph (undirected or directed) equals the $(i,j)th$ element of $A_k$ where $A$ is the adjacency matrix of the graph.

Therefore, the problem of counting a graph’s paths can be solved with an algorithm for computing an appropriate power of its adjacency matrix.

Reduction of Optimization Problems

$$minf(x)=-max[-f(x)].$$$$maxf(x)=-min[-f(x)]$$

Linear Programming

The classic algorithm for this problem is called the simplex method. Although the worst-case efficiency of this algorithm is known to be exponential, it performs very well on typical inputs. Moreover, a more recent algorithm by Narendra Karmarkar [Kar84] not only has a proven polynomial worst-case efficiency but has also performed competitively with the simplex method in empirical tests.

Integer linear programming problem: linear programming problems that limits its variables to integer values. These problems are much more difficult. There is no known polynomial-time algorithm for solving an arbitrary instance of the general integer linear programming problem and such an algorithm quite possibly does not exist.

Reduction to Graph Problems

Form a state-space graph. Thus, the transformation just described reduces the problem to the question about a path from the initial-state vertex to a goal-state vertex.

7. Space and Time Trade-Offs

input enhancement: preprocess the problem’s input
prestructuring: deals with access structuring
dynamic programming: records solutions to overlapping subproblems of a given problem in a table from which a solution to the problem in question is then obtained

The two resources—time and space—do not have to compete with each other in all design situations:

1. use a space-efficient data structure to represent a problem’s input can lead to both a better time and space efficiency.
2. The same situation arises in the manipulation of sparse matrices and sparse polynomials: if the percentage of zeros in such objects is sufficiently high, we can save both space and time by ignoring zeros in the objects’ representation and processing.

7.1 Sorting by counting

comparison-counting sort

For each element of a list to be sorted, the total number of elements smaller than this element and record the results in a table. These numbers will indicate the positions of the elements in the sorted list.

This algorithm uses a linear amount of extra space. makes the minimum number of key moves possible by placing each of them directly in their final position in a sorted array.

distribution counting

This is a linear algorithm, a better time-efficiency class than that of the most efficient sorting algorithms–mergesort, quicksort, and heapsort. But this efficiency is obtained by exploiting the specific nature of inputs.

7.2 Input Enhancement in String Matching

Brute-force algorithm:

• worst-case efficiency: $\mathrm{O}(nm)$, $m$ is the length of pattern, $n$ is the length of text
• average-case efficiency: $\mathrm{O}(n+m)$

Horspool’s Algorithm

We can precompute shift sizes and store them in a table. The table will be indexed by all possible characters that can be encountered in a text, including, for natural language texts, the space, punctuation symbols, and other special characters. The table’s entries will indicate the shift sizes computed by the formula:


Horspool’s algorithm

• Step 1 For a given pattern of length $m$ and the alphabet used in both the pattern and text, construct the shift table as described above.
• Step 2 Align the pattern against the beginning of the text.
• Step 3 Repeat the following until either a matching substring is found or the pattern reaches beyond the last character of the text. Starting with the last character in the pattern, compare the corresponding characters in the pattern and text until either all $m$ characters are matched (then stop) or a mismatching pair is encountered. In the latter case, retrieve the entry $t(c)$ from the $c$’s column of the shift table where $c$ is the text’s character currently aligned against the last character of the pattern, and shift the pattern by $t(c)$ characters to the right along the text.

Note: $i\leftarrow i+Table[T[i]]$, $T[i]$ is the text’s character aligned with the last character of the pattern.

Efficiency analysis:

• worst-case efficiency: $\mathrm{O}(nm)$
• average-case efficiecny: $\Theta (n)$

Boyer-Moore Algorithm

The worst-case efficiency when searching for the first occurrence of the pattern is linear.

7.3 Hashing

Hashing is based on the idea of distributing keys among a one-dimensional array H [0…m − 1] called a hash table. The distribution is done by computing, for each of the keys, the value of some predefined function $h$ called the hash function. This function assigns an integer between $0$ and $m-1$, called the hash address, to a key.

Open Hashing (Separate Chaining)

Keys are stored in linked lists attached to cells of a hash table. Each list contains all the keys hashed to its cell.

If the hash function distributes $n$ keys among $m$ cells of the hash table about evenly, each list will be about $n/m$ keys long. The ratio $\alpha =n/m$, called the load factor of the hash table.

The average number of pointers (chain links) inspected in successful searches, $S$, and unsuccessful searches, $U$, turns out to be:

If we do have the load factor around 1, we have an amazingly efficient scheme that makes it possible to search for a given key for, on average, the price of one or two comparisons!

the efficiency of insertion and deletion operations is identical to that of searching, and they are all $\Theta (1)$ in the average case if the number of keys $n$ is about equal to the hash table’s size $m$.

Closed Hashing (Open Addressing)

In closed hashing, all keys are stored in the hash table itself without the use of linked lists. (Of course, this implies that the table size $m$ must be at least as large as the number of keys $n$.)

Collision resolution: The simplest one—called linear probing—checks the cell following the one where the collision occurs. If that cell is empty, the new key is installed there; if the next cell is already occupied, the availability of that cell’s immediate successor is checked, and so on. Note that if the end of the hash table is reached, the search is wrapped to the beginning of the table; i.e., it is treated as a circular array.

Lazy deletion: to mark previously occupied locations by a special symbol to distinguish them from locations that have not been occupied.

Cluster
Double hashing
Rehashing

7.4 B-trees

A B-tree is a specific type of tree which, among other things, has a maximum number of children per node. The order of a B-tree is that maximum.



Searching, insertion and deletion in a B-tree can be done in $O(logn)$ time.

8. Dynamic Programming

Dynamic programming is a technique for solving problems with overlapping subproblems.

Rather than solving overlapping subproblems again and again, dynamic programming suggests solving each of the smaller subproblems only once and recording the results in a table from which a solution to the original problem can then be obtained.

Crucial step: deriving a recurrence relating a solution to the problem to solutions to its smaller subproblems.

1. Classic bottom-up version
2. Top-down variation: avoid solving unnecessary subproblems

8.1 Three Basic Examples

Time efficiency: $\Theta (n)$, Space efficiency: $\Theta (n)$

Time efficiency: $\mathrm{O}(nm)$, Space efficiency: $\Theta (n)$

Time efficiency: $\Theta (nm)$, Space efficiency: $\Theta (nm)$

Tracing the computations backward makes it possible to get an optimal path, if ties are ignored, one optimal path can be obtained in $\Theta (n+m)$ time.

8.2 The Knapsack Problem and Memory Functions

The Knapsack Problem

Time efficiency: $\Theta (nW)$, Space efficiency: $\Theta (nW)$
The time needed to find the composition of an optimal solution is in $O(n)$.

Memory Functions

Direct top-down approach: inefficient (exponential or worse)
Classic bottom-up approach: solutions to some of these smaller subproblems are often not necessary for getting a solution to the problem given

=> Memory functions