Studying "Introduction to Algorithms"

Chapter 2
2.1-3:
Linear_Search(A, n, v)
for i<-1 to n
   if A[i]=v
      return i
return NIL

Loop invariant: before the loop i, the 1 to i-1 items in the array do not contain the value "v".

Proof
Initialization: i-1=0, so this condition is trivial.
Maintenance: let's suppose before the loop i=k, the condition meets. That is the items [1... k-1] do not contain value "v". Let's consider the loop while i=k: if A[k]=v, then the loop will terminate by returning the index k. If A[k]!=v, then after this loop, the items [1...k] all do not contain value "v".
Termination: When the loop ends, if i=n+1, the return value is NILL. Since the items [1...n] all do not contain value "v", the result is expected; if i<n+1, it means that A[i]=v exists the loop. We have found the item with value "v" and returned the index we want.

2.2-1: n^3
2.2-2:
for j<- 1 to n-1
   min<- j
   for i<- j+1 to n
      if A[i]<A[min]
         min<-i
   temp<- A[j]
   A[j]<- A[min]
   A[min]<- temp

Loop invariant: in A[], the elements with index less than and equal to j-1 are in increasing sorted order and they are the j-1 least elements in the A[].

Proof
Initialization: j=1 so j-1=0 this is trivial.
Maintenance: Suppose we have this conclusion after j=k loop. Let prove it while j=k+1. While j=k+1, the outer for loop will find the minimum element between k+1 and n. After exchanging this minimum element with A[k+1], the A[k+1] is the smallest element in A[k+1, ... n].
Since A[1...k] is in increasing sorted order and they are less than A[k+1...n], after this for loop, the current A[k+1] is also lager than A[1...k]. So A[1... k+1] are in increasing sorted order. Also, A[k+1] is less than A[k+2...n], so are elements A[1...k]. So A[1...k+1] are the K+1 least elements in A[].
Termination: When the outer for loop ends, j=n. So the A[1...n-1] are in increasing sorted order. Also A[1...n-1] are less than A[n]. So the A[1...n] are all in increasing sorted order. This is what we want to achieve.

Based on the termination above, it is easy to understand why the outer loop can end with n-1.

Let tj=the time spent on min<-i for each j loop, then
T(n) = c1*n + c2*(n-1) + sum(j=1...n-1)( n-j + n-j + tj) + (c3+c4+c5)(n-1)
       =  bn+c+ n^2 -n + sum(j=1...n-1)(tj)

Best case: tj=0. This occurs while A[] is in increasing sorted order. T(n)=n^2 +bn+c
Worst case: tj=n-j. This occurs while A[] is in decreasing sorted order. T(n)=3/2*n^2+bn+c

2.2-3: n/2; n; theta(n/2)=theta(n).

2.2-4: We may simply add a testing for the increasing sorted order in font of any algorithm so that if we find it is the best case, we will have a good running time. This practise tells us that the best case running time of an algorithm makes no sense.

2.3-2:
Merge(A, p, q, r)
n1<- q-p+1
n2<- r-q
create arrays L[1..n1], R[1...n2]
for i<-1 to n1
   L[i]<-A[p+i-1]
for j<-1 to n2
   R[i]<-A[q+i]

i<-1
j<-1
k<-p
while i<n1+1 and j<n2+1
   if L[i]<R[j]
      A[k]<-L[i]
      i<-i+1
   else
      A[k]<-R[j]
      j<-j+1

if i<n1+1
   while k<r
      do A[k]<-L[i]
      k<-k+1
      i<-i+1

if j<n2+1
   while k<r
      do A[k]<-R[j]
      k<-k+1
      j<-j+1


2.3-3:
1. when n=2, T(n)=2lg2=2. The conclusion is correct.
2. let's suppose when n=2^k, T(n)=nlgn. =>T(2^k)=2^klg(2^k)
When n=2^(k+1),
T(n)=2T(n/2)+n
      =2T(2^k)+2^(k+1)
      =2*2^k*lg(2^k)+2^(k+1)
      =2^(k+1)*k+ 2^(k+1)
      =(k+1)*2^(k+1)
      =2^(k+1)*lg(2^(k+1))
      =n*lgn

2.3-4:
Insertion_Sort(A, n)
if n<2
   return
Insertion_Sort(A, n-1)
key<-A[n]
i<-n-1
while i>0 and A[i]>key
      A[i+1]<-A[i]
      i<-i-1
A[i+1]<-key

Translate into C# code:

public   static   void  Insertion_Sort( ref   int [] A,  int  n)
{
    if (n < 1)
        return;

    Insertion_Sort(ref A, n - 1);

    int i=n-2;
    int key=A[n-1];
    while (i >= 0 && A[i] > key)
    {
        A[i + 1] = A[i];
        i--;
    }

    A[i+1] = key;
}

static   void  Main( string [] args)
{
    int[] array = new int[] { 12, 35, 22, 354, 542, 124, 52, 2, 5 };
    Insertion_Sort(ref array, array.Length);
    for (int i = 0; i < array.Length-1; i++)
    {
        Console.Write(array[i]);
        Console.Write(", ");
    }
    Console.WriteLine(array[array.Length-1]);
}

Let's suppose t(n) to be the running time of the while loop for given variable n, then:
T(n)= C + T(n-1) + t(n)
In the worst case, the while loop will always terminate when i<=0. In this scenario, the t(n)= theta(n), so:
            theta(1)                    n=1
T(n)= {
            T(n-1) + theta(n)      n>1

2.3-5:
Binary_Search(A, p, r, v)
if p>r
   return NIL
q<- [p+r]/2
if A[q]=v
   return q
if A[q]>v
   return Binary_Search(A, p, q-1, v)
else 
   return Binary_Search(A, q+1, r, v)

The worst case occurs when the recurrence stops when p>r and returns NIL. Let's suppose n is the power of 2 and n=2^k.
T(n)=C+T(n/2) 
      =2C+T(n/4)
      =3C+T(n/8)
      ....
      =kC+T(n/2^k)
      =kC+T(1)
      =(k+1)C

n=2^k  => log2(n)=k

So T(n)=(log2(n)+1)C=theta(lg(n))

Chapter 4
4.1-1
Use mathematics induction. (It seems that there is no basic value condition to prove)
Let's suppose T([n/2]+1)<=clg(n/2+1)=c*lg(n+2) -c
We wanted to prove c*lg(n+2) -c +1 <=clgn
=>c*lg(n/(n+2))>=1-c
=>c*lg(2n/(n+2))>= 1
=> lg(2 - 4/(n+2))>= 1/c

Since f(n) = lg(2 - 4/(n+2)) function increases with the n, f(3) = lg(4/3).
So for any c>=1 we have f(n)>=1/c for all n>=3. That is the induction of T(n)<=n hold true for all n>=3.
So T(n)=O(lgn).

4.1-2.
Use mathematics induction.
Let's suppose T(n/2)>= clgn/2=clgn - c
T(n)=2T(n/2)+n>=2clgn-2c+ n
So for any given const c, we have T(n)>= clgn for all n>[2c]

So T(n)=omega(nlgn). So T(n)=theta(nlgn)

4.1-3
Inductive hypothesis: T(n)=O(nlg(n+1))
T(1)=1<=c*1*lg2=c for any C>=1
Let's suppose T(n/2)<= cn/2*lg(n/2+1)
T(n)<=2*cn/2*lg(n/2+1)+n=cn*lg(n/2+1)+n

To get the conclusion of cn*lg(n/2+1)+n<=cnlg(n+1) <= 1<c*lg((n+1)/(n/2+1)) <=1/c<lg((2n+2)/n+2) <=1/c<lg(2-2/(n+2)) 

2-2/(n+2) will increase with the increasing variable "n". So it will get the minimum value at n=1, which lg(2-2/(n+2))=lg4/3. So let c>=[1/lg4/3], this

Chapter 5
Exercises 5.1-2:
RANDOM(a, b)
value<-a
for i<-a to b-1
    value <- value + RANDOM(0, 1)
return value

The expected running time is Θ(b-a)

Exercises 5.1-3:
Analysis: We only have the procedure BIASED-RANDOM, which output 0 or 1 with 1-p and p probabilities. So we will run it k times to get the unbiased result. By using probability theory, this is a n-times bernoulli trial, the probability of getting x 0 and n-x 1 is C(n, x)(1-p)^x* p^(n-x). Since we do not know the value of p, it is impossible to make C(n, x)(1-p)^x* p^(n-x) to be 1/2.

Let's try some small times, n=2.
(0, 0)  (1-p)^2
(0, 1)  (1-p)*p
(1, 0)  p*(1-p)
(1, 1)  p^2

As we can see, (0, 1) and (1, 0) will appear with equal probability. Also, since these 4 events are independent, we may only check (0, 1) and (1, 0). They should appear with equal probability. We may output value 0 for (0, 1) and value 1 for (1, 0).

UNBIASED-RANDOM()
while true
    x<- BIASED-RANDOM()
    y<- BIASED-RANDOM()
    if x != y
        return x
        
Let's define the random variable Y: the times of trial the BIASED-RANDOM() will first output (0, 1) or (1, 0). This is a geometric distribution.
Let's define x=2*p(1-p)      =>   P{Y=k}= (1-x)^(k-1) * x.
So the expection of Y is:
E[Y]= sigma(k=1 to n)(k*(1-x)^(k-1) * x)
       = 1/x
       = 1/(2*p(1-p))                 

So the expected running time is Θ(1/(2*p(1-p)))

Exercises 5.2-1:
Hire exact one time means the best candidate comes first position, let's define this event as Y1. Since the best candidate can appear in all n position with equal probability, Y1=Y2=....=Yn. Also, P(Y1)+P(Y2)+....P(Yn)=1. So P(Y1)=1/n.
Hire n times means the candidate are placed in increasing order. There is only one element for this order. Since the sample space elements number is n!. So P(A)= 1/n!.

Exercises 5.2-2:
Since the first item will always be hired and the candidate with rank "n" will also be hired, we must keep that rank "n" candidate is not in position 1. Let's define the candidate in position 1 with rank "i", so all the rank above "i" candidates should reside behind the rank "n" candidate.

Let's define the position of rank "n" candidate to be "j". The event that the candidate position has value "i" is Ei. So Pr(Ei) = 1/n.(All n candidates have the equal probability at position 1). When rank "i" is at position 1, let's define event F to be rank i+1, i+2... n candidates are behind candiate rank "n". So what we wanted to get is: sigma(i=1 to n-1)(Pr(Ei)Pr(Ei|F)). The key point is to figure out Pr(Ei|F).

When rank "i" is at position 1, all other n-1 elements have permutation, this is the sample space. In this sample space, we care the candidates with rank>i: i+1, i+2....n. The good elements occur when the rank "n" candidate is the first one in the permutation. Since these n-i candidates all have the same probability to appear first in permutation, so the rank "n" candidate appears first has the probability 1/(n-i). That is Pr(Ei|F)=1/(n-i).

sigma(i=1 to n-1)(Pr(Ei)Pr(Ei|F)) = 1/n * sigma(i = 1 to n-1)(1/i) = 1/n* H(n-1). H(n) is the harmonic number.

Exercises 5.3-2:
No. Let's compute how many permutations do we get in this algorithm. For element 1, we have 2, 3, ...n for choice, so we have (n-1) possible choices. For element 2, we have (n-2) possible choices.  Generally, for element i, we have (n-i) choice. After all the algorithm, we have (n-1)(n-2)(n-3)...1 = (n-1)! permutations.

Since Professor Kelp wanted to get n!-1 permutations, while n!-1 != (n-1)!, the algorithm did not provide the correct result to Professor Kelp.

Exercises 5.3-3:
No. Le'ts also compute how many permutations do we get in the algorithm.
For any element i, we have n choice. After all the algorithm, we have n^n permutations. Although n^n> n!, this does not mean the permutation is not uniform random permutation. Let's assume this is uniform random permutation, then all the n! permutation should have the same number of times in the n^n results.
=> there must be an integer m: 1/n! = m/n^n  => m=n^n/n!  for any givin n will be true. But, this is not always true for all integer n.
For example, let's consider n=3, m=9/6=3/2 is not an integer. So our assumption is not true, the algorithm will not always compute uniform random permutation.(Note: for n=2, it will, interesting!)

Chapter 6
6.1-1:    min--2^h and max--2^(h+1) -1

6.1-2:   Let define the x to be the height of the heap. With 6.1-1, we have 2^x<=n<=2^(x+1)-1
=>   x<=log2(n)<(x+1)
=>   x<=log2(n) and x>log2(n)-1
=>   x=[log2(n)]=[lgn]

6.1-4:   Leaf node. Any non-leaf node will be larger than the leaf node, because all the nodes are distinct.

6.2, Q: How to prove that, for a n-nodes heap, the maximize sub-heap size is less than 2*n/3?
A: Let define the sub-heap size to be k, so k<n. To make k/n maximized, we should get the largest k for the sub-heap. This is because (k+i)/(n+i) is larger than k/n. Ok, we should take the sub-heap a full binary tree. Then, we should get the minimum value n, so the right sub-heap should be empty in the last level. In all, we may get a half full heap. Let's define the heap height as h. So k=1+2^1+2^2+...2^(k-1) = 2^k -1, while n= 1+2^1+2^2+...2^(k-1) + 2^(k-1) = 3*2^(k-1) -1. Since (2/3)*n= 2^k - 2/3 > 2^k -1. So k<2*n/3.

6.3, Let's caculate the running time of BUILD-MAX-HEAP(A) by using the depth d instead of height h.
Since the height of the tree is lgn, the execution time of a node in depth d is: O(lgn - d).  The maximize items in depth d is 2^d. So the execution time is sigma(0=<d<=lgn)(2^d*O(lgn-d)). Let's call it S(n), so 
S(n) = sigma(0=<d<=lgn)(O(2^d*lgn) - O(2^d * d))

sigma(0=<d<=lgn)(O(2^d*lgn)) = lgn* (2^(lgn + 1) - 1) = O((2n -1)*lgn)
sigma(0=<d<=lgn)(O(2^d*d)) = (lgn - 1)* 2^(lgn+1) +2 = O(2n*(lgn - 1) +2)

=> S(n) = O((2n -1)*lgn - 2n*(lgn - 1) +2) = O(2n - lgn +2) = O(n)

I think this is easier to understand than the caculation in the book.

Exercises 6.2-2:
MIN-HEAPIFY(A, i)
smallest<- i
if 2*i<=heap-size[A] and A[2*i] < A[smallest]
    smallest = 2*i
if (2*i+1)<=heap-size[A] and A[2*i + 1]< A[smallest]
    smallest = 2*i +1
if smallest != i
    temp = A[i]
    A[i] = A[smallest]
    A[smallest] = temp
    MIN-HEAPIFY(A, smallest)

The running time should be the same as MAX-HEAPIFY

Exercises 6.2-3: When the A[i] is larger than its children, the largest will remain as index i, so the recurrence will not occur. The MAX-HEAPIFY will return in const time. 

Exercises 6.2-4: When the i> heap-size[A]/2, it is a leaf node. Since it does not have children, the largest will remain as i, so the recurrence will not occur. The MAX-HEAPIFY will return in const time. 

Exercises 6.2-5:  
MAX-HEAPIFY-LOOP(A, i)
largest <- i
do
    if 2*i<= heap-size[A] and A[2*i] >A[largest] 
        largest<- 2*i
    if 2*i+1<= heap-size[A] and A[2*i+1] >A[largest] 
        largest<- 2*i+1
    if largest != i
        exchange(A[i], A[largest])
        i = largest
    else 
        break
while true
Exercises 6.3-2: Because the calling of MAX-HEAPIFY(A, i) must ensure the 2 sub-trees of node-i are max heaps. If we start from i=1, the heap at node-i may not be a max-heap. If we start from ⌊length[A]/2⌋ to 1, we may ensure before the calling to the MAX-HEAPIFY(A, i), heap at node-i is max heap by using loop invariant.
Exercises 6.3-3: The heap height is [lgn]. So the height h has at most 2^([lgn] - h ) = (2^[lgn])/2^h = ...

Exercises 6.4-2: 
Initialization: when i = length[A]. The conclusion is true trivially.
Maintenance: let's assume when i =k, the elements from 1 to k are the smallest k elements in the initial array and the other elements in A[k+1.. n] are sorted. After this loop step for i=k, the heap root node is placed at index k. Since this is a max heap, the root node is the max element in these k nodes max-heap. So after the loop step, the nodes from 1 to k-1 are less than node k. Since these k-1 nodes are also less than the other n-k nodes in assumption, these k-1 nodes are the smallest k-1 nodes in the array. Additionally, based on the assumption, we also know the exchanged node k is less than the other n-k nodes and the n-k nodes are in increasing sorted mode, so these new n-k+1 elements are in increasing sorted mode.
Termination: after the loop, there are only 1 final element in the heap, while the sub-array A[2...n] is sorted increasing. Also, the final node is less than any elements in A[2...n], so all n elements are in sorted increasing.

Exercises 6.4-3: 
Increasing order: although the elements are already in sorted order, the BUILD-MAX-HEAP will still convert the elements into a heap. Then, it will follow the loop to sort the entire array. This will still cost theta(n*lgn). This is based on the conclusion of Exercises 6.4-5.
Decreasing order: although the BUILD-MAX-HEAP will only cost linear time, the loop still needs n times. So it will still cost theta(n*lgn), t his is based on the conclusion of Exercises 6.4-5.

Chatper7
C# version of Quick_Sort implementation:

public   static   int  Partition( int [] A,  int  p,  int  r)
{
    int temp;
    int x = A[r];
    int i = p - 1;
    for (int j = p; j < r; j++)
    {
        if (A[j] <= x)
        {
            i++;

            //exchange the A[i] with A[j]
            temp = A[i];
            A[i] = A[j];
            A[j] = temp;
        }
    }

    //exchange the A[r] with A[i+1]
    temp = A[i + 1];
    A[i + 1] = A[r];
    A[r] = temp;

    return i + 1;
}

public   static   void  Quick_Sort( int [] A,  int  p,  int  r)
{
    if (r > p)
    {
        int q = Partition(A, p, r);
        Quick_Sort(A, p, q - 1);
        Quick_Sort(A, q + 1, r);
    }
}

static   void  Main()
{
    int[] array = new int[] { 12, 35, 22, 354, 542, 124, 52, 2, 5 };
    Quick_Sort(array, 0, array.Length - 1);

    for (int i = 0; i < array.Length - 1; i++)
    {
        Console.Write(array[i]);
        Console.Write(", ");
    }
    Console.WriteLine(array[array.Length - 1]);
}

Exercises 7.2-2:
When all the elements have the same value, in PARTITION() method, A[j]≤x will always be true. So the i and j will increase with each for loop. When the loop terminates, the i=j=r-1. The returned value is i+1=r. So we have the recurrent:
T(n)=T(n-1) + Θ(n). This equals the worst-case and T(n)= Θ(nlgn).

Exercises 7.2-3:
Let's consider the first PARTITION() method call when elements are in decreasing order, In the for loop, A[j]≤x is always false. So the i will remain as "p-1", while j will increase to r-1. The returned value is i+1 = p. Note: A[p] will be exchanged with A[r]. We have the most unbalanced partition.

Let's consider the second call to PARTITION() method. In this time, the A[r] is the largest one, while all other elements are still in decreasing order without change. During for loop, A[j]≤x is always true. So i will increase with j. After the loop terminate, i=j=r-1. Then A[r] will exchange with itself(A[i+1]). The returned index is i+1=r. We have the most unbalanced partition again.

Note: in the 3rd call to PARTITION() method, all the elements are again in the decreasing order, so we will follow the same rule as the first and secodn calls again. Since both of these 2 types of PARTITION() method call will result in the most unbalanced partition, we have T(n)=T(n-1)+Θ(n).

By resolving this recurretn, we got T(n)=Θ(n^2).

PS: In decreasing order condition, we have the most unbalanced partition alternate from the front and the end sides of the array. In increasing order condition, we have most unbalanced partition always from the end side.

Exercises 8.1-1:
To go from the root to the leaf in decision tree, we must deteremine the order of each items in the input. It requires least (n-1) comparison to determine the relative order. So the smallest possible depth in decision tree is (n-1).

Exercises 8.1-3:
Let's assume the 1/k of the n! inputs will execute linear time. From definition, there is a const value c, which makes all the execution time <=c*n. In decision tree, this means that all these 1/k inputs will have path to leaves shorter than c*n.

Decision tree is a binary tree, for all the paths shorter than c*n, these paths can reach leaf nodes number less than 2^(c*n). Our assumption should obey this conclusion. So we have n!/k <= 2^(c*n).

When k=2, we have n!<=2^(c*n +1). => lg(n!) <= c*n +1. Since lg(n!)=omega(n*lgn) and omega(n*lgn) > c*n+1 for enough large n. We get a confliction here. So k can't be 2.

When k=n, we have (n-1)!<=2^(c*n). => lg((n-)!) <= c*n. Since lg((n-1)!)=omega((n-1)*lg(n-1)) and omega((n-1)*lg(n-1)) > c*n for enough large n. We get a confliction here. So k can't be n.

When k=2^n, we have  n!<=2^(c*n +n). => lg(n!) <= (c+1)*n. Since lg(n!)=omega(n*lgn) and omega(n*lgn) > (c+1)n for enough large n. We get a confliction here. So k can't be 2^n.

Conclusion: We can see that the best case condition is very rare in the all possible inputs. Even less than 1/2^n, horrible~~~.

Exercises 8.3-3:
Let's induct on the maximum number of digitals of the all the elements d.
When d= 1, the sorting on this single digital(column) just sorted all the input elements. The proof is trivial.
Let's assume d=k works for radix sort. When d=k+1, all the k least significant columns are performed the sorting and they are in numeric increasing order based on assumption. While performing the sort on the most significant digital, if two numbers have different digital, the larger digital one should have the larger numeric value.
The only question occurs when two numbers have the same most significant digital value. When this happen, the stabability of the sorting algorithm will keep their original relative order before sorting. So their order depends on the k least significant digitals. Based on our assumption, these two numbers are already sorted for all k least significant digitals. Since they did not change relative order after the (k+1) step and they have the same K+1 digital, they are sorted in the correct relative order.