Advanced Algorithm 听课笔记（Useful Inequalities & Balls and Bins）

0x00 前言

作为学术生涯的最后一门课，选了一门据说是最难的，上下来的感觉也确实是难得不行，不太懂……
决定照着ppt和上课的笔记整理一下，以此争取达到复习的目的。
（意思是有些虽然写出来了，但自己都不见得明白，有的部分存疑后续去询问之后再做修改）

Useful Inequalities

在随机算法的问题中有大量不等式常被使用，为了在运用时能想得起来，有些甚至要背熟。

0x01 Union Bound

Randomized Algorithm - Chapter 3.2 (P45)
n个随机事件各自发生的概率之和，不小于这n个事件中至少有一个发生的概率

Let $E_i$ be a random event, then we have
$$Pr[{\cup }_{i=1}^n{E}_i]\le \sum _{i=1}^{n}Pr(E_i)$$

0x02 马尔可夫不等式 (Markov Inequality)

Let $Y$ be a random variable assuming only non-negative values. Then
$$\text{for all }t>0,Pr[Y\ge t]\le \frac{E[Y]}{t}$$

0x03 切比雪夫不等式 (Chebyshev’s Inequality)

Let $X$ be a random variable with expectation ${\mu }_X$ and standard deviation ${\sigma }_X$, then
$$\text{for any }t>0,Pr[|X-{\mu }_X|\ge t{\sigma }_X]\le \frac{1}{t^2}$$

0x04 切尔诺夫约束 (Chernoff’s Bound)

Randomized Algorithm - Chapter 4.1 (P67)
切尔诺夫约束有三种表现方式，在多个独立的泊松实验中

Let $X_1,X_2,\cdots ,X_n$ be independent Poisson trials such that,
for $1\le i\le n,Pr[X_i=1]=p_i$, where $0<p_i<1$. Then

Chernoff’s Bound(1)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }\delta >0,$$
$$Pr[X>(1+\delta )\mu ]<{\left[\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right]}^{\mu }$$

Chernoff’s Bound(2)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }0<\delta <1,$$
$$Pr[X<(1-\delta )\mu ]<{\left[\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\right]}^{\mu }$$

Chernoff’s Bound(3)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }0<\delta <1,$$
$$Pr[|X-\mu |>\delta \mu ]<2e^{-\frac{{\delta }^2}{3}\mu }$$

0x05 Prove in detail

Chebyshev’s Inequality in 0x03

Let $X$ be a random variable with expectation ${\mu }_X$ and standard deviation ${\sigma }_X$, then
$$\text{for any }t>0,Pr[|X-{\mu }_X|\ge t{\sigma }_X]\le \frac{1}{t^2}$$

$$\begin{array}{r}Pr\left(|X-{\mu }_X|\ge t{\sigma }_X\right)\\ =Pr\left((X-{\mu }_X{)}^2\ge (t{\sigma }_X{)}^2\right)\\ \text{set }Y\triangleq (X-{\mu }_X{)}^2\ge 0\\ Pr\left(Y\ge (t\sigma {)}^2\right)\le \frac{E(Y)}{(t{\sigma }_X{)}^2}\\ \because E(Y)=E\left((X-{\mu }_X{)}^2\right)={\sigma }_X^2\\ \therefore Pr\left(Y\ge (t\sigma {)}^2\right)\le \frac{{\sigma }_X^2}{(t{\sigma }_X{)}^2}=\frac{1}{t^2}\end{array}$$

Chernoff’s Bound in 0x04

Let $X_1,X_2,\cdots ,X_n$ be independent Poisson trials such that,
for $1\le i\le n,Pr[X_i=1]=p_i$, where $0<p_i<1$. Then

Chernoff’s Bound(1)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }\delta >0,$$
$$Pr[X>(1+\delta )\mu ]<{\left[\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right]}^{\mu }$$

对于随机变量 (RandomVariable):

$$\begin{array}{rl}& R.V.x_1,x_2,\cdots ,x_n\\ & Pr(X_i=1)=p_i,Pr(X_i=0)=1-p_i\\ & \mu =\sum _{i=1}^n{p}_i,X=\sum _{i=1}^n{x}_i,E(X)=\mu \\ & Pr(X>(1+\delta )\mu )\le \frac{E(X)}{(1+\delta )\mu }=\frac{1}{1+\delta }\\ =& Pr(e^{\lambda X}>e^{\lambda (1+\delta )\mu })\le \frac{E(e\lambda X)}{e^{\lambda (1+\delta )\mu }}\le \frac{e^{\mu (e^{\lambda }-1)}}{e^{\lambda (1+\delta )\mu }}\end{array}$$

令 $\lambda =ln(1+\delta )$，则上式化为${\left(\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right)}^{\mu }$，得证。

Chernoff’s Bound(2)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }0<\delta <1,$$
$$Pr[X<(1-\delta )\mu ]<{\left[\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\right]}^{\mu }$$

其中：

$$\begin{array}{rl}E(e^{-\lambda X})& =E(e^{-\lambda ({\sum }_{i=1}^n{X}_i)})\\ & =E(\prod _{i=1}^n{e}^{-\lambda X_i})=\prod _{i=1}^{n}E(e^{-\lambda X_i})\\ & =\prod _{i=1}^n(p_i\cdot e^{-\lambda }+(1-p_i))\\ & =\prod _{i=1}^n(1+p_i(e^{-\lambda }-1))\\ & =e^{\mu (e^{-\lambda }-1)}\end{array}$$

代入原式子， 有：

$$\begin{array}{rl}Pr[X<(1-\delta )\mu ]& \le \frac{E(e^{-\lambda X})}{e^{-\lambda (1-\delta )\mu }}\\ & =\frac{e^{\mu (e^{-\lambda }-1)}}{e^{-\lambda (1-\delta )\mu }}\\ & =e^{\mu (e^{-\lambda }-1+\lambda -\lambda \delta )}\end{array}$$

令 $f(\lambda )=e^{-\lambda }-1+\lambda -\lambda \delta$,
当 $f^{\prime }(\lambda )=-e^{-\lambda }+1-\delta =0$ 时, $\lambda =-\ln (1-\delta )$
故 $Pr[X<(1-\delta )\mu ]<e^{\mu f(-ln(1-\delta ))}={\left(\frac{e^{-\delta }}{(1-\delta {)}^{(1-\delta )}}\right)}^{\mu }$

Chernoff’s Bound(3)

$$\text{for }X=\sum _{i=1}^n{X}_i,\mu =E[X]=\sum _{i=1}^n{p}_i,\text{ and any }0<\delta <1,$$
$$Pr[|X-\mu |>\delta \mu ]<2e^{-\frac{{\delta }^2}{3}\mu }$$

首先去掉绝对值符号：
$$Pr[|X-\mu |>\delta \mu ]=Pr[X-\mu >\delta \mu ]+Pr[X-\mu <-\delta \mu ]$$
对于第一个部分：
$$\begin{array}{rl}Pr[X-\mu >\delta \mu ]& =Pr[X>(\delta +1)\mu ]\\ & <{\left(\frac{e^{\delta }}{(1+\delta {)}^{(1+\delta )}}\right)}^{\mu }\\ & =e^{\mu \cdot (\delta -(1+\delta )\ln (1+\delta ))}\\ & <e^{-\frac{3}{{\delta }^2}\mu }\end{array}$$
同理可证 $Pr[X-\mu <-\delta \mu ]<e^{-\frac{3}{{\delta }^2}\mu }$
$$\begin{array}{rl}Pr[|X-\mu |>\delta \mu ]& =Pr[X-\mu >\delta \mu ]+Pr[X-\mu <-\delta \mu ]\\ & <e^{-\frac{3}{{\delta }^2}\mu }+e^{-\frac{3}{{\delta }^2}\mu }\\ & =2e^{-\frac{3}{{\delta }^2}\mu }\end{array}$$
故 $Pr[|X-\mu |>\delta \mu ]<2e^{-\frac{3}{{\delta }^2}\mu }$ 得证

Balls and Bins

原先以为往盒子里放球取球只是个抽屉原理或者排列组合的问题，
高等算法里把这研究得还要更深刻一些……

0x01 Balls and Bins

$m$ balls, $n$ bins. You randomly throw each ball to some bin.
$X_i$ : number of balls in the $i$-th bin.
Let $k\triangleq max(X_1,X_2,\cdots ,X_n)$.
Question: expectation and distribution of $k$?

• $m=o(\sqrt{n})$; (Case 1)
  • prove $Pr(k>1)=o(1)$.
  • $k=1w.h.p$
• $m=\Theta (\sqrt{n})$; (Case 2, Birthday Paradox)
  • compute $Pr(k>1)$ again.
  • $k=1or2w.h.p$
• $m=n$; (Case 3)
  • find suitable $x$, such that $Pr(k\le x)=1-o(1)$
  • $k=\Theta (\frac{\ln n}{\ln \ln n})w.h.p$
• $m\ge n\ln n$; (Case 4)
  • $k=\Theta (\frac{m}{n})w.h.p$

0xFF Prove in detail

Case 1

• $m=o(\sqrt{n})$

• prove $Pr(k>1)=o(1)$.

• $k=1w.h.p$

• $m=1,Pr(k=1)=1-o(1)$

• $m=2,\{\begin{array}{l}Pr(k=1)=1-1/n\\ Pr(k=2)=1/n\end{array}$

• $m=?,Pr(k=1)=1-o(1)$

对于这个 $Pr(k=1)=1-o(1)$，我们可以等价地视作：
$$Pr(max(X_1,X_2,\cdots ,X_n)\ge 2)=o(1)$$

那么，根据 Useful Inequalities 中提到过的 Union Bound，有：
$$\begin{array}{rl}Pr(X_1\ge 2or{X}_2\ge 2or\cdots or{X}_n\ge 2)& \le \sum _{i=1}^{n}Pr(X_i\ge 2)\\ & =n\cdot Pr(X_1\ge 2)\end{array}$$

其中，
$$\begin{array}{rl}Pr(X_1\ge 2)& \le \left(\genfrac{}{}{0ex}{}{m}{2}\right){\left(\frac{1}{n}\right)}^2=\Theta (\frac{m^2}{n^2})\\ Pr(X_1\ge 2)& =\sum _{k=2}^{m}Pr(X_1=k)\\ & =\sum _{k=2}^m\left(\genfrac{}{}{0ex}{}{m}{k}\right)\cdot (\frac{1}{n}{)}^k(1-\frac{1}{n}{)}^{m-k}\\ & =1-Pr(X_1=0)-Pr(X_1=1)\\ & =1-(1-\frac{1}{n}{)}^m-m\cdot \frac{1}{n}\cdot (1-\frac{1}{n}{)}^{m-1}\\ & =\Theta (\frac{m^2}{n^2})\end{array}$$

代入原式子，故有：
$$\begin{gathered} n\cdot Pr(X_1\ge 2)=\Theta (m^2/n)=o(1) \\ \therefore m=o(\sqrt{n}) \end{gathered}$$

Case 2

• $m=\Theta (\sqrt{n})$; (Birthday Paradox)
  + compute $Pr(k>1)$ again.
  + $k=1or2w.h.p$

$$\begin{array}{rl}m=\Theta (\sqrt{n})& =c\sqrt{n}\\ Pr(X_1\ge 2)& \le \left(\genfrac{}{}{0ex}{}{m}{2}\right){\left(\frac{1}{n}\right)}^2\approx \frac{c^2}{2n}\\ Pr(k>1)& \le n\cdot Pr(X_1\ge 2)\le \frac{c^2}{2}\\ Pr(k=1)& =\frac{n-1}{n}\cdot \frac{n-2}{n}\cdot \frac{n-3}{n}\cdots \frac{n-m+1}{n}\\ & =Pr(E_1\cdots E_m),E_i\triangleq Pr(E_1)Pr(E_2|E_1)Pr(E_3|E_1{E}_2)\cdots \\ & =(1-\frac{1}{n})\cdot (1-\frac{2}{n})\cdot (1-\frac{3}{n})\cdots (1-\frac{m-1}{n})\end{array}$$

根据 Union Bound：
$$\begin{array}{rl}Pr(k=1)& =(1-\frac{1}{n})\cdot (1-\frac{2}{n})\cdot (1-\frac{3}{n})\cdots (1-\frac{m-1}{n})\\ & \ge (1-\frac{m-1}{n}{)}^{m-1}\text{ (Union Bound)}\\ & \sim (1-\frac{m-1}{n}{)}^{\frac{n}{m-1}\cdot \frac{(m-1{)}^2}{n}}\sim (\frac{1}{e}{)}^{\frac{m^2}{n}}\end{array}$$

又因为 $1-x\le e^{-x}$:
$$\begin{array}{rl}& (1-\frac{1}{n})\cdot (1-\frac{2}{n})\cdot (1-\frac{3}{n})\cdots (1-\frac{m-1}{n})\\ \le & e^{-1/n}\cdot e^{-2/n}\cdot e^{-3/n}\cdots e^{-(m-1)/n}\\ \approx & e^{-m^2/2n}<1\\ \therefore & Pr(k\ge 2)=1-Pr(k=1)\ge 1-e^{-c^2/2}\end{array}$$

而对于 $k\ge 3$时：
(这段的板书顺序较为混乱，资质愚钝足足半个小时仍无法看懂，暂且搁置)

Prepare for case 3

为了 case 3 的证明，我们需要事先准备一个阶乘的近似界
$$(\frac{m}{x}{)}^x\le (\genfrac{}{}{0ex}{}{m}{x})\le (\frac{em}{x}{)}^x$$

先证 $\left(\genfrac{}{}{0ex}{}{m}{x}\right)=\frac{m!}{x!(m-x)!}\sim \frac{m^x}{x!}$
$$\begin{array}{rl}\underset{m\to \infty }{\lim }\frac{\left(\genfrac{}{}{0ex}{}{m}{x}\right)}{\frac{m^x}{x!}}& =\underset{m\to \infty }{\lim }\frac{m(m-1)(m-2)\cdots (m-x+1)}{m^x}\\ & =\underset{m\to \infty }{\lim }1\cdot (1-\frac{1}{m})(1-\frac{2}{m})\cdots (1-\frac{x-1}{m})\\ & =1\end{array}$$

这里，我们需要引入阶乘的逼近公式：斯特林公式(Stirling’s formula):
$$n!\sim \sqrt{2\pi n}(\frac{n}{e}{)}^n$$

$$\frac{m^x}{x!}\sim \frac{m^x}{\sqrt{2\pi x}(\frac{x}{e}{)}^x}=\frac{e^x{m}^x}{\sqrt{2\pi x}{x}^x}=\frac{e^x}{\sqrt{2\pi x}}(\frac{m}{x}{)}^x\le (\frac{em}{x}{)}^x$$
并且
$$\frac{e^x}{\sqrt{2\pi x}}>1$$
所以
$$\frac{e^x}{\sqrt{2\pi x}}(\frac{m}{x}{)}^x\ge (\frac{m}{x}{)}^x$$
即
$$(\frac{m}{x}{)}^x\le (\genfrac{}{}{0ex}{}{m}{x})\le (\frac{em}{x}{)}^x$$

Case 3

• $m=n$
  + find suitable $x$, such that $Pr(k\le x)=1-o(1)$
  + $k=\Theta (\frac{\ln n}{\ln \ln n})w.h.p$

令 $x=\frac{\ln n}{\ln lnn}$，先证下界:
$$Pr(k\le x)=1-o(1)$$

即证：
$$Pr(k\ge x)=o(1)$$

于是，根据 Union Bound 有：
$$Pr(k\ge x)\le n\cdot Pr(X_1\ge x)\le n\cdot \left(\genfrac{}{}{0ex}{}{m}{x}\right){\left(\frac{1}{n}\right)}^x=n\cdot \left(\genfrac{}{}{0ex}{}{n}{x}\right){\left(\frac{1}{n}\right)}^x$$

上一小节我们通过 斯特林公式(Stirling’s formula) 得到:
$$(\frac{m}{x}{)}^x\le (\genfrac{}{}{0ex}{}{m}{x})\le (\frac{em}{x}{)}^x$$

代入，有：
$$n\cdot \left(\genfrac{}{}{0ex}{}{n}{x}\right){\left(\frac{1}{n}\right)}^x\le n\cdot {\left(\frac{en}{x}\right)}^x{\left(\frac{1}{n}\right)}^x=n\cdot {\left(\frac{e}{x}\right)}^x=o(1)$$

再证上界：
$$Pr(k\ge c\cdot x)=1-o(1)$$

即证：
$$Pr(k\le c\cdot x)=Pr(E_1\wedge \cdots \wedge E_n)$$

其中，$E_i$ 表示：
$$x_i\le c\cdot x,Y_i=\{\begin{array}{l}1,E_i\text{ 没发生}\\ 0,E_i\text{ 发生}\end{array}$$

则有：
$$Pr(k\le c\cdot x)=Pr(k\le c\cdot x)=Pr(\forall i,Y_i=0)=Pr(\sum _{i=1}^n{Y}_i=0)$$

而上式不大于：
$$Pr\left(\left|\sum _{i=1}^n-E(\sum _{i=1}^n{Y}_i)\right|\ge E(\sum _{i=1}^n{Y}_i)\right)\le \frac{{\sigma }^2({\sum }_{i=1}^n{Y}_i)}{(E({\sum }_{i=1}^n{Y}_i){)}^2}$$

(期望与方差的推导较长，暂时搁置，事后有时间再补)， 故：
$$Pr(k<cx)=Pr(Y_1+Y_2+\cdots +Y_n=0)$$
$$\le \frac{Var({\sum }_{i=1}^n{Y}_i)}{E^2({\sum }_{i=1}^n{Y}_i)}=O\left(\frac{n}{(n^{1-c}{)}^2}\right)\sim \frac{1}{n^{1/3}},\therefore c=1/3$$

$$\frac{\ln n}{3\ln \ln n}<k<\frac{\ln n}{\ln \ln n}$$

Consider the case with $n$ balls and $n$ bins,
let $X$ be the random variable of the number of empty bins. Compute $E(X)$, and the deviation between $X$ and $E(X)$.
the result should be in the form $Pr(|X-E(X)|>a)<b$

令 $Z_i$ 表示第 $i$ 个盒子里是否没有球: 没有球时为 $Z_i=1$，反之为 $Z_i=0$
则有
$$Y=\sum _{i=1}^n{Z}_i$$
$$E(Y)=E(\sum _{i=1}^n{Z}_i)=\sum _{i=1}^{n}E(Z_i)=nE(Z_1)$$
其中
$$E(Z_1)=p(Z_1=0)\cdot 1+p(Z_1=1)\cdot 0=1-(1-\frac{1}{n}{)}^n=1-e^{-1}$$
所以
$$E(X)=E(n-Y)=n-E(Y)=e^{-1}n$$
对于 $\lambda >0$
$$\mu =E[Z]=n(1-\frac{1}{n}{)}^n\sim n{e}^{-1}$$
$$Pr[|Z-\mu |\ge \lambda ]\le 2\cdot exp(-\frac{{\lambda }^2}{2n})$$

特别地, 当 $m\gg n$ 时:
$$\mu =E[Z]=n(1-\frac{1}{n}{)}^m\sim n{e}^{-m/n}$$
$$Pr[|Z-\mu |\ge \lambda ]\le 2\cdot exp(-\frac{{\lambda }^2(n-1/2)}{n^2-{\mu }^2})$$

Case 4

• $m\ge n\ln n$
  + $k=\Theta (\frac{m}{n})w.h.p$

要证：
$$Pr(k\ge c\cdot \frac{m}{n})=o(1)$$

即证：
$$Pr(x_1\ge c\frac{m}{n}or{x}_2\ge c\frac{m}{n}or\cdots or{x}_n\ge c\frac{m}{n})$$

而根据 Union Bound，
$$Pr(k\ge c\cdot \frac{m}{n})\le n\cdot Pr(x_1\ge c\frac{m}{n})$$

先证上界：
$$Pr\left(x_1\ge c\frac{m}{n}\right)\le \left(\genfrac{}{}{0ex}{}{m}{c\frac{m}{n}}\right){\left(\frac{1}{n}\right)}^{c\frac{m}{n}}\le {\left(\frac{em}{c\frac{m}{n}}\right)}^{c\frac{m}{n}}{\left(\frac{1}{n}\right)}^{c\frac{m}{n}}={\left(\frac{e}{c}\right)}^{c\frac{m}{n}}$$

由于 $m\ge n\ln n$，
$$Pr(k\ge c\frac{m}{n})={\left(\frac{e}{c}\right)}^{c\frac{m}{n}}\le {\left(\frac{e}{c}\right)}^{c\ln n}=o(1/n)$$

再证下界，根据 Chernoff’s Bound:
$$Pr\left(\left|Y_1+\cdots +Y_n-E(Y_1+\cdots +Y_n)\right|\right)\le ?$$

其中，$Y_i$ 指 $i$-th ball 扔进了第一个盒子， $X_1={\sum }_{i=1}^m{Y}_i,Y_i=\{\begin{array}{l}1,1/n\\ 0,1-1/n\end{array}$

$$Pr(|X_1-m/n|>c_1\frac{m}{n})\le 2\cdot exp(-\frac{c_1^2}{3}\cdot \frac{m}{n})\le 2\cdot exp(-\frac{c_1^2}{3}\ln n)=2\frac{1}{n^{\frac{c{1}^2}{3}}}=o(\frac{1}{n})$$