Fundamental Problems

• Numerical Linear Algebra

  • Computations on matrices and vectors

• Optimization

  • Minimizing objective functions

• Simulation/Sampling

  • Generating random numbers from statistical distributions

Scope

• Optimization is a very broad topic

• We can only cover a small subset in this course

• Focus on convex optimization problems arising from statistical and machine learning models

Scope

• We will introduce both classical and modern optimization algorithms

• Also discuss convergence results that are less common in statistics courses (but common in optimization courses)

• There is no globally "best" optimization algorithm

• There may be good algorithms for specific problems

Today's Topics

• Convex functions

• Gradient descent

• Projected gradient descent

Concepts

• Convex set

• Convex function

• Strictly convex function

• Strongly convex function

Convex Set

We only consider subsets of the Euclidean space $R^d$.

A set $C\subseteq R^d$ is convex, if

$$tx+(1-t)y\in C,\forall x,y\in C,0\le t\le 1.$$

The line segment connecting $x$ and $y$ is included in $C$.

Convex Function

$f:R^d\to R$ is convex, if its domain $X$ is a convex set, and

$$f(tx+(1-t\left)y\right)\le tf\left(x\right)+(1-t)f\left(y\right),\forall x,y\in X,0\le t\le 1.$$

The line segment between $x$ and $y$ lies above the function.

Strictly Convex Function

Similar to the definition of convex function, but

$$f(tx+(1-t\left)y\right)<tf\left(x\right)+(1-t)f\left(y\right),\forall x\ne y,0<t<1.$$

Interpretation: The line segment between $x$ and $y$ strictly lies above the function, except for the end points $x$ and $y$.

Strongly Convex Function

$f$ is strongly convex with parameter $m>0$, if $f-\left(m/2\right)\parallel x{\parallel }^2$ is convex.

Interpretation: $f$ is at least as convex as a quadratic function.

Relation

Strongly convex $\Rightarrow$ Strictly convex $\Rightarrow$ Convex

Properties

• If $f$ is differentiable and $X$, the domain of $f$, is convex, then $f$ is convex if and only if

$$f\left(y\right)\ge f\left(x\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(y-x),\forall x,y\in X.$$

• If $f$ is twice differentiable and $X$ is convex, then $f$ is convex if and only if ${\nabla }^{2}f\left(x\right)⪰0$ for all $x\in X$.

• If $f$ is twice differentiable and $X$ is convex, then $f$ is strongly convex with parameter $m>0$ if and only if ${\nabla }^{2}f\left(x\right)⪰mI$ for all $x\in X$.

Properties

• The $\alpha$-sublevel set of a convex function $f$, defined as

$$C_{\alpha }=\{x\in X:f(x)\le \alpha \},$$

is convex for any $\alpha$.

• The converse is not true! Example: $f\left(x\right)=-e^x$.

Properties

• A nonnegative weighted sum of convex functions $$f=w_1{f}_1+\cdots +w_m{f}_m$$ preserves convexity.

• Composition with an affine mapping, $g\left(x\right)=f(Ax+b)$, preserves convexity.

• Pointwise maximum and supermum, $f\left(x\right)=max\left\{f_1\right(x),\dots ,f_m(x\left)\right\}$, preserves convexity.

Jensen's Inequality

If $f$ is convex and $X$ is a random variable supported on $C$, then

$$f\left(E\right(X\left)\right)\le E\left(f\right(X\left)\right).$$

Examples

• Univariate functions

  • $e^{ax}$ for any $a\in R$
  • $x^a$ for $a\ge 1$ or $a\le 0$ with $X=R_{+}$
  • $-x^a$ for $0\le a\le 1$ with $X=R_{+}$
  • $|x{|}^a$ for $a\ge 1$ with $X=R$

• Linear functions: $f\left(x\right)=Ax+b$

• Quadratic functions: $f\left(x\right)=\left(1/2\right)x^{\prime }Ax+b^{\prime }x$ if $A⪰0$

• Norms: $\parallel x\parallel$ (any norm, not limited to Euclidean norm)

• Log-sum-exp (LSE) function: $$LSE(x_1,\dots ,x_n)=\log (e^{x_1}+\cdots +e^{x_n})$$

A Side Note on LSE

• In some sense LSE can be called the "soft max" function, since

$$max(x_1,\dots ,x_n)\le LSE(x_1,\dots ,x_n)\le max(x_1,\dots ,x_n)+\log \left(n\right)$$

• The widely-used Softmax function should better be called "soft argmax"

$$softmax(x_1,\dots ,x_n)=(\frac{e^{x_1}}{{\sum }_i{e}^{x_i}},\dots ,\frac{e^{x_n}}{{\sum }_i{e}^{x_i}})$$

• It can be verified that Softmax is the gradient of LSE

Convex Optimization Problem

A convex optimization problem is concerned with finding some $x^{\ast }$ that attains the infimum $$\underset{x\in C}{inf}f\left(x\right),$$ where $f:R^d\to R$ is a convex function with domain $X$, and $C\subseteq X$ is a convex set. Such an $x^{\ast }$, if it exists, is called a solution to the convex optimization problem.

In general, a convex optimization problem may have zero, one, or many solutions.

Concepts

• The optimal value is ${inf}_{x\in C}f\left(x\right)$

• A solution $x^{\ast }$ is also called an optimal point

• The set of all optimal points is called the optimal set

Properties

• The optimal set, if nonempty, is convex

• If the objective function $f$ is strictly convex, then the problem has at most one optimal point

• If $f$ is differentiable, then $x^{\ast }$ is an optimal point if and only if $x^{\ast }\in C$ and $$\left[\nabla f\right(x^{\ast }\left){]}^{\prime }\right(y-x^{\ast })\ge 0,\forall y\in C.$$

• If $C=X$, then $x^{\ast }$ is an optimal point if and only if $\nabla f\left(x^{\ast }\right)=0$.

Gradient Descent

Gradient descent (GD) is a well-known and widely-used technique to solve the unconstrained, smooth convex optimization problem

$$\underset{x}{min}f\left(x\right),$$

where $f$ is convex and differentiable with domain $X=R^d$.

The algorithm is simple: given an initial value $x^{\left(0\right)}$, iterate

$$x^{(k+1)}=x^{\left(k\right)}-{\alpha }_k\cdot \nabla f\left(x^{\left(k\right)}\right),k=0,1,\dots ,$$

where ${\alpha }_k$ is the step size.

Questions

• We are familiar with using GD to solve statistical models

• But in this course we need to understand GD better

• Does GD always converge?

• How fast does it converge?

• How to pick ${\alpha }_k$?

Lipschitz Continuity

We first introduce the concept of Lipschitz continuity, which plays an important role in analyzing GD.

A function $f:R^d\to R^r$ is Lipschitz continuous with constant $L>0$, if $$\parallel f\left(x\right)-f\left(y\right)\parallel \le L\parallel x-y\parallel ,\forall x,y\in R^d.$$

For example, linear functions are Lipschitz continuous, whereas quadratic functions are not.

Convergence of GD

Suppose that $f:R^d\to R$ is convex and differentiable, and assume its gradient is Lipschitz continuous with constant $L>0$, i.e., $$\parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le L\parallel x-y\parallel ,\forall x,y\in R^d.$$ Then using a fixed step size ${\alpha }_k\equiv \alpha \le 1/L$, we have $$f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le \frac{\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{2\alpha k},$$ where $f\left(x^{\ast }\right)$ is the optimal value.

This is a non-asymptotic result.

Proof

Step 1: We show that if $\nabla f\left(x\right)$ is Lipschitz continuous with constant $L>0$, then $$f\left(y\right)\le f\left(x\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(y-x)+\frac{L}{2}\parallel y-x{\parallel }^2,\forall x,y.$$

Step 2: Plugging in $y=x-\alpha \cdot \nabla f\left(x\right)$, then $$\begin{array}{cc}f\left(y\right)& \le f\left(x\right)-\alpha \left[\nabla f\right(x\left){]}^{\prime }\right[\nabla f\left(x\right)]+\frac{{\alpha }^{2}L}{2}\parallel \nabla f(x){\parallel }^2\\ & =f\left(x\right)-(1-\alpha L/2)\alpha \parallel \nabla f\left(x\right){\parallel }^2.\end{array}$$ This means that if $\alpha \le 1/L$, we have $f\left(y\right)\le f\left(x\right)-\left(\alpha /2\right)\parallel \nabla f\left(x\right){\parallel }^2$, implying that $f$ is nonincreasing along iterations.

Proof

Step 3: Since $f$ is convex, we have $$\begin{array}{cc}f\left(x^{\ast }\right)& \ge f\left(x\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(x^{\ast }-x),\\ f\left(x\right)& \le f\left(x^{\ast }\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(x-x^{\ast }).\end{array}$$

Step 4: Combining Step 2 and Step 3, $$\begin{array}{cc}f\left(y\right)& \le f\left(x^{\ast }\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(x-x^{\ast })-\frac{\alpha }{2}\parallel \nabla f(x){\parallel }^2,\\ f\left(y\right)-f\left(x^{\ast }\right)& \le \frac{1}{2\alpha }\left(2\alpha \right[\nabla f\left(x\right){]}^{\prime }(x-x^{\ast })-{\alpha }^2\parallel \nabla f\left(x\right){\parallel }^2)\\ & =\frac{1}{2\alpha }(-\parallel x-x^{\ast }-\alpha \nabla f(x){\parallel }^2+\parallel x-x^{\ast }{\parallel }^2)\\ & =\frac{1}{2\alpha }(-\parallel y-x^{\ast }{\parallel }^2+\parallel x-x^{\ast }{\parallel }^2).\end{array}$$

Proof

Step 5: This means that $$\begin{array}{cc}f\left(x^{(k+1)}\right)-f\left(x^{\ast }\right)& \le \frac{1}{2\alpha }(\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2-\parallel x^{(k+1)}-x^{\ast }{\parallel }^2),\\ \sum _{i=0}^{k-1}\left[f\right(x^{(i+1)})-f(x^{\ast }\left)\right]& \le \frac{1}{2\alpha }(\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2-\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2)\\ & \le \frac{\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{2\alpha }.\end{array}$$

Step 6: Step 2 shows that $$\sum _{i=0}^{k-1}\left[f\right(x^{(i+1)})-f(x^{\ast }\left)\right]\ge k\left[f\right(x^{\left(k\right)})-f(x^{\ast }\left)\right],$$ and then we get the final result.

Convergence with Strong Convexity ^[2]

If in addition, $f$ is strongly convex with parameter $m>0$, then with a fixed step size $\alpha \le 1/L$, we have $$\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2\le (1-m\alpha {)}^k\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2.$$

Note: Here the convergence of $\parallel x^{\left(k\right)}-x^{\ast }\parallel$ implies the convergence of $f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)$. (Why?)

$$\begin{array}{cc}f\left(x^{\left(k\right)}\right)& \le f\left(x^{\ast }\right)+\left[\nabla f\right(x^{\ast }\left){]}^{\prime }\right(x^{\left(k\right)}-x^{\ast })+\frac{L}{2}\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2\\ & =f\left(x^{\ast }\right)+\frac{L}{2}\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2.\end{array}$$

Summary

• If $f$ is $L$-smooth (i.e., $\nabla f\left(x\right)$ is Lipschitz continuous with constant $L>0$), then the optimization error, $f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)$, decays at the rate of $O\left(1/k\right)$.

• If $f$ is $L$-smooth and $m$-strongly-convex, then the optimization error decays exponentially fast at the rate of $O\left({\rho }^k\right)$ for some $\rho \in (0,1)$.

Line Search

The theory assumes that we use a fixed step size ${\alpha }_k\equiv \alpha$. In practice, the step size can be selected adaptively using the line search scheme: at each iteration $k$, find the largest possible ${\alpha }_k\le 1$ such that $$f(x^{\left(k\right)}-{\alpha }_k\nabla f(x^{\left(k\right)}\left)\right)\le f\left(x^{\left(k\right)}\right)-\beta {\alpha }_k\parallel \nabla f\left(x^{\left(k\right)}\right){\parallel }^2,$$ where $0<\beta \le 1/2$.

Constrained Optimization

• Recall that GD solves an unconstrained and smooth convex optimization problem

• However, many problems do not satisfy one or two of the conditions

• We first consider the extension to constrained problems

Constrained Optimization

Consider the constrained optimization problem $$\underset{x\in C}{min}f\left(x\right),$$ where $C$ is a closed and non-empty convex set.

Now we assume that we can compute the projection operator $P_C\left(x\right)$ easily.

Projection Operator

The projection operator itself is also an optimization problem:

$$P_C\left(x\right)=\underset{u\in C}{\arg min}\frac{1}{2}\parallel u-x{\parallel }^2.$$

However, for some convex sets $C$, computing $P_C\left(x\right)$ is trivial.

Projection is the closest point that satisfies the constraints.

Examples

• If $C=\{x\in R^d:a^{\prime }x=b\}$, then $$P_C\left(x\right)=x+(b-a^{\prime }x)a/\parallel a{\parallel }^2.$$

• If $C=\{x\in R^d:a^{\prime }x\le b\}$, then $$P_C\left(x\right)=\{\begin{array}{cc}x+(b-a^{\prime }x)a/\parallel a{\parallel }^2,& a^{\prime }x>b\\ x,& a^{\prime }x\le b\end{array}.$$

Examples

• If $C=\{x\in R^d:l_i\le x_i\le u_i,i=1,\dots ,d\}$, then $$P_C(x{)}_i=\{\begin{array}{cc}{l}_i,& x_i\le l_i\\ x_i,& l_i<x_i\le u_i\\ u_i,& x_i>u_i\end{array}.$$

• Special case: $C=\{x\in R^d:x_i\ge 0\}$, $P_C\left(x\right)=[x{]}_{+}$.

• Special case: $C=\{x\in R^d:\parallel x{\parallel }_{\infty }\le t\}$, $$P_C(x{)}_i=\{\begin{array}{cc}{x}_i,& |x_i|\le t\\ sign\left(x_i\right)t,& |x_i|>t\end{array}.$$

Examples

• If $C=\{x\in R^d:\parallel x\parallel \le t\}$, then $$P_C\left(x\right)=\{\begin{array}{cc}x,& \parallel x\parallel \le t\\ tx/\parallel x\parallel ,& \parallel x\parallel >t\end{array}.$$

• Question: How to compute $P_C\left(x\right)$, where $C=\{x\in R^d:\parallel x{\parallel }_1\le t\}$? Is it easy or hard?

• Question: How to compute $P_C\left(X\right)$, where $C=\{X\in R^d:X⪰0\}$, assuming $X$ is symmetric? Is it easy or hard? (Use the Frobenius norm in the definition)

Non-expansive Property of Projection ^[3]

An interesting property of the projection operator is that it is non-expansive.

Suppose that $C$ is a closed and non-empty convex set, then for any $x\in C$ and $y\in R^d$, $$\left(P_C\right(y)-x{)}^{\prime }(P_C\left(y\right)-y)\le 0.$$ This implies that $$\parallel x-P_C\left(y\right)\parallel \le \parallel x-y\parallel .$$

Projected Gradient Descent

Projected gradient descent (PGD) modifies GD by adding a projection step at each iteration:

$$x^{(k+1)}=P_C(x^{\left(k\right)}-{\alpha }_k\cdot \nabla f(x^{\left(k\right)}\left)\right),k=0,1,\dots .$$

Computational cost almost identical to GD if the projection operator is cheap.

Convergence Property ^[3]

Suppose that $f:R^d\to R$ is convex and $L$-smooth on $C$. Then using a fixed step size ${\alpha }_k\equiv \alpha =1/L$, we have $$f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le \frac{3L\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2+f\left(x^{\left(0\right)}\right)-f\left(x^{\ast }\right)}{k+1},$$ where $f\left(x^{\ast }\right)$ is the optimal value.

Proof

Although the convergence result is similar to that of GD, the proof here is much harder.

We only show the proof of a key intermediate result, $$f\left(x^{(k+1)}\right)\le f\left(x^{\left(k\right)}\right)-\frac{L}{2}\parallel x^{(k+1)}-x^{\left(k\right)}{\parallel }^2.$$

A Complete proof can be found at https://www.stats.ox.ac.uk/~rebeschi/teaching/AFoL /20/material/lecture09.pdf

Proof

Step 1: Same as GD. $$f\left(z\right)\le f\left(x\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(z-x)+\frac{L}{2}\parallel z-x{\parallel }^2,\forall x,z\in C.$$

Step 2: Let $y=x-\alpha \cdot \nabla f\left(x\right)$ and $z=P_C\left(y\right)$, and then we have $\nabla f\left(x\right)=\left(1/\alpha \right)(x-y)$, and $$f\left(z\right)\le f\left(x\right)+{\alpha }^{-1}(x-y{)}^{\prime }(z-x)+(L/2)\parallel z-x{\parallel }^2.$$ Also note that $$\begin{array}{cc}(x-y{)}^{\prime }(z-x)& =(z-y{)}^{\prime }(z-x)+(x-z{)}^{\prime }(z-x)\\ & =(z-y{)}^{\prime }(z-x)-\parallel x-z{\parallel }^2.\end{array}$$

Proof

Step 3: Since $z=P_C\left(y\right)$, by the non-expansive property we have $$(z-x{)}^{\prime }(z-x)\le 0,$$ and hence $$f\left(z\right)\le f\left(x\right)-{\alpha }^{-1}\parallel x-z{\parallel }^2+\left(L/2\right)\parallel z-x{\parallel }^2.$$

Take $x=x^{\left(k\right)}$, $z=x^{(k+1)}$, $\alpha =1/L$, and we have $$f\left(x^{(k+1)}\right)\le f\left(x^{\left(k\right)}\right)-\frac{L}{2}\parallel x^{(k+1)}-x^{\left(k\right)}{\parallel }^2.$$ This shows that $f\left(x^{\left(k\right)}\right)$ is nonincreasing.

Convergence with Strong Convexity ^[2]

If in addition, $f$ is strongly convex with parameter $m>0$ on $C$, then with a fixed step size $\alpha =1/L$, we have $$f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le {(1-\frac{m}{L})}^k\left[f\right(x^{\left(0\right)})-f(x^{\ast }\left)\right].$$

Summary

• PGD has the same convergence rates as GD in the following two cases.

• If $f$ is $L$-smooth, then the optimization error, $f\left(x^{\left(k\right)}\right)-f\left(x^{\ast }\right)$, decays at the rate of $O\left(1/k\right)$.

• If $f$ is $L$-smooth and $m$-strongly-convex, then the optimization error decays exponentially fast at the rate of $O\left({\rho }^k\right)$, where $\rho =1-m/L$.

References