Today's Topics

• Subgradient and subdifferential

• Subgradient methods

• Proximal gradient descent

Last Time

• The simplest gradient descent method deals with unconstrained and smooth problems

• Projected gradient descent extends it to constrained problems

• Today we discuss how to deal with nonsmooth problems

Subgradient

A subgradient of a convex function $f:R^d\to R$ at a point $x$ is any vector $g\in R^d$ such that

$$f\left(y\right)\ge f\left(x\right)+g^{\prime }(y-x),\forall y\in X,$$ where $X$ is the domain of $f$.

Recall that if $f$ is differentiable, then $f$ is convex if and only if

$$f\left(y\right)\ge f\left(x\right)+\left[\nabla f\right(x\left){]}^{\prime }\right(y-x),\forall x,y\in X.$$

Subdifferential

The set of all subgradients of $f$ at $x$ is called the subdifferential:

$$\partial f\left(x\right)=\{g\in R^d:g\text{is a subgradient of}f\text{at}x\}.$$

Properties

• Subgradient always exists for convex functions (subdifferential is nonempty)

• $\partial f\left(x\right)$ is closed and convex

• $f$ is differentiable at $x$ if and only if the subdifferential is a singleton containing the gradient: $\partial f\left(x\right)=\left\{\nabla f\right(x\left)\right\}$

Properties

• Linear combination: $\partial ({\alpha }_1{f}_1+{\alpha }_2{f}_2)={\alpha }_1\cdot \partial f_1+{\alpha }_2\cdot \partial f_2$, where ${\alpha }_1,{\alpha }_2\ge 0$, and for two sets $C_1,C_2$, $\alpha C_1+\beta C_2=\{\alpha x_1+\beta x_2:x_1\in C_1,x_2\in C_2\}$

• Affine transformation: If $g\left(x\right)=f(Ax+b)$, then $\partial g\left(x\right)=A^{\prime }\partial f(Ax+b)$

• Chain rule: Let $f$ be convex, $g$ be convex, differentiable, and nondecreasing. Denote by $h\left(x\right)=g\left(f\right(x\left)\right)$, and then $\partial h\left(x\right)=g^{\prime }\left(f\right(x\left)\right)\partial f\left(x\right)$

Examples

For $f=\parallel x\parallel$:

• If $x\ne 0$, $\partial f\left(x\right)=\{x/\parallel x\parallel \}$

• If $x=0$, $\partial f\left(x\right)=\{z:\parallel z\parallel \le 1\}$

For $f=\parallel x{\parallel }_1$:

• $\partial f\left(x\right)=\left\{\right(g_1,\dots ,g_d{)}^{\prime }:g_i\in G_i\}$, where

• If $x_i\ne 0$, $G_i=\left\{sign\right(x_i\left)\right\}$

• If $x_i=0$, $G_i=[-1,1]=\{z:|z|\le 1\}$

Examples ^[3,4]

For $f\left(X\right)={\lambda }_{max}\left(X\right)$ defined on all symmetric $d\times d$ matrices, let ${\lambda }_1\ge \cdots \ge {\lambda }_d$ be the eigenvalues and ${\gamma }_1,\dots ,{\gamma }_d$ be the associated eigenvectors. Then

$$\partial f\left(X\right)=\{T\in K:({\lambda }_1-{\lambda }_i){\gamma }_i^{\prime }T{\gamma }_i=0,i=2,\dots ,d\},$$ where $K=\{T\in R^d:T^{\prime }=T,T⪰0,tr(T)=1\}$.

Special cases:

• If ${\lambda }_1>{\lambda }_2$, then $\partial f\left(X\right)=\left\{\nabla f\right(X\left)\right\}=\left\{{\gamma }_1{\gamma }_1^{\prime }\right\}$.
• If ${\lambda }_1=\cdots ={\lambda }_r>{\lambda }_{r+1}$, then $$\partial f\left(X\right)=\{\sum _{i=1}^r{w}_i{\gamma }_i{\gamma }_i^{\prime }:w\in R^r,w\ge 0,\sum _{i=1}^r{w}_i=1\}.$$

Relation to Optimality Condition

There is a very general rule: $x^{\ast }$ is an optimal point of $f\left(x\right)$ if and only if 0 is a subgradient of $f$ at $x^{\ast }$:

$$f\left(x^{\ast }\right)=\underset{x}{min}f\left(x\right)\iff 0\in \partial f\left(x^{\ast }\right).$$

This is true even for nondifferentiable $f$.

If $f$ is differentiable, then the condition reduces to $$\nabla f\left(x^{\ast }\right)=0.$$

Subgradient Descent

Recall that gradient descent solves $$\underset{x}{min}f\left(x\right),$$ but it requires $f$ to be differentiable.

Subgradient descent loosens this condition by replacing the gradient with a subgradient: given an initial value $x^{\left(0\right)}$, iterate

$$x^{(k+1)}=x^{\left(k\right)}-{\alpha }_k\cdot g^{\left(k\right)},k=0,1,\dots ,$$

where ${\alpha }_k$ is the step size, and $g^{\left(k\right)}\in \partial f\left(x^{\left(k\right)}\right)$ is any subgradient of $f$ at $x^{\left(k\right)}$.

Projected Subgradient Descent

If the optimization problem is constrained on a closed and non-empty convex set $C$, then similar to projected gradient descent, we can use the projected subgradient descent method:

$$x^{(k+1)}=P_C(x^{\left(k\right)}-{\alpha }_k\cdot g^{\left(k\right)}),k=0,1,\dots .$$

Questions

• Does (projected) subgradient descent converge?

• How fast does it converge?

• How to pick ${\alpha }_k$?

Convergence Analysis

• Subgradient descent can be viewed as a special case of projected subgradient descent

• Their convergence properties are similar

• We show the results of projected subgradient descent for generality

A Fundamental Lemma

The projected subgradient descent iterations satisfy

$$\parallel x^{(k+1)}-x^{\ast }{\parallel }^2\le \parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2-2{\alpha }_k\left(f\right(x^{\left(k\right)})-f(x^{\ast }\left)\right)+{\alpha }_k^2\parallel g^{\left(k\right)}{\parallel }^2.$$

Proof: $$\begin{array}{cc}\parallel x^{(k+1)}-x^{\ast }{\parallel }^2& =\parallel P_C(x^{\left(k\right)}-{\alpha }_k{g}^{\left(k\right)})-x^{\ast }{\parallel }^2\\ & \le \parallel x^{\left(k\right)}-{\alpha }_k{g}^{\left(k\right)}-x^{\ast }{\parallel }^2\text{(projection is nonexpansive)}\\ & =\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2-2{\alpha }_k(x^{\left(k\right)}-x^{\ast }{)}^{\prime }{g}^{\left(k\right)}+{\alpha }_k^2\parallel g^{\left(k\right)}{\parallel }^2\end{array}$$

Subgradient satisfy $f\left(x^{\ast }\right)-f\left(x^{\left(k\right)}\right)\ge (x^{\ast }-x^{\left(k\right)}{)}^{\prime }{g}^{\left(k\right)}$, so the lemma holds.

Convergence Property

Suppose that $f:R^d\to R$ is convex and Lipschitz continuous on $C$ with constant $L$. Then

$$f\left(x_{\text{best}}^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le \frac{\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2+L^2{\sum }_{i=1}^k{\alpha }_i^2}{2{\sum }_{i=0}^k{\alpha }_i}.$$

For subgradient methods we can no longer guarantee the objective function values to be nonincreasing, so we select the best iterate $x_{\text{best}}^{\left(k\right)}$, defined by

$$f\left(x_{\text{best}}^{\left(k\right)}\right)=\underset{i=0,\dots ,k}{min}f\left(x^{\left(i\right)}\right).$$

Implications

• With a fixed step size ${\alpha }_k\equiv \alpha$, $$f\left(x_{\text{best}}^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le \frac{\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{2\alpha k}+\frac{L^2\alpha }{2}.$$

• With a diminishing step size ${\alpha }_k$ that satisfies ${\sum }_k{\alpha }_k^2<\infty$ and ${\sum }_k{\alpha }_k=\infty$, we have $f\left(x_{\text{best}}^{\left(k\right)}\right)\to f\left(x^{\ast }\right)$.

• The "best" choice is ${\alpha }_k=O\left(1/\sqrt{k}\right)$, which gives $$f\left(x_{\text{best}}^{\left(k\right)}\right)-f\left(x^{\ast }\right)\sim \frac{\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2+L^2\log \left(k\right)}{\sqrt{k}}.$$

Proof

Applying the fundamental lemma recursively, and we have

$$\parallel x^{(k+1)}-x^{\ast }{\parallel }^2\le \parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2-2\sum _{i=0}^k{\alpha }_i\left(f\right(x^{\left(i\right)})-f(x^{\ast }\left)\right)+\sum _{i=0}^k{\alpha }_i^2\parallel g^{\left(i\right)}{\parallel }^2.$$

Rearranging the terms gives

$$\begin{array}{cc}2\sum _{i=0}^k{\alpha }_i\left(f\right(x^{\left(i\right)})-f(x^{\ast }\left)\right)& \le \parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2-\parallel x^{(k+1)}-x^{\ast }{\parallel }^2+\sum _{i=0}^k{\alpha }_i^2\parallel g^{\left(i\right)}{\parallel }^2\\ & \le \parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2+L^2\sum _{i=0}^k{\alpha }_i^2.\end{array}$$

Convergence with Strong Convexity ^[5]

If in addition, $f$ is strongly convex with parameter $m>0$, then with a step size ${\alpha }_k=2/\left(m\right(k+1\left)\right)$, we have $$f\left(x_{\text{best}}^{\left(k\right)}\right)-f\left(x^{\ast }\right)\le \frac{2L^2}{m}\cdot \frac{1}{k+1}.$$

Summary

• Overall, subgradient methods have slower convergence rates than gradient descent counterparts.

• If $f$ is Lipschitz continuous, then the optimization error decays at the rate of $O\left(1/\sqrt{k}\right)$ (gradient descent is $O\left(1/k\right)$).

• If $f$ is Lipschitz continuous and strongly convex, then the optimization error decays at the rate of $O\left(1/k\right)$ (gradient descent is $O\left({\rho }^k\right)$).

Motivation

• Although subgradient methods are very simple and general in solving nonsmooth convex optimization problems

• They have undesirable convergence speed

• In many statistical models, the nonsmooth function we want to optimize has a special structure: $$F\left(x\right)=f\left(x\right)+h\left(x\right),$$ where $f$ is convex and smooth, and $h$ is convex but possibly nonsmooth

Examples

• For the Lasso problem, $F\left(\beta \right)=f\left(\beta \right)+h\left(\beta \right)$, where $f\left(\beta \right)=\frac{1}{2}\parallel y-X\beta {\parallel }^2$, and $h\left(\beta \right)=\lambda \parallel \beta {\parallel }_1$

• Constrained problems ${min}_{x\in C}f\left(x\right)$ can also be written as $F\left(x\right)=f\left(x\right)+h\left(x\right)$, where $$h\left(x\right)=I_C\left(x\right)=\{\begin{array}{cc}0,& x\in C\\ \infty ,& x\notin C\end{array}.$$

Proximal Gradient Descent

The proximal gradient descent algorithm solves ${min}_{x}F\left(x\right)$ using the following iteration scheme: given an initial value $x^{\left(0\right)}$, iterate

$$x^{(k+1)}={prox}_{\alpha h}(x^{\left(k\right)}-\alpha \cdot \nabla f(x^{\left(k\right)}\left)\right),k=0,1,\dots ,$$

where ${prox}_{\alpha h}(\cdot )$ is the proximal operator (defined later) of $h$ with step size $\alpha$.

Proximal Operator

The proximal operator of a convex function $h$ with step size $\alpha$ is defined as

$${prox}_{\alpha h}\left(x\right)=\underset{u}{\arg min}h\left(u\right)+\frac{1}{2\alpha }\parallel u-x{\parallel }^2.$$

• A Strongly convex optimization problem

• But has closed form for many $h$ functions

Examples

• If $h\left(x\right)=\parallel x\parallel$, then

$${prox}_{\alpha h}\left(x\right)=\{\begin{array}{cc}(1-\alpha /\parallel x\parallel )x,& \parallel x\parallel \ge \alpha \\ 0,& \parallel x\parallel <\alpha \end{array}.$$

• If $h\left(x\right)=\left(1/2\right)\parallel x{\parallel }^2$, then ${prox}_{\alpha h}\left(x\right)=(1+\alpha {)}^{-1}x.$

• If $h\left(x\right)=\left(1/2\right)x^{\prime }Ax+b^{\prime }x+c$, where $A$ is positive definite, then

$${prox}_{\alpha h}\left(x\right)=(I+\alpha A{)}^{-1}(x-\alpha b).$$

Examples

• If $h\left(x\right)=\parallel x{\parallel }_1$, then ${prox}_{\alpha h}\left(x\right)=S_{\alpha }\left(x\right)$, the soft-thresholding operator,

$$\left(S_{\alpha }\right(x){)}_i=\{\begin{array}{cc}{x}_i-\alpha ,& x_i>\alpha \\ 0,& |x_i|\le \alpha \\ x_i+\alpha ,& x_i<-\alpha \end{array}.$$

• If $h\left(x\right)=I_C\left(x\right)$, then ${prox}_{\alpha h}\left(x\right)=P_C\left(x\right)$, the projection operator.

• This means that proximal gradient descent reduces to projected gradient descent if $h\left(x\right)=I_C\left(x\right).$

Properties ^[6]

• If $h\left(x\right)=a\cdot g\left(x\right)+b$, $a>0$, then ${prox}_{\alpha h}\left(x\right)={prox}_{a\alpha g}\left(x\right)$

• If $h\left(x\right)=g(ax+b)$, $a\ne 0$, then ${prox}_{\alpha h}\left(x\right)=\left({prox}_{a^2\alpha g}\right(ax+b)-b)/a$

• If $h\left(x\right)=g\left(x\right)+a^{\prime }x+b$, then ${prox}_{\alpha h}\left(x\right)={prox}_{\alpha g}(x-a\alpha )$

• If $h\left(x\right)=g\left(x\right)+\left(\rho /2\right)\parallel x-a{\parallel }^2$, then ${prox}_{\alpha h}\left(x\right)={prox}_{\tilde{\alpha }g}\left(\right(\tilde{\alpha }/\alpha )x-(\rho \tilde{\alpha }\left)a\right)$, where $\tilde{\alpha }=\alpha /(1+\alpha \rho )$

Convergence Property

We first present a nice property of the proximal gradient descent algorithm. The proof can be found at [7].

Suppose $f$ is convex and $L$-smooth. Take the step size $\alpha =1/L$, and then

$$F\left(x^{(k+1)}\right)\le F\left(x^{\left(k\right)}\right),\parallel x^{(k+1)}-x^{\ast }\parallel \le \parallel x^{\left(k\right)}-x^{\ast }\parallel .$$

Convergence Property ^[7]

The main convergence theorem: Suppose $f$ is convex and $L$-smooth. Take the step size $\alpha =1/L$, and then

$$F\left(x^{\left(k\right)}\right)-F\left(x^{\ast }\right)\le \frac{L\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{2k}.$$

Convergence with Strong Convexity ^[7]

If in addition, $f$ is strongly convex with parameter $m>0$, then with a fixed step size $\alpha =1/L$, we have

$$\parallel x^{\left(k\right)}-x^{\ast }{\parallel }^2\le {(1-\frac{m}{L})}^k\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2.$$

Summary

• Proximal gradient descent matches the convergence rate of gradient descent.

• Useful when ${prox}_{\alpha h}$ can be efficiently evaluated.

• If $f$ is $L$-smooth, then the optimization error, $F\left(x^{\left(k\right)}\right)-F\left(x^{\ast }\right)$, decays at the rate of $O\left(1/k\right)$.

• If $f$ is $L$-smooth and $m$-strongly-convex, then the optimization error decays exponentially fast at the rate of $O\left({\rho }^k\right)$, where $\rho =1-m/L$.

Nesterov Acceleration

It turns out that the proximal gradient descent algorithm can be accelerated almost for free, using a somewhat magical technique.

The accelerated version: given an initial value $x^{\left(0\right)}=x^{(-1)}$, iterate

$$\begin{array}{cc}{y}^{(k+1)}& =x^{\left(k\right)}+\frac{k-1}{k+2}(x^{\left(k\right)}-x^{(k-1)}),\\ x^{(k+1)}& ={prox}_{\alpha h}(y^{(k+1)}-\alpha \cdot \nabla f(y^{(k+1)}\left)\right),k=0,1,\dots \end{array}$$

Convergence Property ^[8]

The accelerated convergence rate: Suppose $f$ is convex and $L$-smooth. Take the step size $\alpha =1/L$, and then

$$F\left(x^{\left(k\right)}\right)-F\left(x^{\ast }\right)\le \frac{2L\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{(k+1{)}^2}.$$

Convergence with Strong Convexity ^[8]

If in addition, $f$ is strongly convex with parameter $m>0$, then with $\kappa =L/m$, the iteration becomes

$$\begin{array}{cc}{y}^{(k+1)}& =x^{\left(k\right)}+\frac{\sqrt{\kappa }-1}{\sqrt{\kappa }+1}(x^{\left(k\right)}-x^{(k-1)}),\\ x^{(k+1)}& ={prox}_{\alpha h}(y^{(k+1)}-\alpha \cdot \nabla f(y^{(k+1)}\left)\right),k=0,1,\dots \end{array}$$

With a fixed step size $\alpha =1/L$, we have

$$F\left(x^{\left(k\right)}\right)-F\left(x^{\ast }\right)\le {(1-\frac{1}{\sqrt{\kappa }})}^k\left(F\right(x^{\left(0\right)})-F(x^{\ast })+\frac{m\parallel x^{\left(0\right)}-x^{\ast }{\parallel }^2}{2}).$$

Summary

• (Projected) subgradient descent is a general method to optimize a nonsmooth convex function

• However, its convergence is typically slow

• If the nonsmooth part has a simple proximal operator, then the proximal gradient descent algorithm is much preferred

• Also get accelerations "for free"

References

References