Today's Topics

• Langevin algorithm

• Hamiltonian Monte Carlo

Langevin Dynamics

The Langevin dynamics, or Langevin diffusion process, is the solution to the following stochastic differential equation (SDE) on $R^d$:

$$d{X}_t=-\nabla V\left(X_t\right)dt+\sqrt{2}d{B}_t,X_0\sim p_0\left(x\right),$$

where $\nabla V(\cdot )$ is the gradient of some smooth function $V:R^d\to R$, and $B_t$ is the $d$-dimensional Brownian motion.

Langevin Dynamics

The SDE can be understood as the continuous-time limit of the following iteration:

$${\tau }^{-1}(X_{t+\tau }-X_t)=-\nabla V\left(X_t\right)+\sqrt{2}{\tau }^{-1}(B_{t+\tau }-B_t).$$

It is known that $B_{t+\tau }-B_t\sim N(0,\tau I)$, so we can rewrite the formula above as

$$X_{t+\tau }-X_t=-\tau \nabla V\left(X_t\right)+\sqrt{2\tau }{\xi }_{t+\tau },$$

where ${\xi }_{t+\tau }\sim N(0,I)$ and is independent of $X_t$.

It is very similar to the gradient descent iteration, except for the additional noise term.

Invariant Distribution

An important property of the Langevin dynamics is that under mild conditions, it has an explicit invariant distribution. Theorem 2.1 of [1]:

Suppose that $\nabla V(\cdot )$ is continuously differentiable, and there exist some constants $a,b,C<\infty$ such that

$$⟨-\nabla V\left(x\right),x⟩\le a\parallel x{\parallel }^2+b,\forall \parallel x\parallel >C.$$

Let $p_t\left(x\right)$ be the density function of $X_t$. Then $\parallel p_t-\pi {\parallel }_{TV}\to 0$, where $\parallel f-g{\parallel }_{TV}$ is the total variation distance between two densities $f$ and $g$, and $\pi \left(x\right)\propto \exp \{-V(x\left)\right\}$.

Invariant Distribution

This means that no matter what the initial distribution $p_0\left(x\right)$ is, $X_t$ will eventually follow the distribution $\pi \left(x\right)\propto \exp \{-V(x\left)\right\}$.

In other words, if we are interested in sampling from some target distribution $\pi \left(x\right)\propto \exp \{-V(x\left)\right\}$, we can simulate the Langevin diffusion process of some particles, starting from an arbitrary initial distribution.

This is the foundation of the Langevin Monte Carlo algorithm.

Convergence Rate ^[2]

In addition, if $V(\cdot )$ is $m$-strongly-convex and $L$-smooth, then

$$\parallel p_t-\pi {\parallel }_{TV}\le \frac{1}{2}{\chi }^2(p_0\parallel \pi {)}^{1/2}{e}^{-mt/2},$$

where ${\chi }^2\left(f||g\right)=\int \left(f\right(x\left)/g\right(x)-1{)}^{2}g(x)dx$ is the ${\chi }^2$ divergence between two densities $f$ and $g$.

This means that if $V(\cdot )$ is both smooth and strongly convex, then the distribution of $X_t$ converges to $\pi$ exponentially fast.

Unadjusted Langevin Algorithm

A natural and obvious method to simulate the SDE is using the discretized iteration:

$$\begin{array}{cc}{X}^{\left(0\right)}& \sim p_0\left(x\right),{\xi }^{\left(k\right)}\stackrel{iid}{\sim }N(0,I_d),\\ X^{(k+1)}& =X^{\left(k\right)}-\tau \nabla V\left(X^{\left(k\right)}\right)+\sqrt{2\tau }{\xi }^{(k+1)},k=0,1,\dots \end{array}$$

This is called the unadjusted Langevin algorithm (ULA).

Convergence of ULA ^[3]

Let $p_{k\tau }$ denote the distribution of $X^{\left(k\right)}$ with step size $\tau$. Suppose that $V(\cdot )$ is $m$-strongly-convex and $L$-smooth, and set $p_0=N(0,m^{-1}I)$. Then with

$$\tau =\frac{m\epsilon }{16d{L}^2},k=\frac{16L^2}{m^2}\cdot \frac{d\log \left(\frac{Ld}{m\epsilon }\right)}{\epsilon },$$

we have

$$KL(p_{k\tau }\parallel \pi )\le \epsilon ,\parallel p_{k\tau }-\pi {\parallel }_{TV}\le \sqrt{\epsilon },W_2(p_{k\tau },\pi )\le \sqrt{2\epsilon /m},$$

where $KL(\cdot \parallel \cdot )$ is the KL divergenve, and $W_2(\cdot ,\cdot )$ is the 2-Wasserstein distance.

Metropolis-Adjusted Langevin Algorithm

Due to the discretization, in general the distribution of $X^{\left(k\right)}$ generated by ULA would NOT converge to $\pi$.

However, ULA provides a good proposal distribution for the Metropolis-Hastings algorithm.

Random walk Metropolis:

• Propose $Y^{(k+1)}\sim N(X^{\left(k\right)},{\sigma }^{2}I)$

• Define $\alpha =min\{1,\pi (X^{(k+1)}\left)/\pi \right(X^{\left(k\right)}\left)\right\}$

• Set $X^{(k+1)}=Y^{(k+1)}$ with probability $\alpha$, otherwise $X^{(k+1)}=X^{\left(k\right)}$

Metropolis-Adjusted Langevin Algorithm

Metropolis-Adjusted Langevin Algorithm (MALA):

• Propose $Y^{(k+1)}\sim N(X^{\left(k\right)}-\tau \nabla V(X^{\left(k\right)}),2\tau I)$

• Define $$\alpha =min\{1,\frac{\exp [-V(Y^{(k+1)})-\parallel X^{\left(k\right)}-Y^{(k+1)}+\tau \nabla V(Y^{(k+1)}\left){\parallel }^2/\right(4\tau \left)\right]}{\exp [-V(X^{\left(k\right)})-\parallel Y^{(k+1)}-X^{\left(k\right)}+\tau \nabla V(X^{\left(k\right)}\left){\parallel }^2/\right(4\tau \left)\right]}\}$$

• Set $X^{(k+1)}=Y^{(k+1)}$ with probability $\alpha$, otherwise $X^{(k+1)}=X^{\left(k\right)}$

Convergence Rate of MALA ^[4]

For smooth and strongly convex $V(\cdot )$, [4] shows that MALA has advantages over ULA and random walk Metropolis.

Roughly speaking, to obtain samples with total variation error at most $\epsilon$, ULA requires $O\left({\kappa }^{2}d/{\epsilon }^2\right)$ steps from a warm start, whereas MALA only requires $O\left(\kappa d\log \right(1/\epsilon \left)\right)$. Here $\kappa =L/m$ is the condition number.

As a comparison, the random walk Metropolis requires $O\left({\kappa }^{2}d\log \right(1/\epsilon \left)\right)$ steps.

Underdamped Langevin Diffusion

Another variant of Langevin diffusion, called underdamped Langevin diffusion, solves the following SDE:

$$\begin{array}{cc}{X}_0& =x_0,Z_0=z_0,\\ d{X}_t& =Z_{t}dt,\\ d{Z}_t& =-\gamma Z_{t}dt-\nabla V\left(X_t\right)dt+\sqrt{2\gamma }d{B}_t,t\ge 0.\end{array}$$

Here we introduce an auxiliary process $\left\{Z_t\right\}$. Interestingly, $(X_t,Z_t)$ has a joint invariant distribution of the form

$$\pi (x,z)\propto \exp \{-V(x)-\parallel z{\parallel }^2/2\}\propto \pi \left(x\right)\cdot \exp (-\parallel z{\parallel }^2/2),$$

where $\pi \left(x\right)\propto \exp \{-V(x\left)\right\}$ is the target distribution.

Underdamped Langevin Diffusion

This means that if the distribution of $(X_t,Z_t)$ converges to the invariant distribution, then $X_t$ eventually follows $\pi \left(x\right)$, and $Z_t$ converges to a normal distribution $N(0,I_d)$ independent of $X$.

The discretized version is

$$\begin{array}{cc}{X}^{(k+1)}& =X^{\left(k\right)}+\tau Z^{\left(k\right)},\\ Z^{(k+1)}& =(1-\gamma \tau )Z^{\left(k\right)}-\tau \nabla V\left(X^{\left(k\right)}\right)+\sqrt{2\gamma \tau }{\xi }^{(k+1)},k=0,1,\dots \end{array}$$

However, for the underdamped Langevin diffusion we do not have an obvious Metropolis correction method.

Hamiltonian Dynamics

• Imagine a particle with position $x\in R^d$, velocity $v\in R^d$, and unit mass

• Define the total energy as $$H(x,v)=U\left(x\right)+\frac{1}{2}\parallel v{\parallel }^2$$

• Also called the Hamiltonian of the particle

Hamiltonian Dynamics

• The Hamiltonian dynamics are a set of differential equations: $$\frac{dx}{dt}=\frac{\partial H(x,v)}{\partial v},\frac{dv}{dt}=-\frac{\partial H(x,v)}{\partial x}$$

• We immediately find that $dH/dt=0$

• With the given form $H(x,v)=U\left(x\right)+\frac{1}{2}\parallel v{\parallel }^2$, we have $$\frac{dx}{dt}=v,\frac{dv}{dt}=-\nabla U\left(x\right)$$

Hamiltonian Dynamics

• Starting from an initial value $(x,v)$, denote the solution to the equations by $${\phi }_t(x,v)=\left(x_t\right(x,v),v_t(x,v\left)\right)$$

• Properties

  • ${\phi }_t$ is a deterministic operator, i.e., no random variables involved so far

  • ${\phi }_t\left(x_t\right(x,v),-v_t(x,v\left)\right)=(x,-v)$

  • $H\left({\phi }_t\right(x,v\left)\right)=H(u,v)$

Stationarity Theorem ^[5]

Suppose $(X,V)\sim \pi (x,v)\propto e^{-H(x,v)}$, then for any $T\ge 0$, ${\phi }_T(X,V)\sim \pi (x,v).$

• $(X,V)\sim \pi (x,v)$ implies that $X$ and $V$ are independent with $X\sim \pi \left(x\right)\propto e^{-U\left(x\right)}$ and $V\sim N(0,I_d)$

• The theorem indicates that the operator ${\phi }_T$ does not change the distribution of the random vectors $(X,V)$

• Marginally, ${\phi }_T$ does not change the distribution of $X$ for any $V$ that is independent of $X$

Idealized HMC

If we define a Markov chain with the following transition algorithm from $X^{\left(k\right)}$:

1. Simulate $V\sim N(0,I_d)$ independent of $X^{\left(k\right)}$

2. Let $(X^{(k+1)},V^{\prime })={\phi }_T(X^{\left(k\right)},V)$

Then this transition kernel $T(x^{\left(k\right)}\to x^{(k+1)})$ satisfies $$\pi \left(y\right)=\int \pi \left(x\right)T(x\to y)dx,$$ where $\pi \left(x\right)\propto e^{-U\left(x\right)}$.

Idealized HMC

For $k=1,...,K$:

1. Sample $\xi \sim N(0,I_d)$

2. Set $(X^{(k+1)},{\xi }^{\prime })={\phi }_T(X^{\left(k\right)},\xi )$

Then under mild conditions on $U\left(x\right)$ we obtain the ergodicity of $\left\{X^{\left(k\right)}\right\}$.

Note that ${\phi }_T$ is essentially determined by $\nabla U\left(x\right)$, and we assume the solution to the differential equations can be solved exactly.

Convergence Rate ^[6]

Assume that $U:R^d\to R$ is $m$-strongly-convex and $L$-smooth. Let ${\nu }_k$ be the marginal distribution of $X^{\left(k\right)}$ from the idealized HMC algorithm. Then for any $0<\epsilon <\sqrt{d}$ with $X^{\left(0\right)}=\arg {min}_{x}U\left(x\right)$, $T=\Omega \left(\frac{1}{\sqrt{L}}\right)$, and $k=O\left(\frac{L}{m}\log \right(d/\epsilon \left)\right)$, we have $$W_2({\nu }_k,\pi )\le \frac{\epsilon }{\sqrt{m}},$$ where $W_2(\cdot ,\cdot )$ is the 2-Wasserstein distance between two distributions.

Some Remarks

The previous theorem means that if $U(\cdot )$ is both smooth and strongly convex, then the distribution of $X^{\left(k\right)}$ converges to $\pi$ exponentially fast.

At first glance it is based on the same assumption as in the Langevin algorithm, and the result is also similar.

However, empirical results show that HMC tends to perform better for more complicated distributions, e.g. multimodal distributions.

Discretized HMC

• In reality, of course, we cannot solve the differential equations exactly

• Need to discretize the solution

• Let $q\left(0\right)=X^{\left(k\right)}$, $p\left(0\right)\sim N(0,I_d)$. For $l=0,\dots ,L-1$, $$\begin{array}{cc}p(l+1/2)& =p\left(l\right)-\frac{\tau }{2}\nabla U\left(q\right(l\left)\right)\\ q(l+1)& =q\left(l\right)+\tau p(l+1/2)\\ p(l+1)& =p(l+1/2)-\frac{\tau }{2}\nabla U\left(q\right(l+1\left)\right)\end{array}$$

• Propose $Y=q\left(L\right)$ as the next state

• $\tau$ and $L$ are tuning parameters

Discretized HMC

• But now we lose the preservation of Hamiltonian

• Need a Metropolis-Hastings correction!

• Acceptance probability $$\alpha =min(1,\frac{\exp [-H(Y,\tilde{V}\left)\right]}{\exp [-H(X^{\left(k\right)},V\left)\right]}),V=p\left(0\right),\tilde{V}=p\left(L\right)$$

• With probability $\alpha$ set $X^{(k+1)}=Y$; otherwise set $X^{(k+1)}=X^{\left(k\right)}$

References

References