DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models

Introduction

Diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have achieved impressive success on various tasks, such as high-resolution image synthesis (Dhariwal & Nichol, 2021; Ho et al., 2022; Rombach et al., 2022), image editing (Meng et al., 2022; Saharia et al., 2022a; Zhao et al., 2022), text-to-image generation (Nichol et al., 2021; Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Gu et al., 2022), voice synthesis (Liu et al., 2022a; Chen et al., 2021a; b), molecule generation (Xu et al., 2022; Hoogeboom et al., 2022; Wu et al., 2022) and data compression (Theis et al., 2022; Kingma et al., 2021). Compared with other deep generative models such as GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2014), DPMs can even achieve better sample quality by leveraging an essential technique called guided sampling (Dhariwal & Nichol, 2021; Ho & Salimans, 2021), which uses additional guidance models to improve the sample fidelity and the condition-sample alignment. Through it, DPMs in text-to-image and image-to-image tasks can generate high-resolution photorealistic and artistic images which are highly correlated to the given condition, bringing a new trend in artificial intelligence art painting.

The sampling procedure of DPMs gradually removes the noise from pure Gaussian random variables to obtain clear data, which can be viewed as discretizing either the diffusion SDEs (Ho et al., 2020; Song et al., 2021b) or the diffusion ODEs (Song et al., 2021b; a) defined by a parameterized noise prediction model or data prediction model (Ho et al., 2020; Kingma et al., 2021). Guided sampling of DPMs can also be formalized with such discretizations by combining an unconditional model with a guidance model, where a hyperparameter controls the scale of the guidance model (i.e. guidance scale). The commonly-used method for guided sampling is DDIM (Song et al., 2021a), which is proven as a first-order diffusion ODE solver (Salimans & Ho, 2022; Lu et al., 2022) and it generally needs 100 to 250 steps of large neural network evaluations to converge, which is time-consuming.

Dedicated high-order diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) can generate high-quality samples in 10 to 20 steps for sampling without guidance. However, their effectiveness for guided sampling has not been carefully examined before. In this work, we demonstrate that previous high-order solvers for DPMs generate unsatisfactory samples for guided sampling, even worse than the simple first-order solver DDIM. We identify two challenges of applying high-order solvers to guided sampling: (1) the large guidance scale narrows the convergence radius of high-order solvers, making them unstable; and (2) the converged solution does not fall into the same range with the original data (a.k.a. “train-test mismatch” (Saharia et al., 2022b)).

Based on the observations, we propose DPM-Solver++, a training-free fast diffusion ODE solver for guided sampling. We find that the parameterization of the DPM critically impacts the solution quality. Subsequently, we solve the diffusion ODE defined by the data prediction model, which predicts the clean data given the noisy ones. We derive a high-order solver for solving the ODE with the data prediction parameterization, and adopt dynamic thresholding methods (Saharia et al., 2022b) to mitigate the train-test mismatch problem. Furthermore, we develop a multistep solver which uses smaller step sizes to address instability.

As shown in Fig. 1, DPM-Solver++ can generate high-quality samples in only 15 steps, which is much faster than all the previous training-free samplers for guided sampling. Our additional experimental results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, for a wide variety of guided sampling applications, including both pixel-space DPMs and latent-space DPMs.

Diffusion Probabilistic Models

In this section, we review diffusion probabilistic models (DPMs) and their sampling methods.

where $\alpha_{t},\sigma_{t}>0$ and the signal-to-noise-ratio (SNR) $\alpha_{t}^{2}/\sigma_{t}^{2}$ is strictly decreasing w.r.t. $t$ (Kingma et al., 2021). Eq. (1) can be written as $\bm{x}_{t}=\alpha_{t}\bm{x}_{0}+\sigma_{t}\bm{\epsilon}$, where $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$.

DPMs learn to recover the data $\bm{x}_{0}$ based on the noisy input $\bm{x}_{T}$ with a sequential denoising procedure. There are two alternative ways to define the model. The noise prediction model $\bm{\epsilon}_{\theta}(\bm{x}_{t},t)$ attempts to predict the noise $\epsilon$ from the data $\bm{x}_{t}$, which optimizes the parameter $\theta$ by the following objective (Ho et al., 2020; Song et al., 2021b):

where $\bm{x}_{0}\sim q_{0}(\bm{x}_{0})$, $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$, $t\sim\mathcal{U}()$, and $\omega(t)>0$ is a weighting function. Alternatively, the data prediction model $\bm{x}_{\theta}(\bm{x}_{t},t)$ predicts the original data $\bm{x}_{0}$ based on the noisy $\bm{x}_{t}$, and its relationship with $\bm{\epsilon}_{\theta}(\bm{x}_{t},t)$ is given by $\bm{x}_{\theta}(\bm{x}_{t},t)\coloneqq(\bm{x}_{t}-\sigma_{t}\bm{\epsilon}_{\theta}(\bm{x}_{t},t))/\alpha_{t}$ (Kingma et al., 2021).

Sampling by DPMs can be implemented by solving the diffusion ODEs (Song et al., 2021b; a; Liu et al., 2022b; Zhang & Chen, 2022; Lu et al., 2022), which is generally faster than other sampling methods. Specifically, sampling by diffusion ODEs need to discretize the following ODE (Song et al., 2021b) with $t$ changing from $T$ to :

and the equivalent diffusion ODE w.r.t. the data prediction model $\bm{x}_{\theta}$ is

2 Guided Sampling for DPMs

where $s>0$ is the guidance scale. In practice, a large $s$ is usually preferred for improving the condition-sample alignment (Rombach et al., 2022; Saharia et al., 2022b) for guided sampling. Classifier-free guidance (Ho & Salimans, 2021) shares the same parameterized model $\bm{\epsilon}_{\theta}(\bm{x}_{t},t,c)$ for the unconditional and conditional noise prediction models, where the input $c$ for the unconditional model is a special placeholder $\varnothing$. The corresponding conditional model is defined by:

Then, samples can be drawn by solving the ODE (3) with $\bm{\epsilon}_{\theta}(\bm{x}_{t},t,c)$ in place of $\bm{\epsilon}_{\theta}(\bm{x}_{t},t)$. DDIM (Song et al., 2021a) is a typical solver for guided sampling, which generates samples in a few hundreds of steps.

3 Exponential Integrators and High-Order ODE Solvers

It is shown in recent works (Lu et al., 2022; Zhang & Chen, 2022) that ODE solvers based on exponential integrators (Hochbruck & Ostermann, 2010) converge much faster than the traditional solvers for solving the unconditional diffusion ODE (3). Given an initial value $\bm{x}_{s}$ at time $s>0$, Lu et al. (2022) derive the solution $\bm{x}_{t}$ of the diffusion ODE (3) at time $t$ as:

where the ODE is changed from the time ($t$) domain to the log-SNR ($\lambda$) domain by the change-of-variables formula. Here, the log-SNR $\lambda_{t}\coloneqq\log(\alpha_{t}/\sigma_{t})$ is a strictly decreasing function of $t$ with the inverse function $t_{\lambda}(\cdot)$, and $\hat{\bm{x}}_{\lambda}\coloneqq\bm{x}_{t_{\lambda}(\lambda)}$, $\hat{\bm{\epsilon}}_{\theta}(\hat{\bm{x}}_{\lambda},\lambda)\coloneqq\bm{\epsilon}_{\theta}(\bm{x}_{t_{\lambda}(\lambda)},t_{\lambda}(\lambda))$ are the corresponding change-of-variable forms for $\lambda$. Lu et al. (2022) showed that DDIM is a first-order solver for Eq. (7). They further proposed a high-order solver named “DPM-Solver”, which can generate realistic samples for the unconditional model in only 10-20 steps.

Unfortunately, the outstanding efficiency of existing high-order solvers does not transfer to guided sampling, which we shall discuss soon.

Challenges of High-Order Solvers for Guided Sampling

Before developing new fast solvers, we first examine the performance of existing high-order diffusion ODE solvers and highlight the challenges.

The first challenge is the large guidance scale causes high-order solvers to be instable. As shown in Fig. 1, for a large guidance scale $s=8.0$ and $15$ function evaluations, previous high-order diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022; Liu et al., 2022b) produce low-quality images. Their sample quality is even worse than the first-order DDIM. Moreover, the sample quality becomes even worse as the order of the solver gets higher.

Designing Training-Free Fast Samplers for Guided Sampling

For solving the diffusion ODE w.r.t. $\bm{x}_{\theta}$ in Eq. (4), we firstly propose a simplified formulation of the exact solution of diffusion ODEs w.r.t. $\bm{x}_{\theta}$ below. Such formulation exactly computes the linear term in Eq. (4) and only remains an exponentially-weighted integral of $\bm{x}_{\theta}$. Denote $\hat{\bm{x}}_{\theta}(\hat{\bm{x}}_{\lambda},\lambda)\coloneqq\bm{x}_{\theta}(\bm{x}_{t_{\lambda}(\lambda)},t_{\lambda}(\lambda))$ as the change-of-variable form of $\bm{x}_{\theta}$ for $\lambda$, we have:

Given an initial value $\bm{x}_{s}$ at time $s>0$, the solution $\bm{x}_{t}$ at time $t\in[0,s]$ of diffusion ODEs in Eq. (4) is:

We name the algorithm as DPM-Solver++(2S), which means that the proposed solver is a second-order singlestep method. We present the theoretical guarantee of the convergence order in Appendix A. For $k\geq 3$, as discussed in Sec. 3, high-order solvers may be unsuitable for large guidance scales, thus we mainly consider $k=2$ in this work, and leave the solvers for higher orders for future study.

Moreover, we provide a theoretical comparison between DPM-Solver-2 (Lu et al., 2022) and DPM-Solver++(2S) in Appendix B. We find that DPM-Solver++(2S) has a smaller constant before the high-order error terms, thus generally has a smaller discretization error than DPM-Solver-2.

2 From Singlestep to Multistep

At each step (from $t_{i-1}$ to $t_{i}$), the proposed singlestep solver needs two sequential function evaluations of the neural network $\bm{x}_{\theta}$. Moreover, the intermediate values $\bm{u}_{i}$ are only used once and then discarded. Such method loses the previous information and may be inefficient. In this section, we propose another second-order diffusion ODE solver which uses the previous information at each step.

For a fixed budget $N$ of the total number of function evaluations, multistep methods can use $M=N$ steps, while $k$-th order singlestep methods can only use no more than $M=N/k$ steps. Therefore, each step size $h_{i}$ of multistep methods is around $1/k$ of that of singlestep methods, so the high-order error terms $\mathcal{O}(h_{i}^{k})$ in Eq. (9) of multistep methods may also be smaller than those of singlestep methods. We show in Sec. 7.1 that the multistep methods are slightly better than singlestep methods.

3 Combining Thresholding with DPM-Solver++

For distributions of bounded data (such as the image data), thresholding methods (Ho et al., 2020; Saharia et al., 2022b) can push out-of-bound samples inwards and somehow reduce the adverse impact of the large guidance scale. Specifically, thresholding methods define a clipped data prediction model $\hat{\bm{x}}_{\theta}(\bm{x}_{t},t,c)$ by elementwise clipping the original model $\bm{x}_{\theta}\coloneqq(\bm{x}_{t}-\sigma_{t}\bm{\epsilon}_{\theta})/\alpha_{t}$ within the data bound, which results in better sample quality for large guidance scales (Saharia et al., 2022b). As our proposed DPM-Solver++ is designed for the $\bm{x}_{\theta}$ model, we can straightforwardly combine thresholding methods with DPM-Solver++.

Fast Solvers for Diffusion SDEs

Sampling by diffusion models can be alternatively implemented by solving Diffusion SDEs (Song et al., 2021b):

where $\bar{\bm{w}}_{t}$ is the reverse-time Wiener process from $T$ to . In this section, we consider the diffusion SDEs w.r.t. log-SNR $\lambda$, and derive the corresponding second-order solvers.

By applying variation-of-constants formula, we propose the exact solution of diffusion SDEs as following.

Given an initial value $\bm{x}_{s}$ at time $s>0$, the solution $\bm{x}_{t}$ at time $t\in[0,s]$ of diffusion SDEs in Eq. (10) is:

Moreover, we can compute the Itô-integral by

where $\bm{z}_{s}\sim\mathcal{N}(\bm{0},\bm{I})$. Thus, we can discretize the integral w.r.t. $\bm{\epsilon}_{\theta}$ or $\bm{x}_{\theta}$ to get the corresponding solvers for diffusion SDEs, which is presented below. For simplicity, we denote that $h\coloneqq\lambda_{t}-\lambda_{s}$.

Let $\bm{z}_{s}\sim\mathcal{N}(\bm{0},\bm{I})$. By assuming $\bm{\epsilon}_{\theta}(\bm{x}_{\lambda},\lambda)\approx\bm{\epsilon}_{\theta}(\bm{x}_{s},s)$, we have

Let $\bm{z}_{s}\sim\mathcal{N}(\bm{0},\bm{I})$. By assuming $\bm{x}_{\theta}(\bm{x}_{\lambda},\lambda)\approx\bm{x}_{\theta}(\bm{x}_{s},s)$, we have

Let $\bm{z}_{s}\sim\mathcal{N}(\bm{0},\bm{I})$. Assume we have a previous solution $\bm{x}_{r}$ with its model output $\bm{\epsilon}_{\theta}(\bm{x}_{r},r)$ at time $r<t$. Denote $r_{1}=\frac{\lambda_{r}-\lambda_{s}}{h}$. By assuming $\bm{\epsilon}_{\theta}(\bm{x}_{\lambda},\lambda)\approx\bm{\epsilon}_{\theta}(\bm{x}_{s},s)+\frac{\lambda-\lambda_{s}}{r_{1}h}(\bm{\epsilon}_{\theta}(\bm{x}_{r},r)-\bm{\epsilon}_{\theta}(\bm{x}_{s},s))$, we have

Let $\bm{z}_{s}\sim\mathcal{N}(\bm{0},\bm{I})$. Assume we have a previous solution $\bm{x}_{r}$ with its model output $\bm{x}_{\theta}(\bm{x}_{r},r)$ at time $r<t$. Denote $r_{1}=\frac{\lambda_{r}-\lambda_{s}}{h}$. By assuming $\bm{x}_{\theta}(\bm{x}_{\lambda},\lambda)\approx\bm{x}_{\theta}(\bm{x}_{s},s)+\frac{\lambda-\lambda_{s}}{r_{1}h}(\bm{x}_{\theta}(\bm{x}_{r},r)-\bm{x}_{\theta}(\bm{x}_{s},s))$, we have

Relationship with Other Fast Sampling Methods

In essence, all training-free sampling methods for DPMs can be understood as either discretizing diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) or discretizing diffusion ODEs (Song et al., 2021b; a; Liu et al., 2022b; Zhang & Chen, 2022; Lu et al., 2022). As DPM-Solver++ is designed for solving diffusion ODEs, in this section, we discuss the relationship between DPM-Solver++ and other diffusion ODE solvers. We further briefly discuss other fast sampling methods for DPMs.

The general version of DDIM with $\eta\geq 0$ is (Song et al., 2021a)

Previous state-of-the-art fast diffusion ODE solvers (Lu et al., 2022; Zhang & Chen, 2022) leverages exponential integrators to solve diffusion ODEs with noise prediction models $\bm{\epsilon}_{\theta}$. In short, these solvers approximate the exact solution in Eq. (7) and include DDIM (Song et al., 2021a) with $\eta=0$ as the first-order case. Below we show that the first-order case for DPM-Solver++ is also DDIM.

For $k=1$, Eq. (9) becomes (after omitting the $\mathcal{O}(h_{i}^{k+1})$ terms)

Therefore, our proposed DPM-Solver++ is the high-order generalization of DDIM ($\eta=0$) w.r.t. the data prediction model $\bm{x}_{\theta}$. To the best of our knowledge, such generalization has not been proposed before. We list the detailed difference between previous high-order solvers based on exponential integrators and DPM-Solver++ in Table 1. We emphasize that although the first-order version of these solvers are equivalent, the high-order versions of these solvers are rather different.

In addition, for DDIM with $\eta=\sigma_{t}\sqrt{1-e^{-2h}}$, it is easy to verify that such a stochastic DDIM is equivalent to SDE-DPM-Solver++1. Therefore, our proposed SDE-DPM-Solver++(2M) is a second-order generalized version of the first-order stochastic DDIM. To the best of our knowledge, such a finding is not revealed in previous works.

2 Other Fast Sampling Methods

Samplers based on diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) generally needs more steps to converge than those based on diffusion ODEs (Lu et al., 2022), because SDEs introduce more randomness and make denoising more difficult. Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022). However, training-based samplers are hard to scale-up to pre-trained large DPMs (Saharia et al., 2022b; Rombach et al., 2022; Ramesh et al., 2022). There are other fast sampling methods by modifying the original DPMs to a latent space (Vahdat et al., 2021) or with momentum (Dockhorn et al., 2022). In addition, combining DPMs with GANs (Xiao et al., 2022; Wang et al., 2022) improves the sample quality of GANs and sampling speed of DPMs.

Experiments

In this section, we show that DPM-Solver++ can speed up both the pixel-space DPMs and the latent-space DPMs for guided sampling. We vary different number of function evaluations (NFE) which is the numebr of calls to the model $\bm{\epsilon}_{\theta}(\bm{x}_{t},t,c)$ or $\bm{x}_{\theta}(\bm{x}_{t},t,c)$, and compare DPM-Solver++ with the previous state-of-the-art fast samplers for DPMs including DPM-Solver (Lu et al., 2022), DEIS (Zhang & Chen, 2022), PNDM (Liu et al., 2022b) and DDIM (Song et al., 2021a). We also convert the discrete-time DPMs to the continuous-time and use these continuous-time solvers. We refer to Appendix C for the detailed implementations and experiment settings.

As previous solvers did not test the performance in guided sampling, we also carefully tune the baseline samplers by ablating the step size schedule (i.e. the choice for the time steps $\{t_{i}\}_{i=0}^{M}$) and the solver order. We find that

For the step size schedule, we search the time steps in the following choices: uniform $t$ (a widely-used setting in high-resolution image synthesis), uniform $\lambda$ (used in (Lu et al., 2022)), uniform split of the power functions of $t$ (used in (Zhang & Chen, 2022), detailed in Appendix C), and we find that the best choice is uniform $t$. Thus, we use uniform $t$ for the time steps in all of our experiments for all of the solvers.

We find that for a large guidance scale, the best choice for all the previous solvers is the second-order (i.e. DPM-Solver-2 and DEIS-1). However, for a comprehensive comparison, we run all the orders of previous solvers, including DPM-Solver-2 and DPM-Solver-3; DEIS-1, DEIS-2 and DEIS-3 and choose their best result for each NFE in our comparison.

We run both DPM-Solver++(2S) and DPM-Solver++(2M), and we find that for large guidance scales, the multistep DPM-Solver++(2M) performs better; and for a slightly small guidance scales, the singlestep DPM-Solver++(2S) performs better. We report the best results of DPM-Solver++ and all of the previous samplers in the following sections, the detailed values are listed in Appendix D.

We firstly compare DPM-Solver++ with other samplers for the guided sampling with classifier-guidance on ImageNet 256x256 dataset by the pretrained DPMs(Dhariwal & Nichol, 2021). We measure the sample quality by drawing 10K samples and computing the widely-used FID score (Heusel et al., 2017), where lower FID usually implies better sample quality. We also adopt the dynamic thresholding method (Saharia et al., 2022b) for both DDIM and DPM-Solver++. We vary the guidance scale $s$ in 8.0, 4.0 and 2.0, the results are shown in Fig. 4(a-c). We find that for large guidance scales, all the previous high-order samplers (DEIS, PNDM, DPM-Solver) converge slower than the first-order DDIM, which shows that previous high-order samplers are unstable. Instead, DPM-Solver++ achieve the best speedup performance for both large guidance scales and small guidance scales. Especially for large guidance scales, DPM-Solver++ can almost converge within only $15$ NFE.

As an ablation, we also compare the singlestep DPM-Solver-2, the singlestep DPM-Solver++(2S) and the multistep DPM-Solver++(2M) to demonstrate the effectiveness of our method. We use a large guidance scale $s=8.0$ and conduct the following ablations:

From $\bm{\epsilon}_{\theta}$ to $\bm{x}_{\theta}$: As shown in Fig. 3(a), by simply changing the solver from $\bm{\epsilon}_{\theta}$ to $\bm{x}_{\theta}$ (i.e. from DPM-Solver-2 to DPM-Solver++(2S)), the solver can achieve a stable acceleration performance which is faster than the first-order DDIM. Such result indicates that for guided sampling, high-order solvers w.r.t. $\bm{x}_{\theta}$ may be more preferred than those w.r.t. $\bm{\epsilon}_{\theta}$.

From singlestep to multistep: As show in Fig. 3(b), the multistep DPM-Solver++(2M) converges slightly faster than the singlestep DPM-Solver++(2S), which almost converges in 15 NFE. Such result indicates that for guided sampling with a large guidance scale, multistep methods may be faster than singlestep methods.

With or without thresholding: We compare the performance of DDIM and DPM-Solver++ with / without thresholding methods in Fig. 3(c). Note that the thresholding method changes the model $\bm{x}_{\theta}$ and thus also changes the converged solutions of diffusion ODEs. Firstly, we find that after using the thresholding method, the diffusion ODE can generate higher quality samples, which is consistent with the conclusion in (Saharia et al., 2022b). Secondly, the sample quality of DPM-Solver++ with thresholding outperforms DPM-Solver++ without thresholding under the same NFE. Moreover, when combined with thresholding, DPM-Solver++ is faster than the first-order DDIM, which shows that DPM-Solver++ can also speed up guided sampling by DPMs with thresholding methods.

2 Latent-Space DPMs with Guidance

We also evaluate DPM-Solver++ on the latent-space DPMs (Rombach et al., 2022), which is recently popular among the community due to their official code “stable-diffusion”. We use the default guidance scale $s=7.5$ in their official code. The latent-space DPMs map the image data with a latent code by training a pair of encoder and decoder, and then train a DPM for the latent code. As the latent code is unbounded, we do not apply the thresholding method.

Specifically, we randomly sample 10K caption-image pairs from the MS-COCO2014 validation dataset and use the captions as conditions to draw 10K images from the pretrained “stable-diffusion” model, and we only draw a single image sample of each caption, following the standard evaluation procedures in (Nichol et al., 2021; Rombach et al., 2022). We find that all the solvers can achieve a FID around 15.0 to 16.0 even within only 10 steps, which is very close to the FID computed by the converged samples reported in the official page of “stable-diffusion”. We believe it is due to the powerful pretrained decoder, which can map a non-converged latent code to a good image sample.

For latent-space DPMs, different diffusion ODE solvers directly affect the convergence speed on the latent space. To further compare different samplers for latent-space DPMs, we directly compare different solvers according to the convergence error on the latent space by the L2-norm between the sampled $\bm{x}_{0}$ and the true solution $\bm{x}_{0}^{*}$ (and the error between them is $\|\bm{x}_{0}-\bm{x}_{0}^{*}\|_{2}/\sqrt{D}$). Specifically, we firstly sample 10K noise variables from the standard normal distribution and fix them. Then we sample 10K latent codes by different DPM samplers, starting from the 10K fixed noise variables. As all these solvers can be understood as discretizing diffusion ODEs, we compare the sampled latent codes by the true solution $\bm{x}_{0}^{*}$ from a 999-step DDIM with samples $\bm{x}_{0}$ by different samplers within different NFE, and the results are shown in Fig. 4(d). We find that the supported fast samplers (DDIM and PNDM) in “stable-diffusion” converge much slower than DPM-Solver++ and DEIS, and we find that the second-order multistep DPM-Solver++ and DEIS achieve a quite close speedup on the latent space. Moreover, as “stable-diffusion” by default use PNDM with 50 steps, we find that DPM-Solver++ can achieve a similar convergence error with only 15 to 20 steps. We also present an empirical comparison of the sampled images between different solvers in Appendix D, and we find that DPM-Solver++ can indeed generate quite good image samples within only 15 to 20 steps.

Conclusions

We study the problem of accelerating guided sampling of DPMs. We demonstrate that previous high-order solvers based on the noise prediction models are abnormally unstable and generate worse-quality samples than the first-order solver DDIM for guided sampling with large guidance scales. To address this issue and speed up guided sampling, we propose DPM-Solver++, a training-free fast diffusion ODE solver for guided sampling. DPM-Solver++ is based on the diffusion ODE with the data prediction models, which can directly adopt the thresholding methods to stabilize the sampling procedure further. We propose both singlestep and multistep variants of DPM-Solver++. Experiment results show that DPM-Solver++ can generate high-fidelity samples and almost converge within only 15 to 20 steps, applicable for pixel-space and latent-space DPMs.

Ethics Statement

Like other deep generative models such as GANs, DPMs may also be used to generate adverse fake contents (images). The proposed solver can accelerate the guided sampling by DPMs which can further be used for image editing and generate photorealistic fake images. Such influence may further amplify the potential undesirable affects of DPMs for malicious applications.

Reproducibility Statement

Our code is based on the official code of DPM-Solver (Lu et al., 2022) and the pretrained checkpoints in Dhariwal & Nichol (2021) and Stable-Diffusion (Rombach et al., 2022). We will release it after the blind review. In addition, datasets used in experiments are publicly available. Our detailed experiment settings and implementations are listed in Appendix C, and the proof of the solver convergence guarantee are presented in Appendix A.

References

Appendix A Additional Proofs

Taking derivative w.r.t. $t$ in Eq. (8) yields

A.2 Proof of Proposition 5.1

For diffusion SDEs w.r.t. the noise prediction model $\bm{\epsilon}_{\theta}$, we have

And for diffusion SDEs w.r.t. the data prediction model $\bm{x}_{\theta}$, we have

A.3 Derivation of SDE Solvers

We can also applying the same approximation as in Lu et al. (2022) by

We can also applying the same approximation as in Lu et al. (2022) by

A.4 Convergence of Algorithms

We make the following assumptions as in Lu et al. (2022) for $\bm{x}_{\theta}$, i.e.,

$\bm{x}_{\theta}^{(0)}$, $\bm{x}_{\theta}^{(1)}$ and $\bm{x}_{\theta}^{(2)}$ exist and are continuous (and hence are bounded).

The map $\bm{x}\mapsto\bm{x}_{\theta}(\bm{x},t)$ is $L$-Lipschitz.

$h_{max}:=\max_{1\leq j\leq M}h_{j}=O(1/M)$.

The convergence proof of this algorithm is similar to that in DPM-Solver-2 (Lu et al., 2022). We give it in this section for completeness.

Since $r_{i}$ is bounded away from zero, and $e^{-h_{i}}=1-h_{i}+h_{i}^{2}/2+O(h_{i}^{3})$, we know

where $M_{i}=1+\sup_{t_{i-1}\leq t\leq t_{i}}\|\bm{x}_{\theta}^{(1)}(\bm{x}_{t},t)\|$. Then, $\Delta_{i}$ could be estimated as follows.

Thus, $\Delta_{i}=O(h^{2}_{max})$ as long as $h_{max}$ is sufficiently small.

A.5 Convergence of Algorithm 2

where $C$ is a constant depends on $\bm{x}^{(2)}_{\theta}$. Also note that

Since $r_{i}$ is bounded away from zero, and $e^{-h_{i}}=1-h_{i}+h_{i}^{2}/2+O(h_{i}^{3})$, we know

where $M_{i}=1+\sup_{t_{i-1}\leq t\leq t_{i}}\|\bm{x}_{\theta}^{(1)}(\bm{x}_{t},t)\|$. Then, $\Delta_{i}$ could be estimated as follows.

Thus, $\Delta_{i}=O(h^{2}_{max})$ as long as $h_{max}$ is sufficiently small and $\Delta_{0}+\Delta_{1}=O(h^{2}_{max})$, which can be verified by the Taylor’s expansion.

Appendix B Comparison between DPM-Solver and DPM-Solver++

In this section, we convert DPM-Solver++(2S) to the formulation w.r.t. the noise prediction model and compare it with the second-order DPM-Solver (Lu et al., 2022).

At each step, the second-order DPM-Solver (DPM-Solver-2 (Lu et al., 2022)) has the following updating:

while DPM-Solver++(2S) has the following updating:

we can rewrite DPM-Solver++(2S) w.r.t. the noise prediction model (see Appendix B.1 for details):

Comparing with Eq. 44, we can find that the only difference between DPM-Solver-2 and DPM-Solver++(2S) is that DPM-Solver++(2S) has an additional coefficient $e^{-r_{i}h_{i}}<1$ at the second term (which is corresponding to approximating the first-order total derivative $\bm{\epsilon}^{(1)}_{\theta}$). Specifically, we have

As DPM-Solver++(2S) multiplies a smaller coefficient into the $\mathcal{O}(h_{i})$ error term, the constant before the high-order error term of DPM-Solver++(2S) is smaller than that of DPM-Solver-2. As they both are equivalent to a second-order discretization of the diffusion ODE, a smaller constant before the error term can result in a smaller discretization error and reducing the numerical instabilities (especially for large guidance scales). Therefore, using the data prediction model is a key for stabilizing the sampling, and DPM-Solver++(2S) is more stable than DPM-Solver-2.

Appendix C Implementation Details

Moreover, for the widely-used DDPM (Ho et al., 2020), we usually choose a sequence $\{\beta_{n}\}_{n=1}^{N}$ which is defined by either linear schedule (Ho et al., 2020) or cosine schedule (Nichol & Dhariwal, 2021). After obtained the $\beta_{n}$ sequence, the noise schedule $\alpha_{n}$ is defined by

where each $\alpha_{n}$ is corresponding to the continuous-time $t_{n}=\frac{nT}{N}$, i.e. $\alpha_{t_{n}}=\alpha_{n}$. To generalize the discrete $\alpha_{n}$ to the continuous version, we use a linear interpolation for the function $\log\alpha_{n}$. Specifically, for each $t\in[t_{n},t_{n+1}]$, we define

Therefore, we can obtain a continuous-time noise schedule $\alpha_{t}$ defined for all $t\in[\frac{T}{N},T]$, and the std $\sigma_{t}=\sqrt{1-\alpha_{t}^{2}}$ and the logSNR $\lambda_{t}=\log\alpha_{t}-\log\sigma_{t}$. Moreover, the logSNR $\lambda_{t}$ is strictly decreasing for $t$, thus the change-of-variable for $\lambda$ is still valid.

In practice, we usually have $T=1$ and $N=1000$, thus the smallest time is $10^{-3}$. Therefore, we solve the diffusion ODEs from time $t=1$ to time $t=10^{-3}$ to get our final sample. Such sampling can reduce the first-order discrete-time DDIM solver when using a uniform time step.

C.2 Ablating Time Steps

Previous DEIS only tuned on low-resolutional data CIFAR-10, which may be not suitable for high-resolutional data such as ImageNet 256x256 and large guidance scales for guided sampling. For a fair comparison with the baseline samplers, we firstly do ablation study for the time steps with the pretrained DPMs (Dhariwal & Nichol, 2021) on ImageNet 256x256 and vary the classifier guidance scale. In our experiments, we tune the time step schedule according to their power function choices. Specifically, let $t_{M}=10^{-3}$ and $t_{0}=1$, the time steps $\{t_{i}\}_{i=0}^{M}$ satisfies

where $\kappa$ is a hyperparameter. Following Zhang & Chen (2022), we search $\kappa$ in $1,2,3$ by DEIS, and the results are shown in Table 2. We find that for all guidance scales, the best setting is $\kappa=1$, i.e. the uniform $t$ for time steps. We further compare uniform $t$ and uniform $\lambda$ and find that the uniform $t$ time step schedule is still the best choice. Therefore, in all of our experiments, we use the uniform $t$ for evaluations.

C.3 Experiment Settings

We use uniform time step schedule for all experiments. Particularly, as DPM-Solver (Lu et al., 2022) is designed for uniform $\lambda$ (the intermediate time steps are a half of the step size w.r.t. $\lambda$), we also convert the intermediate time steps to ensure all the time steps are uniform $t$. We find that such conversion can improve the sample quality of both the singlestep DPM-Solver the singlestep DPM-Solver++.

We run NFE in 10, 15, 20, 25 for the high-order solvers and additional 50, 100, 250 for DDIM. For all experiments, we solver diffusion ODEs from $t=1$ to $t=10^{-3}$ with the interpolation of noise schedule detailed in Appendix C.1. For DEIS, we use the “t-AB-$k$” methods for $k=1,2,3$, which is the fastest method in their original paper, and we name them as DEIS-$k$, respectively.

For the sampled image in Fig. 6, we use the prompt “A beautiful castle beside a waterfall in the woods, by Josef Thoma, matte painting, trending on artstation HQ”.

Appendix D Experiment Details

We list all the detailed experimental results in this section.