Diffusion Models in Vision: A Survey (accepted at IEEE TPAMI 2023)

Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling. A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. In the forward diffusion stage, the input data is gradually perturbed over several steps by adding Gaussian noise. In the reverse stage, a model is tasked at recovering the original input data by learning to gradually reverse the diffusion process, step by step. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens, i.e. low speeds due to the high number of steps involved during sampling. This repository categorizes the papers about diffusion models, applied in computer vision, according to their target task. The classifcation is based on our survey Diffusion Models in Vision: A Survey, which was accepted for publication in IEEE TPAMI.

Summary

1. Unconditional Generation
2. Conditional Generation
3. Text-to-Image generation
4. Super-Resolution
5. Image Editing
6. Region Image Editing
7. Inpainting
8. Image-to-Image Translation
9. Image Segmentation
10. Multi-Task
11. Medical Image-to-Image Translation
12. Medical Image Generation
13. Medical Image Segmentation
14. Medical Image Anomaly Detection
15. Video Generation
16. Few-Shot Image Generation
17. Counterfactual Explanations and Estimations
18. Image Restoration
19. Image Registration
20. Adversarial Purification
21. Semantic Image Generation
22. Shape Generation and Completion
23. Classification
24. Point Cloud Generation
25. Theoretical
26. Graphs
27. Deblurring
28. Face Morphing Attack Detection
29. Trajectory prediction
30. Attacks
31. Study on data memorization

Content

Unconditional Generation

1. Deep unsupervised learning using non-equilibrium thermodynamics
2. Denoising diffusion probabilistic models
3. Improved techniques for training score-based generative models
4. Adversarial score matching and improved sampling for image generation
5. Maximum likelihood training of score-based diffusion models
6. D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation
7. Diffusion Normalizing Flow
8. Diffusion Schrodinger bridge with applications to score-based generative modeling
9. Structured denoising diffusion models in discrete state-spaces
10. Score-based generative modeling in latent space
11. Improved denoising diffusion probabilistic models
12. Denoising Diffusion Implicit Models
13. Non-Gaussian denoising diffusion models
14. Bilateral denoising diffusion models
15. Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
16. Noise estimation for generative diffusion models
17. Gotta go fast when generating data with score-based models
18. Learning to efficiently sample from diffusion probabilistic models
19. Deep generative learning via Schrodinger bridge
20. VAEs meet Diffusion Models: Efficient and High-Fidelity Generation
21. Variational diffusion models
22. Score-based generative modeling with critically-damped Langevin diffusion
23. Tackling the generative learning trilemma with Denoising Diffusion GANs
24. Heavy-tailed denoising score matching
25. Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models
26. Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
27. Truncated Diffusion Probabilistic Models
28. Subspace Diffusion Generative Models
29. Maximum Likelihood Training of Implicit Nonlinear Diffusion Models
30. On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
31. Diffusion-GAN: Training GANs with Diffusion
32. Accelerating Score-based Generative Models for High-Resolution Image Synthesis
33. Soft Diffusion: Score Matching for General Corruptions
34. Post-Training Quantization on Diffusion Models
35. Lookahead Diffusion Probabilistic Models for Refining Mean Estimation
36. Wavelet Diffusion Models are fast and scalable Image Generators
37. All are Worth Words: A ViT Backbone for Diffusion Models
38. Diffusion Probabilistic Model Made Slim

Conditional Generation

1. Diffusion models beat gans on image synthesis
2. Classifier-Free Diffusion Guidance
3. On Fast Sampling of Diffusion Probabilistic Models
4. DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
5. Pseudo Numerical Methods for Diffusion Models on Manifolds
6. Cascaded Diffusion Models for High Fidelity Image Generation
7. High Fidelity Visualization of What Your Self-Supervised Representation Knows About
8. Itô-Taylor Sampling Scheme for Denoising Diffusion Probabilistic Models using Ideal Derivatives
9. {Dynamic Dual-Output Diffusion Models
10. Generating High Fidelity Data from Low-density Regions using Diffusion Models
11. Perception Prioritized Training of Diffusion Models
12. Elucidating the Design Space of Diffusion-Based Generative Models
13. Progressive distillation for fast sampling of diffusion models
14. Denoising Likelihood Score Matching for Conditional Score-based Data Generation
15. On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models
16. A Continuous Time Framework for Discrete Denoising Models
17. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
18. Compositional Visual Generation with Composable Diffusion Models
19. TryOnDiffusion: A Tale of Two UNets
20. High-Fidelity Guided Image Synthesis with Latent Diffusion Models
21. Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models
22. Towards Practical Plug-and-Play Diffusion Models
23. Inversion-based Style Transfer with Diffusion Models
24. Conditional Text Image Generation with Diffusion Models
25. Generative Diffusion Prior for Unified Image Restoration and Enhancement
26. DCFace: Synthetic Face Generation With Dual Condition Diffusion Model
27. Controllable Light Diffusion for Portraits
28. LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation
29. Self-Guided Diffusion Models

Text-to-Image generation

1. Vector quantized diffusion model for text-to-image synthesis
2. Hierarchical text-conditional image generation with CLIP latents
3. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
4. Fast Sampling of Diffusion Models with Exponential Integrator
5. DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder
6. Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models
7. Text2Human: Text-Driven Controllable Human Image Generation
8. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
9. SpaText: Spatio-Textual Representation for Controllable Image Generation
10. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
11. Person Image Synthesis via Denoising Diffusion Model
12. Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models
13. Multi-Concept Customization of Text-to-Image Diffusion
14. ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts
15. Shifted Diffusion for Text-to-image Generation
16. Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models To Learn Any Unseen Style
17. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models

Super-Resolution

1. Image super-resolution via iterative refinement
2. Score-based Generative Neural Networks for Large-Scale Optimal Transport
3. Implicit Diffusion Models for Continuous Super-Resolution

Image Editing

1. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
2. Blended Latent Diffusion
3. [SINE: SINgle Image Editing with Text-to-Image Diffusion Models (https://openaccess.thecvf.com/content/CVPR2023/papers/Zhang_SINE_SINgle_Image_Editing_With_Text-to-Image_Diffusion_Models_CVPR_2023_paper.pdf)
4. Imagic: Text-Based Real Image Editing with Diffusion Models
5. Collaborative Diffusion for Multi-Modal Face Generation and Editing
6. Null-text Inversion for Editing Real Images using Guided Diffusion Models
7. DiffusionRig: Learning Personalized Priors for Facial Appearance Editing
8. [RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation] (https://openaccess.thecvf.com/content/CVPR2023/papers/Anciukevicius_RenderDiffusion_Image_Diffusion_for_3D_Reconstruction_Inpainting_and_Generation_CVPR_2023_paper.pdf)
9. Paint by Example: Exemplar-based Image Editing with Diffusion Models

Region Image Editing

1. Blended diffusion for text-driven editing of natural images

Inpainting

1. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
2. RePaint: Inpainting using Denoising Diffusion Probabilistic Models
3. [RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models] (https://openaccess.thecvf.com/content/CVPR2023/papers/Lei_RGBD2_Generative_Scene_Synthesis_via_Incremental_View_Inpainting_Using_RGBD_CVPR_2023_paper.pdf) 4.SmartBrush: Text and Shape Guided Object Inpainting With Diffusion Model

Image-to-Image Translation

1. Palette: Image-to-Image Diffusion Models
2. UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
3. EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
4. Pretraining is All You Need for Image-to-Image Translation
5. VQBB: Image-to-image Translation with Vector Quantized Brownian Bridge
6. The Swiss Army Knife for Image-to-Image Translation: Multi-Task Diffusion Models
7. Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance
8. BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models
9. Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

Image Segmentation

1. Label-Efficient Semantic Segmentation with Diffusion Models
2. SegDiff: Image Segmentation with Diffusion Probabilistic Models
3. Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion
4. Ambiguous Medical Image Segmentation using Diffusion Models

Multi-Task

1. Generative modeling by estimating gradients of the data distribution
2. Score-Based Generative Modeling through Stochastic Differential Equations
3. ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
4. Learning Energy-Based Models by Diffusion Recovery Likelihood
5. Conditional image generation with score-based diffusion models
6. More control for free! Image synthesis with semantic diffusion guidance
7. ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
8. Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation
9. High-Resolution Image Synthesis with Latent Diffusion Models
10. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
11. Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction
12. DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
13. Understanding DDPM Latent Codes Through Optimal Transport
14. Conditional Simulation Using Diffusion Schrödinger Bridges
15. Retrieval-Augmented Diffusion Models
16. Accelerating Diffusion Models via Early Stop of the Diffusion Process
17. Diffusion Models as Plug-and-Play Priors
18. Non-Uniform Diffusion Models
19. Diffusion Probabilistic Model Made Slim
20. Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
21. On Distillation of Guided Diffusion Model
22. DiffCollage: Parallel Generation of Large Content With Diffusion Models

Medical Image-to-Image Translation

1. Unsupervised Medical Image Translation with Adversarial Diffusion Models
2. Unsupervised Denoising of Retinal OCT with Diffusion Probabilistic Model
3. Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models

Medical Image Generation

1. Solving inverse problems in medical imaging with score-based generative models
2. Score-based diffusion models for accelerated MRI
3. Diffusion Models For Medical Image Analysis: A Comprehensive Survey
4. Low-Dose CT Using Denoising Diffusion Probabilistic Model for 20× Speedup
5. Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models

Medical Image Segmentation

1. Diffusion Models for Implicit Image Segmentation Ensembles
2. Accelerating Diffusion Models via Pre-segmentation Diffusion Sampling for Medical Image Segmentation

Medical Image Anomaly Detection

1. Diffusion Models for Medical Anomaly Detection
2. Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models
3. AnoDDPM: Anomaly Detection With Denoising Diffusion Probabilistic Models Using Simplex Noise
4. What is Healthy? Generative Counterfactual Diffusion for Lesion Localization

Video Generation

1. Video Diffusion Models
2. Diffusion Probabilistic Modeling for Video Generation
3. Flexible Diffusion Modeling of Long Videos
4. Diffusion Models for Video Prediction and Infilling
5. Dreamix: Video Diffusion Models are General Video Editors
6. Conditional Image-to-Video Generation with Latent Flow Diffusion Models
7. Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding
8. DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
9. MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
10. VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
11. Video Probabilistic Diffusion Models in Projected Latent Space
12. Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Few-Shot Image Generation

1. Few-Shot Diffusion Models

Counterfactual Explanations and Estimations

1. Diffusion Models for Counterfactual Explanations
2. Diffusion Causal Models for Counterfactual Estimation

Image Restoration

1. Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models
2. Denoising Diffusion Restoration Models
3. Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition
4. High-resolution image reconstruction with latent diffusion models from human brain activity
5. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding

Image Registration

1. DiffuseMorph: Unsupervised Deformable Image Registration Along Continuous Trajectory Using Diffusion Models

Adversarial Purification

1. Diffusion Models for Adversarial Purification

Semantic Image Generation

1. Semantic Image Synthesis via Diffusion Models

3D Generation

1. 3D shape generation and completion through point-voxel diffusion
2. RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
3. Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model
4. NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
5. Diffusion-SDF: Text-to-Shape via Voxelized Diffusion
6. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
7. DATID-3D: Diversity-Preserved Domain Adaptation Using Text-to-Image Diffusion for 3D Generative Model
8. HOLODIFFUSION: Training a 3D Diffusion Model using 2D Images
9. Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
10. Consistent View Synthesis with Pose-Guided Diffusion Models

Classification

1. Score-based generative classifiers
2. Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images

Point Cloud Generation

1. Diffusion Probabilistic Models for 3D Point Cloud Generation

Theoretical

1. A variational perspective on diffusion-based generative models and score matching
2. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Graphs

1. Generative Diffusion Models on Graphs: Methods and Applications

Deblurring

1. Image Deblurring with Domain Generalizable Diffusion Models

Face Morphing Attack Detection

1. Face Morphing Attack Detection with Denoising Diffusion Probabilistic Models

Trajectory Prediction

1. Leapfrog Diffusion Model for Stochastic Trajectory Prediction

Attacks

1. How to Backdoor Diffusion Models?
2. TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets

Study on data memorization

1.Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models