File Download

There are no files associated with this item.

Supplementary

Conference Paper: DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines

TitleDiffusionPipe: Training Large Diffusion Models with Efficient Pipelines
Authors
Issue Date15-May-2024
Abstract

Diffusion models have emerged as dominant performers for image generation. To support training large diffusion models, this paper studies pipeline parallel training of diffusion models and proposes DiffusionPipe, a synchronous pipeline training system that advocates innovative pipeline bubble filling technique, catering to structural characteristics of diffusion models. State-of-the-art diffusion models typically include trainable (the backbone) and non-trainable (e.g., frozen input encoders) parts. We first unify optimal stage partitioning and pipeline scheduling of single and multiple backbones in representative diffusion models with a dynamic programming approach. We then propose to fill the computation of non-trainable model parts into idle periods of the pipeline training of the backbones by an efficient greedy algorithm, thus achieving high training throughput. Extensive experiments show that DiffusionPipe can achieve up to 1.41x speedup over pipeline parallel methods and 1.28x speedup over data parallel training on popular diffusion models.


Persistent Identifierhttp://hdl.handle.net/10722/347513

 

DC FieldValueLanguage
dc.contributor.authorTian, Ye-
dc.contributor.authorJia, Zhen-
dc.contributor.authorLuo, Ziyue-
dc.contributor.authorWang, Yida-
dc.contributor.authorWu, Chuan-
dc.date.accessioned2024-09-24T00:30:41Z-
dc.date.available2024-09-24T00:30:41Z-
dc.date.issued2024-05-15-
dc.identifier.urihttp://hdl.handle.net/10722/347513-
dc.description.abstract<p>Diffusion models have emerged as dominant performers for image generation. To support training large diffusion models, this paper studies pipeline parallel training of diffusion models and proposes DiffusionPipe, a synchronous pipeline training system that advocates innovative pipeline bubble filling technique, catering to structural characteristics of diffusion models. State-of-the-art diffusion models typically include trainable (the backbone) and non-trainable (e.g., frozen input encoders) parts. We first unify optimal stage partitioning and pipeline scheduling of single and multiple backbones in representative diffusion models with a dynamic programming approach. We then propose to fill the computation of non-trainable model parts into idle periods of the pipeline training of the backbones by an efficient greedy algorithm, thus achieving high training throughput. Extensive experiments show that DiffusionPipe can achieve up to 1.41x speedup over pipeline parallel methods and 1.28x speedup over data parallel training on popular diffusion models.<br></p>-
dc.languageeng-
dc.relation.ispartofThe Seventh Conference on Machine Learning and Systems (MLSys) (13/05/2024-16/05/2024, Santa Clara)-
dc.titleDiffusionPipe: Training Large Diffusion Models with Efficient Pipelines-
dc.typeConference_Paper-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats