Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models

References

[1] Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., & Vetter, T. (2019). Analyz-

[2] ing and reducing the damage of dataset bias to face recognition with synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 2261- 2268).

[3] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851.

[4] Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34, 8780-8794.

[5] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10684-10695).

[6] Podell, D., English, K., Lacey, A., Blattmann, A., Black, S., Goh, G., Huot, M., Lee, J., Luccioni, A., Uesato, J., & others. (2023). SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.

[7] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., & others. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (pp. 8748-8763).

[8] Sariyildiz, M. B., Kalantidis, Y., Larlus, D., & Alahari, K. (2023). Fake it till you make it: Learn- ing transferable representations from synthetic ImageNet clones. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20391-20401).

[9] Li, X., Chen, Z., Panda, R., Karlinsky, L., Darrell, T., & Saenko, K. (2022). Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4750-4759).

[10] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

[11] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (pp. 2790-2799).

[12] Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3045-3059).

[13] Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (pp. 4582-4597).

[14] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741.

[15] Wang, P., Wu, Y., Peebles, W., Zhang, H., Lu, J., & Efros, A. A. (2023). Image quality assessment for text-to-image generation: A benchmark and objective metric. arXiv preprint arXiv:2305.10355.

[16] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Analyzing and im- proving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8110-8119).

[17] He, Z., Shakeri, M., Zhang, H., Lee, K., Walshe, C., Kanazawa, A., & others. (2022). Synthetic data in vision: Opportunities and challenges. arXiv preprint arXiv:2203.10674.

[18] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 586-595).

[19] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30.

[20] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612.

[21] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

@article{1708190,
      author = {Shivam Singh},
      title = {Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models},
      journal = {Iconic Research And Engineering Journals},
      year = {2025},
      volume = {8},
      number = {11},
      pages = {31-42},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1708190.pdf},
      abstract = {This paper presents an innovative framework for improving computationally constrained image generation models by distilling knowledge from more powerful but resource-intensive models. We demonstrate that Stable Diffusion XL (SDXL) can generate high-fidelity dog images that effectively train a smaller Stable Diffusion 1.5 model via Low-Rank Adaptation (LoRA). Our method eliminates the need for real-world data collection while achieving significant improvements in perceptual quality (31.46% SSIM increase in standard poses, p < 0.001) and structural accuracy. Through extensive evaluation using multiple metrics (SSIM, MSE, histogram similarity, perceptual hash similarity, FID score, and LPIPS), we reveal that knowledge transfer between diffusion models follows a hierarchical pattern where coarse structural features transfer more readily than fine details. We observe context- dependent performance variations, with dramatic improvement in standard poses and challenging scenarios but limitations in closeup details. Our findings demonstrate that extremely parameter- efficient adaptation (2.8MB) can achieve substantial quality improvements in resource-constrained environments, offering a promising pathway toward self-improving AI ecosystems with bidirectional knowledge flow between models of different capabilities.},
      month = {May},
  }