Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models

• Author(s): Shivam Singh
• Paper ID: 1708190
• Page: 31-42
• Published Date: 02-05-2025
• Published In: Iconic Research And Engineering Journals
• Publisher: IRE Journals
• e-ISSN: 2456-8880
• Volume/Issue: Volume 8 Issue 11 May-2025

Abstract

This paper presents an innovative framework for improving computationally constrained image generation models by distilling knowledge from more powerful but resource-intensive models. We demonstrate that Stable Diffusion XL (SDXL) can generate high-fidelity dog images that effectively train a smaller Stable Diffusion 1.5 model via Low-Rank Adaptation (LoRA). Our method eliminates the need for real-world data collection while achieving significant improvements in perceptual quality (31.46% SSIM increase in standard poses, p < 0.001) and structural accuracy. Through extensive evaluation using multiple metrics (SSIM, MSE, histogram similarity, perceptual hash similarity, FID score, and LPIPS), we reveal that knowledge transfer between diffusion models follows a hierarchical pattern where coarse structural features transfer more readily than fine details. We observe context- dependent performance variations, with dramatic improvement in standard poses and challenging scenarios but limitations in closeup details. Our findings demonstrate that extremely parameter- efficient adaptation (2.8MB) can achieve substantial quality improvements in resource-constrained environments, offering a promising pathway toward self-improving AI ecosystems with bidirectional knowledge flow between models of different capabilities.

Citations

IRE Journals:
Shivam Singh "Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models" Iconic Research And Engineering Journals Volume 8 Issue 11 2025 Page 31-42

IEEE:
Shivam Singh "Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models" Iconic Research And Engineering Journals, 8(11)