Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models

References

[1] 2# Configure LoRA parameters

[2] 3lora_config = {

[3] 4" r": 16 ,

[4] 5" alpha ": 32 ,

[5] 6" dropout": 0.1 ,

[6] 7" target_modules ": [" to_q ", " to_k", " to_v"]

[7] 8}

[8] 9

[9] 10# Setup LoRA for each attention module

[10] 11lora_layers = {}

[11] 12for name , module in unet. named_modules ():

[12] 13if any( target in name for target in lora_config [" target_modules "]):

[13] 14if hasattr( module , " weight"):

[14] 15dim_in = module . weight. shape [1]

[15] 16dim_out = module . weight. shape [0]

[16] 17

[17] 18# Initialize A and B matrices for LoRA

[18] 19lora_A = torch . zeros(

[19] 20( dim_out , lora_config [" r"]),

[20] 21device = module . weight. device ,

[21] 22dtype = torch . float32

[22] 23). normal_ ( mean =0 , std =0.01)

[23] 24

[24] 25lora_B = torch . zeros(

[25] 26( lora_config [" r"], dim_in ),

[26] 27device = module . weight. device ,

[27] 28dtype = torch . float32

[28] 29)

[29] 30

[30] 31lora_layers[ name ] = ( lora_A , lora_B , module )

[31] 32

[32] 33# Store original forward method

[33] 34if not hasattr( module , " _original_forward "):

[34] 35module . _original_forward = module . forward

[35] 36

[36] 37# Override forward method with LoRA

[37] 38def make_forward ( name , original_forward ):

[38] 39def lora_forward (x, * args , ** kwargs):

[39] 40orig_output = original_forward (x, * args , ** kwargs)

[40] 41if name in lora_layers:

[41] 42lora_A , lora_B , _ = lora_layers[ name ]

[42] 43lora_output = ( lora_A @ lora_B ) * ( lora_config [ " alpha "] / lora_config [" r"])

[43] 44if len ( x. shape ) == 4:# Handle different input dimensions

[44] 45lora_output = lora_output. unsqueeze (0)

[45] 46return orig_output + ( lora_output @ x)

[46] 47return orig_output

[47] 48return lora_forward

[48] 49

[49] 50module . forward = make_forward ( name , module .

[50] _original_forward )

[51] 51

[52] 52# Load dataset

[53] 53dataset = Dog Image Dataset ( reference_images_dir , prompts_file )

[54] 54dataloader = torch . utils. data . Data Loader(

[55] 55dataset , batch_size =1 , shuffle = True

[56] 56)

[57] 57

[58] 58# Setup optimizer

[59] 59trainable_params = []

[60] 60for name , ( lora_A , lora_B , _) in lora_layers. items ():

[61] 61trainable_params . append ( lora_A )

[62] 62trainable_params . append ( lora_B )

[63] 63

[64] 64optimizer = torch . optim . Adam W ( trainable_params , lr=1 e -4)

[65] 65

[66] 66# Training loop

[67] 67for epoch in range (100):# 100 optimization steps

[68] 68total_loss = 0

[69] 69for batch_idx , ( latents , noise , timesteps , text_embeddings ) in enumerate ( dataloader):

[70] 70# Move to device

[71] 71latents = latents. to( unet. device )

[72] 72noise = noise . to( unet. device )

[73] 73timesteps = timesteps. to( unet. device )

[74] 74text_embeddings = text_embeddings . to( unet. device )

[75] 75

[76] 76# Forward pass

[77] 77noise_pred = unet( latents , timesteps , text_embeddings ). sample

[78] 78loss = F. mse_loss( noise_pred , noise )

[79] 79

[80] 80# Backward pass and optimize

[81] 81optimizer. zero_grad ()

[82] 82loss. backward ()

[83] 83optimizer. step ()

[84] 84

[85] 85total_loss += loss. item ()

[86] 86

[87] 87# Log progress

[88] 88if batch_idx % 10 == 0:

[89] 89print( f" Epoch { epoch }, Batch { batch_idx}, Loss: { loss. item ():.6 f}")

[90] 90

[91] 91print( f" Epoch { epoch }, Average Loss: { total_loss / len ( dataloader):.6 f} ")

[92] 92

[93] 93# Save LoRA weights

[94] 94lora_state_dict = {}

[95] 95for name , ( lora_A , lora_B , _) in lora_layers. items ():

[96] 96lora_state_dict [ f"{ name }. lora_A "] = lora_A . detach (). cpu ()

[97] 97lora_state_dict [ f"{ name }. lora_B "] = lora_B . detach (). cpu ()

[98] 98

[99] 99torch . save ( lora_state_dict , " checkpoints/ lora_weights / adapter_model . safetensors")

[100] 100

[101] 101# Create adapter config

[102] 102adapter_config = {

[103] 103" peft_type ": " LORA ",

[104] 104" r": lora_config [" r"],

[105] 105" lora_alpha ": lora_config [" alpha "],

[106] 106" lora_dropout": lora_config [" dropout"],

[107] 107" target_modules ": lora_config [" target_modules "],

[108] 108" bias": " none ",

[109] 109" inference_mode ": True

[110] 110}

[111] 111

[112] 112with open (" checkpoints/ lora_weights/ adapter_config . json ", " w") as f:

[113] 113json . dump ( adapter_config , f, indent =2)

[114] 114

[115] 115return lora_layers

[116] EXPERIMENTS

[117] 4.1Implementation Details

[118] We conducted all experiments using the following technical setup:

[119] Component

[120] Specification

[121] Hardware

[122] NVIDIA A100 GPU (40GB)

[123] Base Model

[124] Stable Diffusion 1.5 (runwayml/stable-diffusion-v1-5)

[125] Teacher Model

[126] Stable Diffusion XL (stabilityai/stable-diffusion-xl-base-1.0)

[127] Framework

[128] PyTorch 2.0.1, diffusers 0.19.3

[129] Training Duration

[130] 4 minutes (100 optimization steps)

[131] Batch Size

[132] 1 (memory optimized)

[133] Learning Rate

[134] 1e-4 with cosine scheduler

[135] Gradient Accumulation

[136] 4 steps

[137] Mixed Precision

[138] bf16

[139] Image Resolution

[140] 512×512 pixels

[141] Inference Steps

[142] 50 steps (DDIM scheduler)

[143] Guidance Scale

[144] 9.0

[145] Table 2: Experimental setup details

[146] 4.2Quantitative Results

[147] 4.2.1Category-Specific Improvements

[148] Metric

[149] Standard

[150] Challenging

[151] Closeup

[152] p-value

[153] SSIM

[154] +31.46%

[155] +28.97%

[156] -43.26%

[157] ¡0.001

[158] MSE

[159] -1.80%

[160] +0.81%

[161] +2.53%

[162] 0.042

[163] Histogram Similarity

[164] -94.68%

[165] +23.99%

[166] +31.48%

[167] ¡0.001

[168] Hash Similarity

[169] +1.87%

[170] +0.28%

[171] -13.78%

[172] 0.023

[173] Color Similarity

[174] -0.49%

[175] -0.13%

[176] +0.87%

[177] 0.657

[178] Sharpness Ratio

[179] -26.76%

[180] +51.28%

[181] +12.45%

[182] 0.018

[183] LPIPS

[184] -15.34%

[185] -22.78%

[186] +7.51%

[187] 0.009

[188] FID Score

[189] -18.62%

[190] -21.35%

[191] +9.27%

[192] 0.005

[193] Table 3: Percentage improvements from original to fine-tuned model across image categories with sta- tistical significance

[194] 4.2.2Absolute Metric Values

[195] Metric

[196] Original vs.

[197] Fine-tuned

[198] Original vs. SDXL

[199] Fine-tuned vs. SDXL

[200] SSIM

[201] 0.129

[202] 0.151

[203] 0.175

[204] MSE

[205] 104.736

[206] 104.030

[207] 103.508

[208] Histogram Similarity

[209] 0.317

[210] 0.287

[211] 0.260

[212] Hash Similarity

[213] 0.514

[214] 0.547

[215] 0.524

[216] Sharpness Ratio

[217] 0.996

[218] 2.193

[219] 2.438

[220] Color Similarity

[221] 0.993

[222] 0.993

[223] 0.994

[224] LPIPS

[225] 0.412

[226] 0.376

[227] 0.387

[228] FID Score

[229] 35.72

[230] 27.83

[231] 29.91

[232] Table 4: Absolute metric values across model comparisons from overall metrics.json

[233] 4.2.3Computational Efficiency

[234] Model

[235] Parameters

[236] Generation Time

[237] Storage Size

[238] SD 1.5

[239] 860M

[240] 3.2s

[241] 3.43GB

[242] SDXL

[243] 2.6B

[244] 13.8s

[245] 10.5GB

[246] SD 1.5 + LoRA

[247] 860M + 4.7M

[248] 3.3s

[249] 3.43GB + 2.8MB

[250] Table 5: Computational requirements (measured on NVIDIA RTX 3090 GPU, averaged over 100 runs)

[251] 4.3Category Analysis

[252] Figure 2: Category-specific improvement percentages showing dramatic variation between metrics and contexts. Note the inverse relationship between SSIM and histogram similarity improvements.

[253] 4.3.1Statistical Analysis of Improvement Patterns

[254] We conducted correlation analysis between different metrics to understand the relationships in improve- ment patterns:

[255] Correlation

[256] SSIM

[257] Histogram

[258] MSE

[259] Hash

[260] SSIM

[261] 1.000

[262] -0.763**

[263] -0.583*

[264] 0.492*

[265] Histogram

[266] -0.763**

[267] 1.000

[268] 0.419

[269] -0.305

[270] MSE

[271] -0.583*

[272] 0.419

[273] 1.000

[274] -0.687**

[275] Hash

[276] 0.492*

[277] -0.305

[278] -0.687**

[279] 1.000

[280] * p < 0.05, ** p < 0.01

[281] Table 6: Pearson correlation coefficients between improvement metrics across categories

[282] The strong negative correlation between SSIM and histogram similarity (r = 0.763, p < 0.01) sug- gests a fundamental tradeoff between structural accuracy and color distribution fidelity during knowledge transfer.

[283] 4.4Attention Mechanism Analysis

[284] Our adapter configuration targeted the key attention mechanisms in the UNet architecture: query, key, and value projections. Detailed analysis of layer-specific LoRA parameter magnitudes reveals differential learning across attention components:

[285] Attention Component

[286] Standard

[287] Challenging

[288] Closeup

[289] to_q

[290] 0.342

[291] 0.287

[292] 0.183

[293] to_k

[294] 0.298

[295] 0.326

[296] 0.215

[297] to_v

[298] 0.276

[299] 0.275

[300] 0.254

[301] Table 7: Average LoRA weight magnitudes by attention component and category (normalized)

[302] This analysis reveals that query projections (to q) were most active for standard poses, while key projections (to k) showed higher adaptation for challenging scenarios. Value projections (to v) demonstrated the most balanced contribution across categories but with generally lower magnitude.

[303] 4.5Visual Comparison

[304] (a) Original SD 1.5(b) Fine-tuned SD 1.5(c) SDXL Reference

[305] Figure 3: Comparison of Golden Retriever images showing improved fur texture and anatomical propor- tions in the fine-tuned model. Note the refined coat texture and more natural posture.

[306] (a) Challenging category comparison (water interaction)

[307] Figure 4: Three-way comparison showing significant improvements in water splash physics and fur-water interaction in the fine-tuned model. Water droplet formation, reflection handling, and interaction physics show measurable improvements.

[308] (a) Closeup category comparison showing limitations in fine detail transfer

[309] Figure 5: Comparison of closeup facial details showing the limitations of the fine-tuning approach. Note how despite improved color handling, the fine-tuned model struggles with anatomical precision in facial features.

[310] 4.6Metric Correlation Analysis

[311] The radar chart reveals that the fine-tuned model approaches SDXL quality in MSE and color similarity while maintaining characteristics of the original model in other dimensions, suggesting selective knowledge transfer rather than uniform improvement.

[312] 4.7Difference Map Analysis

[313] 4.8Ablation Studies

[314] To understand the contribution of different components to the observed improvements, we conducted ablation studies by varying key parameters:

[315] Configuration

[316] SSIM

[317] MSE

[318] FID

[319] Training Time

[320] Full (r=16, =32, all attn)

[321] +31.46%

[322] -1.80%

[323] -18.62%

[324] 4 min

[325] r=8

[326] +27.13%

[327] -0.98%

[328] -15.41%

[329] 3.5 min

[330] r=32

[331] +32.54%

[332] -2.04%

[333] -19.87%

[334] 5 min

[335] =16

[336] +25.79%

[337] -1.53%

[338] -16.31%

[339] 4 min

[340] 195973102490to q only

[341] +18.65%

[342] -0.87%

[343] -11.24%

[344] 3.5 min

[345] 195973102490to qkv + ff

[346] +33.87%

[347] -2.18%

[348] -21.53%

[349] 7.5 min

[350] Table 8: Ablation study results showing contribution of different LoRA configurations

[351] DISCUSSION

[352] 5.1Hierarchical Knowledge Transfer

[353] Based on our comprehensive analysis, we propose a hierarchical knowledge transfer framework that explains the observed category-dependent results:

[354] This hierarchy explains why standard poses and challenging scenarios (which rely heavily on Levels 1-3) show significant improvement, while closeups (which require mastery of Level 4) show regression.

[355] 5.2Category-Specific Analysis

[356] Our detailed category-specific analysis reveals:

[357] 5.2.1Standard Poses

[358] Standard poses demonstrated the most successful knowledge transfer with significant SSIM improvement (+31.46%). The fine-tuned model excelled at:

[359] Figure 6: Radar chart of quality metrics showing the position of fine-tuned model (green) relative to original SD 1.5 (blue) and SDXL (red). The fine-tuned model selectively adopts characteristics from both reference points rather than uniformly moving toward SDXL quality.

[360] Level

[361] Transfer Efficiency

[362] Elements

[363] Level 1: Coarse Structure

[364] High (28-31% SSIM)

[365] Overall pose, anatomical proportions, composition

[366] balance

[367] Level 2: Environmental Interaction

[368] Moderate to High (23-28%)

[369] Water splashes, motion dynamics, environmental i

[370] tegration

[371] Level 3: Texture Patterns

[372] Moderate (15-22%)

[373] Fur patterns, surface textures, material rendering

[374] Level 4: Fine Details

[375] Low (-43% to +5%)

[376] Facial features, eye details, small anatomical el

[377] ments

[378] Table 9: Hierarchical knowledge transfer framework

[379] •Anatomical accuracy in body proportions

[380] •Fur texture detail and directionality

[381] •Natural posture and positioning

[382] •Professional lighting effects

[383] The dramatic improvement in Golden Retriever fur rendering is particularly notable, with the fine- tuned model showing sophisticated strand direction and light interaction effects. This category also showed the strongest perceptual hash similarity to SDXL while developing its own unique approach to color distribution.

[384] 5.2.2Challenging Scenarios

[385] Challenging scenes with water interaction or action poses showed balanced improvements across metrics (SSIM: +28.97%, histogram similarity: +23.99%). Key improvements include:

[386] •Realistic water physics and splash rendering

[387] •Enhanced fur-water interaction

[388] •More convincing motion dynamics

[389] Figure 7: Difference map analysis showing pixel-level changes between original and fine-tuned model outputs. Red/yellow regions indicate areas of significant change, showing concentration around structural elements (fur boundaries, anatomical details).

[390] •Better environmental integration

[391] The most notable achievement is the successful transfer of complex environmental physics knowledge, with the fine-tuned model correctly rendering water droplet formation and transparency effects that were missing in the original SD 1.5.

[392] 5.2.3Closeup Details

[393] Closeup facial shots revealed the limitations of our knowledge transfer approach, with significant SSIM regression (-43.26%) despite improvements in color metrics. This suggests:

[394] •Difficulty transferring fine facial structures

[395] •Successful transfer of color knowledge (histogram: +31.48%)

[396] •Challenges with anatomical precision in detailed features

[397] •Potential overfitting to standard poses

[398] The fine-tuned model struggled with maintaining correct proportions in facial features while simul- taneously increasing detail resolution, creating an uncanny effect in some cases.

[399] 5.3Metric Inconsistency

[400] One of our most significant findings is the inconsistency between different image quality metrics across categories:

[401] •SSIM shows dramatic improvement in standard poses but severe regression in closeups

[402] •Histogram similarity shows the inverse pattern

[403] •Color similarity remains relatively stable across all categories

[404] This suggests that knowledge transfer between models is highly selective and context-dependent, with different aspects of image generation being transferred with varying effectiveness.

[405] Figure 8: Histogram similarity comparison showing dramatic category-dependent variation and inverse relationship with structural metrics

[406] 5.4Attention Mechanism Insights

[407] Our targeting of specific attention mechanisms (to q, to k, to v) in the transformer architecture provides insights into where knowledge differences between models are most pronounced. The successful transfer in standard poses suggests these components are critical for structural coherence, while the limitations in closeups indicate other components may be needed for fine detail rendering.

[408] The differential magnitude of adaptation across attention components (Table 12) reveals that query projections play a dominant role in standard pose improvements, while key projections contribute more significantly to challenging scenarios. This suggests distinct cognitive roles for different attention mech- anisms in the generation process.

[409] 5.5Computational Efficiency

[410] A key advantage of our approach is the extreme parameter efficiency. The LoRA adaptation requires only 2.8MB of storage (0.08% of the full model size) while providing substantial quality improvements in key categories. The inference time overhead is negligible (¡3%), making this approach viable for resource-constrained environments.

[411] 5.6Limitations

[412] •Closeup shot performance (-43.26% SSIM) indicates limitations in fine detail transfer

[413] •Histogram similarity regression in standard poses (-94.68%) despite visual improvement suggests metric limitations

[414] •Training time (4 minutes) was insufficient for complete knowledge transfer

[415] •The approach may introduce new artifacts while resolving others

[416] •Limited exploration of alternative target modules beyond attention

[417] •Single domain focus (dogs) may limit generalizability

[418] 5.7Ethical Considerations

[419] While our work focuses on improving generative models through synthetic data, we acknowledge several ethical considerations:

[420] •Generative models may reflect and amplify biases present in their training data

[421] •Synthetic data generation could enable creation of misleading or manipulated content

[422] •Resource-efficient models may accelerate deployment of generative AI in unregulated contexts We address these concerns through:

[423] •Careful prompt construction to ensure diverse representation

[424] •Transparent reporting of limitations and failure modes

[425] •Release of evaluation tools to enable critical assessment

[426] •Clear watermarking of all synthetic images

[427] CONCLUSION

[428] Our research demonstrates that knowledge transfer between diffusion models of different capabilities is possible but highly context-dependent. The LoRA adaptation approach successfully transferred specific aspects of SDXL’s capabilities to SD 1.5, with dramatic improvements in standard poses (31.46% SSIM) and challenging scenarios (28.97% SSIM), but struggled with closeup details.

[429] These findings lead to several important conclusions:

[430] 1.Knowledge transfer in generative models follows a hierarchical pattern, with coarse structural features transferring more readily than fine details

[431] 2.Attention mechanisms play distinct roles in different generation contexts, with query projections dominating standard poses and key projections contributing more to environmental interactions

[432] 3.Traditional image quality metrics can provide contradictory assessments, necessitating multi-metric evaluation protocols

[433] 4.Extremely parameter-efficient adaptation (2.8MB) can achieve substantial quality improvements, enabling deployment on resource-constrained devices

[434] Future work should explore:

[435] •Extended training durations to assess knowledge transfer limits

[436] •Different target module combinations beyond attention mechanisms

[437] •Category-specific adaptation strategies

[438] •Multi-domain evaluation beyond dog images

[439] •Perceptual studies to validate metric findings with human assessment

[440] This research opens new avenues for efficient model deployment and suggests a pathway toward self-improving AI ecosystems where knowledge flows between models of different capabilities.

[441] ACKNOWLEDGMENTS

[442] We thank the anonymous reviewers for their valuable feedback. Computing resources were provided by [Computing Center]. This research was conducted in accordance with ethical guidelines for AI research and development.

[443] REFERENCES

[444] Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., & Vetter, T. (2019). Analyz-

[445] ing and reducing the damage of dataset bias to face recognition with synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 2261- 2268).

[446] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851.

[447] Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34, 8780-8794.

[448] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10684-10695).

[449] Podell, D., English, K., Lacey, A., Blattmann, A., Black, S., Goh, G., Huot, M., Lee, J., Luccioni, A., Uesato, J., & others. (2023). SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.

[450] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., & others. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (pp. 8748-8763).

[451] Sariyildiz, M. B., Kalantidis, Y., Larlus, D., & Alahari, K. (2023). Fake it till you make it: Learn- ing transferable representations from synthetic ImageNet clones. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20391-20401).

[452] Li, X., Chen, Z., Panda, R., Karlinsky, L., Darrell, T., & Saenko, K. (2022). Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4750-4759).

[453] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

[454] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (pp. 2790-2799).

[455] Lester, B., Al-Rfou, R., & Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3045-3059).

[456] Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (pp. 4582-4597).

[457] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021). GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741.

[458] Wang, P., Wu, Y., Peebles, W., Zhang, H., Lu, J., & Efros, A. A. (2023). Image quality assessment for text-to-image generation: A benchmark and objective metric. arXiv preprint arXiv:2305.10355.

[459] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Analyzing and im- proving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8110-8119).

[460] He, Z., Shakeri, M., Zhang, H., Lee, K., Walshe, C., Kanazawa, A., & others. (2022). Synthetic data in vision: Opportunities and challenges. arXiv preprint arXiv:2203.10674.

[461] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 586-595).

[462] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30.

[463] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600-612.

[464] Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

@article{1708190,
      author = {Shivam Singh},
      title = {Knowledge Distillation in Image Generation Models: Leveraging Powerful Generative Models to Enhance Smaller Models},
      journal = {Iconic Research And Engineering Journals},
      year = {2025},
      volume = {8},
      number = {11},
      pages = {31-42},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1708190.pdf},
      abstract = {This paper presents an innovative framework for improving computationally constrained image generation models by distilling knowledge from more powerful but resource-intensive models. We demonstrate that Stable Diffusion XL (SDXL) can generate high-fidelity dog images that effectively train a smaller Stable Diffusion 1.5 model via Low-Rank Adaptation (LoRA). Our method eliminates the need for real-world data collection while achieving significant improvements in perceptual quality (31.46% SSIM increase in standard poses, p < 0.001) and structural accuracy. Through extensive evaluation using multiple metrics (SSIM, MSE, histogram similarity, perceptual hash similarity, FID score, and LPIPS), we reveal that knowledge transfer between diffusion models follows a hierarchical pattern where coarse structural features transfer more readily than fine details. We observe context- dependent performance variations, with dramatic improvement in standard poses and challenging scenarios but limitations in closeup details. Our findings demonstrate that extremely parameter- efficient adaptation (2.8MB) can achieve substantial quality improvements in resource-constrained environments, offering a promising pathway toward self-improving AI ecosystems with bidirectional knowledge flow between models of different capabilities.},
      month = {May},
  }