Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP

• Author(s): Dr. Madumere Smart Onyemaechi; Frank Uchehara O.
• Paper ID: 1718150
• Page: 3019-3021
• Published Date: 31-01-2026
• Published In: Iconic Research And Engineering Journals
• Publisher: IRE Journals
• e-ISSN: 2456-8880
• Volume/Issue: Volume 9 Issue 7 January-2026

Abstract

Vision-Language Models like CLIP and SigLIP align images and text in a shared embedding space, but the internal mechanisms by which they bind visual concepts to language remain opaque. We apply mechanistic interpretability methods to localize and characterize circuits responsible for cross-modal concept binding in CLIP-ViT-B/32 and SigLIP-SO400M. Using causal tracing, sparse autoencoders, and activation patching, we identify three functional subsystems: early visual feature extraction, cross-attention-mediated grounding, and late-stage concept fusion. We find that 12–18% of MLP neurons and 7% of attention heads exhibit polysemantic, modality-invariant concept selectivity for objects, colors, and relations. Causal intervention on these circuits produces predictable changes in image-text alignment scores, confirming causal roles. Our analysis reveals that concept binding relies on a small set of highly interpretable circuits rather than distributed representations. These findings provide a foundation for targeted model editing, bias mitigation, and robust multimodal alignment.

Keywords

mechanistic interpretability, vision-language models, CLIP, concept binding, sparse autoencoders, causal tracing

Citations

IRE Journals:
Dr. Madumere Smart Onyemaechi, Frank Uchehara O. "Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP" Iconic Research And Engineering Journals Volume 9 Issue 7 2026 Page 3019-3021

IEEE:
Dr. Madumere Smart Onyemaechi, Frank Uchehara O. "Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP" Iconic Research And Engineering Journals, vol. 9, no. 7, Jan. 2026

APA:
Dr. Madumere Smart Onyemaechi, Frank Uchehara O. (2026). Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP. Iconic Research And Engineering Journals, 9(7).

MLA:
Dr. Madumere Smart Onyemaechi, Frank Uchehara O. "Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP" Iconic Research And Engineering Journals, vol. 9, no. 7, Jan. 2026.

BibTeX

@article{1718150,
author = {Dr. Madumere Smart Onyemaechi, Frank Uchehara O.},
title = {Mechanistic Interpretability of Vision-Language Models: Tracing Multimodal Concept Binding in CLIP and SigLIP},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {9},
number = {7},
pages = {3019-3021},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1718150.pdf},
abstract = {Vision-Language Models like CLIP and SigLIP align images and text in a shared embedding space, but the internal mechanisms by which they bind visual concepts to language remain opaque. We apply mechanistic interpretability methods to localize and characterize circuits responsible for cross-modal concept binding in CLIP-ViT-B/32 and SigLIP-SO400M. Using causal tracing, sparse autoencoders, and activation patching, we identify three functional subsystems: early visual feature extraction, cross-attention-mediated grounding, and late-stage concept fusion. We find that 12–18% of MLP neurons and 7% of attention heads exhibit polysemantic, modality-invariant concept selectivity for objects, colors, and relations. Causal intervention on these circuits produces predictable changes in image-text alignment scores, confirming causal roles. Our analysis reveals that concept binding relies on a small set of highly interpretable circuits rather than distributed representations. These findings provide a foundation for targeted model editing, bias mitigation, and robust multimodal alignment.},
keywords = {mechanistic interpretability, vision-language models, CLIP, concept binding, sparse autoencoders, causal tracing},
month = {January}
}