Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding

Ravi Deshmukh; Hannah Lindqvist; Wei Zhang

Home / Current Issue / Paper 1722527

1722527 Vol 7 · Issue 6 Download Paper

Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding

Ravi Deshmukh Hannah Lindqvist Wei Zhang

Subject area: Science,Engineering and Technology · Area of research: Cross-Modal Scene Understanding

Abstract

This paper introduces a dual-stream transformer for cross-modal scene understanding that keeps separate image and text encoders while coupling them through shared attention at every layer. Built on a DeiT-B vision backbone and a RoBERTa text encoder, it converts captions to segmentation masks and back on Mapillary Vistas and SUN-RGBD. The dual-stream design reaches 66.8% mIoU and a BLEU-4 of 31, exceeding a diffusion-based segmentation baseline on both directions.

How to cite this paper

Ravi Deshmukh, Hannah Lindqvist, Wei Zhang "Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding" Iconic Research And Engineering Journals Volume 7 Issue 6 2023 Page 682-687

Ravi Deshmukh, Hannah Lindqvist, Wei Zhang "Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding" Iconic Research And Engineering Journals, vol. 7, no. 6, Dec. 2023

Ravi Deshmukh, Hannah Lindqvist, Wei Zhang (2023). Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding. Iconic Research And Engineering Journals, 7(6).

Ravi Deshmukh, Hannah Lindqvist, Wei Zhang "Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding" Iconic Research And Engineering Journals, vol. 7, no. 6, Dec. 2023.

@article{1722527,
      author = {Ravi Deshmukh, Hannah Lindqvist, Wei Zhang},
      title = {Text-to-Mask and Mask-to-Text: A Dual-Stream Transformer for Cross-Modal Scene Understanding},
      journal = {Iconic Research And Engineering Journals},
      year = {2023},
      volume = {7},
      number = {6},
      pages = {682-687},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1722527.pdf},
      abstract = {This paper introduces a dual-stream transformer for cross-modal scene understanding that keeps separate image and text encoders while coupling them through shared attention at every layer. Built on a DeiT-B vision backbone and a RoBERTa text encoder, it converts captions to segmentation masks and back on Mapillary Vistas and SUN-RGBD. The dual-stream design reaches 66.8% mIoU and a BLEU-4 of 31, exceeding a diffusion-based segmentation baseline on both directions.},
      month = {December},
  }