SpatialMoR-VGGT: Spatially Adaptive Efficient 3D Scene Reconstruction

References

[1] Bommasani, R., et al. (2021). On the opportunities and risks of foun dation models. arXiv preprint arXiv:2108.07258.

[2] Brown, T., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 18771-18790.

[3] Chen, X., et al. (2020). Generative pretraining from pixels. In Interna tional conference on machine learning (pp. 1691-1703). PMLR.

[4] Fedus, W., et al. (2021). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961.

[5] He, K., et al. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9729-9738).

[6] Jia, M., et al. (2022). Visual prompt tuning. In European conference on computer vision (pp. 709-727). Springer, Cham.

[7] Liu, Z., et al. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 10012-10022).

[8] Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

[9] Rombach, R., et al. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).

[10] Touvron, H., et al. (2021). Training data-efficient image transformers & distillation through attention. In International conference on machine learning (pp. 10347-10357). PMLR.

[11] Vaswani, A., et al. (2017). Attention is all you need. Advances in neural information processing systems, 30.

[12] Bengio, Y., et al. (2013). Deep generative stochastic networks trainable by backprop. In International conference on machine learning (pp. 226 234). PMLR.

[13] Wang, J., et al. (2024). VGGT: Visual Geometry Grounded Transformer. arXiv preprint arXiv:2403.11651.

[14] Min, R., et al. (2024). Mixture-of-Recursions: Learning Dynamic Re cursive Depths for Adaptive Token-Level Computation. arXiv preprint arXiv:2407.10524.

[15] Lepikhin, D., et al. (2020). Gshard: Scaling giant models with sparse parallelism and pipeline parallelism. arXiv preprint arXiv:2006.16668.

[16] Dai, B., et al. (2022). SpMoE: Learning Mixture of Experts via Soft Permutation. NeurIPS.

[17] Dai, Y., et al. (2024). Early-Exit Mechanisms for Efficient Reasoning. arXiv preprint arXiv:2401.03215.

[18] Teerapittayanon, S., et al. (2016). Branchynet: Fast inference via early exiting from deep neural networks. In 2016 international conference on pattern recognition (ICPR) (pp. 2464-2469). IEEE.

[19] Shazeer, N. (2017). Outrageously large neural networks: The sparsely gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

[20] Xu, J., et al. (2022). Mofq: Mixture of few-shot quants for parameter efficient transfer learning. In International Conference on Machine Learning (pp. 24781-24797). PMLR.

[21] Chen, M., et al. (2023). DUSt3R: Geometric 3D Vision Made Easy. arXiv preprint arXiv:2312.02101.

[22] Pumarola, A., et al. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. CVPR.

[23] Raposo, D., et al. (2024). Mixture-of-Depths. arXiv preprint arXiv:2403.13373.

[24] Jacob, B., et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR.

[25] Han, S., et al. (2015). Learning both Weights and Connections for Efficient Neural Networks. NeurIPS.

[26] Zhang, H., et al. (2022). MixVPR: A universal visual place recognition network with mixture-of-experts. In 2022 International Conference on Robotics and Automation (ICRA) (pp. 13115-13121). IEEE.

[27] Wang, X., et al. (2023). MoE-LLaVA: Mixture of Experts for Large Language and Vision Assistant. arXiv preprint arXiv:2312.03325.

[28] Chen, T., et al. (2024). MoR: Mixture of Recursions for Efficient Inference. arXiv preprint arXiv:2403.10524.

[29] OpenAI. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.12712.

[30] Kirillov, A., et al. (2023). Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4015 4026).

[31] Raposo, D., et al. (2024). Mixture-of-Depths. arXiv preprint arXiv:2403.13373.

[32] Teerapittayanon, S., et al. (2017). Dynamic exit: New analysis and training methods. arXiv preprint arXiv:1704.08266.

[33] Wang, Y., et al. (2023). FastViT: Lightweight convolution-transformer hybrid for efficient image recognition. arXiv preprint arXiv:2303.14186.

[34] Rombach, R., et al. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10684-10695).

[35] Min, R., et al. (2024). Mixture-of-Recursions: Learning Dynamic Re cursive Depths for Adaptive Token-Level Computation. arXiv preprint arXiv:2407.10524.

[36] Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (pp. 8748-8763). PMLR.

[37] Yu, F., et al. (2024). Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406.09756.

[38] Li, H., et al. (2024). Taptr: Tracking any point with transformers as detection. arXiv preprint arXiv:2403.13042.

[39] Li, Z., & Snavely, N. (2018). Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2041-2050).

[40] Lin, A., et al. (2023). Relpose++: Recovering 6d poses from sparse-view observations. arXiv preprint arXiv:2305.04926.

[41] Jin, Y., et al. (2021). Image matching across wide baselines: From paper to practice. International Journal of Computer Vision, 129(2), 517-547.

[42] Karaev, N., et al. (2024). Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11831.

[43] Liu, L., et al. (2023). Neural sparse voxel fields for time travel from a single video. In SIGGRAPH Asia 2023 Conference Papers (pp. 1-12).

[44] You, Y., et al. (2024). DUSt3R: Geometric 3D Vision Made Easy. ECCV.

@article{1710307,
      author = {Saksham Gupta, Dr. Ramanjot Kaur},
      title = {SpatialMoR-VGGT: Spatially Adaptive Efficient 3D Scene Reconstruction},
      journal = {Iconic Research And Engineering Journals},
      year = {2025},
      volume = {9},
      number = {2},
      pages = {830-839},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1710307.pdf},
      abstract = {We present SpatialMoR-VGGT, a novel framework that extends the Mixture-of-Recursions (MoR) paradigm to spatial reasoning in 3D vision tasks. While VGGT has demonstrated remarkable capabilities as a feed-forward transformer that directly infers all key 3D attributes of a scene?including camera parameters, point maps, depth maps, and point tracks?it processes all spatial regions with uniform computational depth. Our framework dynamically adjusts the recursion depth for different spatial regions of the scene, allocating more computational resources to complex areas while maintaining efficiency in simpler regions. This adaptation requires addressing fundamental differences between sequential token processing in language and spatially coherent processing in vision. We introduce spatially-aware routing mechanisms and KV caching strategies specifically designed for visual data, along with a balanced training objective that preserves spatial coherence while enabling adaptive computation. Through rigorous experimentation on standard 3D reconstruction benchmarks, we demonstrate that SpatialMoR-VGGT achieves comparable reconstruction quality to standard VGGT with 18-22% reduced computational requirements. This work establishes a foundation for adaptive computation in 3D vision tasks, with potential applications across AR/VR, robotics, and real-time 3D content creation.},
      keywords = {3D Reconstruction, Adaptive Computation, Recursive Transformers, Visual Geometry},
      month = {August},
  }