Semantic Change Detection in Evolving Documents Using Contextual Embeddings and Transformer Models

References

[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019, pp. 4171–4186.

[2] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. EMNLP-IJCNLP, Hong Kong, 2019, pp. 3982–3992.

[3] H. Touvron et al., “LLaMA: Open and efficient foundation language models,” arXiv:2302.13971, 2023.

[4] G. Salton and C. Buckley, “Term-weighting approaches in automatic text retrieval,” Inf. Process. Manage., vol. 24, no. 5, pp. 513–523, 1988.

[5] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proc. NeurIPS, Lake Tahoe, NV, USA, 2013, pp. 3111–3119.

[6] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Trans. Assoc. Comput. Linguist., vol. 5, pp. 135–146, 2017.

[7] M. E. Peters et al., “Deep contextualized word representations,” in Proc. NAACL-HLT, New Orleans, LA, USA, 2018, pp. 2227–2237.

[8] M. Martinc, P. K. Novak, and S. Pollak, “Leveraging contextual embeddings for detecting diachronic semantic shift,” in Proc. LREC, Marseille, France, 2020, pp. 4811–4819.

[9] A. Kutuzov and M. Giulianelli, “UiO-UvA at SemEval-2020 Task 1: Contextualised embeddings for lexical semantic change detection,” in Proc. SemEval, 2020, pp. 126–134.

[10] D. Schlechtweg et al., “SemEval-2020 Task 1: Unsupervised lexical semantic change detection,” in Proc. SemEval, 2020, pp. 1–23.

[11] Hugging Face, “sentence-transformers/all-MiniLM-L6-v2,” Model Card, 2021. [Online]. Available: https://huggingface.co/sentence-transformers/ all-MiniLM-L6-v2

[12] MetricGate, “Semantic Similarity Threshold Calibration Guide,” 2024. [Online]. Available: https://metricgate.com/docs/semantic-similarity/

[13] M. Douze et al., “The Faiss library,” arXiv:2401.08281, 2024.

[14] A. Jaiswal and E. Milios, “Breaking the token barrier: Chunking and convolution for efficient long text classification with BERT,” arXiv:2310.20558, 2023.

[15] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv:2004.05150, 2020.

[16] Anonymous, “TempoFormer: A transformer for temporally-aware repre-sentations in change detection,” arXiv:2408.15689, 2024.

[17] L. Rosin and R. Radinsky, “Temporal attention for language models,” arXiv:2202.02093, 2022.

[18] F. Periti, H. Dubossarsky, and N. Tahmasebi, “(Chat)GPT v BERT: Dawn of Justice for semantic change detection,” in Findings of EACL, 2024.

[19] J. M. C. de Sa, C. Pruski, and M. Da Silveira, “Semantic change characterization with LLMs using rhetorics,” arXiv:2407.16624, 2024.

[20] D. Huwiler, K. Stockinger, and J. Fuerst, “VersionRAG: Version-aware retrieval-augmented generation for evolving documents,” arXiv:2510.08109, 2025.

[21] Anonymous, “OwlerLite: Scope- and freshness-aware web retrieval for LLM assistants,” arXiv:2601.17824, 2026.

[22] J. Vamvas et al., “SwissGov-RSD: A human-annotated, cross-lingual benchmark for token-level recognition of semantic differences between related documents,” arXiv:2512.07538, 2024.

@article{1718361,
      author = {Nikita Bachute, Prof. Mrs. Ashwini Garkhedkar},
      title = {Semantic Change Detection in Evolving Documents Using Contextual Embeddings and Transformer Models},
      journal = {Iconic Research And Engineering Journals},
      year = {2026},
      volume = {9},
      number = {11},
      pages = {4659-4666},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1718361.pdf},
      abstract = {Automatically finding meaningful differences between two versions of a document is a hard problem that current tools handle poorly. Tools that compare documents word-by-word or line-by-line cannot tell whether a rewritten sentence still means the same thing, nor can they spot a subtle but important change hidden in an otherwise unchanged paragraph. This paper presents the Semantic Document Evolution Tracker (SDET), a four-step system that breaks documents into paragraphs, converts each paragraph into a compact numerical fingerprint of its meaning using the all-MiniLM-L6-v2 Sentence-BERT model, finds the closest-matching paragraph between the two versions by comparing those fingerprints, and then passes the potentially changed pairs to the LLaMA 3.3 70B language model to decide whether each one is Added, Modified, or Deleted and how serious the change is. Tests on 120 hand-labelled document pairs from corporate policy documents and software requirement specifications showed a correct classification rate of 91% and a correct severity rating of 87%, beating a standard keyword-matching approach by 19 percentage points. A step-by-step removal test confirmed that every part of the system adds measurable value. The full system is deployed as a web service using FastAPI, SQLite, and Streamlit.},
      keywords = {Semantic Change Detection, Contextual Embed-Dings, Sentence-BERT, FAISS, Llama, Document Versioning, Transformer Models, Cosine Similarity},
      month = {May},
      doi = {https://doi.org/10.64388/IREV9I11-1718361}
  }