Home / Current Issue / Paper 1715544
AI Voice Call Assistant for Automated Call Management
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence
Abstract
This paper presents the design and implementation of a production-ready AI Voice Call Assistant platform for automated call management. The proposed system integrates real-time WebRTC audio transport via LiveKit, streaming Speech-to-Text (STT) using Deepgram Nova-2, a large language model reasoning layer powered by OpenAI GPT-4o through LangChain, and streaming Text-to-Speech (TTS) via ElevenLabs Multilingual v2. The backend is built on FastAPI with WebSocket-based pipeline orchestration, while the frontend dashboard is developed using Next.js 14 and Tailwind CSS. Session state is managed with Redis and persistent data is stored in PostgreSQL. All services are containerized using Docker and served through an NGINX reverse proxy. The system achieves low-latency end-to-end voice interaction, interrupt handling (barge-in), multi-turn context memory, function calling via LangChain tools, and agent configuration via a web dashboard. Experimental results demonstrate end-to-end response latency of 1.2-1.8 seconds, competitive with commercial voice AI platforms such as Vapi.ai and Retell AI, while remaining fully open-source and self-hostable.
References
[1] D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. Pearson Education, 2021.
[2] A. Graves, A. Mohamed, and G. Hinton, "Deep recurrent neural networks for acoustic modelling," in Proc. IEEE ICASSP, Vancouver, BC, Canada, 2013, pp. 6645-6649.
[3] A. van den Oord et al., "WaveNet: A generative model for raw audio," arXiv preprint arXiv:1609.03499, 2016.
[4] OpenAI, "GPT-4 Technical Report," arXiv preprint arXiv:2303.08774, 2023.
[5] C. Jennings, H. Bostrom, and J. P. Bruaroey, "WebRTC: Real-time communication for the open web platform," IEEE Communications Magazine, vol. 51, no. 8, pp. 20-29, Aug. 2013.
[6] Deepgram, "Nova-2 Streaming ASR Model," Deepgram Technical Documentation, 2024. [Online]. Available: https://deepgram.com/learn/nova-2-speech-to-text
[7] ElevenLabs, "Multilingual v2 Text-to-Speech API Documentation," 2024. [Online]. Available: https://elevenlabs.io/docs/api-reference
[8] LiveKit, "LiveKit Open Source Real-time Infrastructure," 2024. [Online]. Available: https://livekit.io
[9] H. Chase et al., "LangChain: Building applications with large language models through composability," GitHub, 2023. [Online]. Available: https://github.com/langchain-ai/langchain
[10] S. Ramachandran, R. K. Nair, and A. Arjunan, "Low-Latency Streaming Text-to-Speech for Voice Assistants," in Proc. Interspeech, Incheon, Korea, 2022, pp. 2541-2545.
How to cite this paper
@article{1715544,
author = {Asiq Sikkander T N, Revathy D},
title = {AI Voice Call Assistant for Automated Call Management},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {9},
number = {9},
pages = {2544-2552},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1715544.pdf},
abstract = {This paper presents the design and implementation of a production-ready AI Voice Call Assistant platform for automated call management. The proposed system integrates real-time WebRTC audio transport via LiveKit, streaming Speech-to-Text (STT) using Deepgram Nova-2, a large language model reasoning layer powered by OpenAI GPT-4o through LangChain, and streaming Text-to-Speech (TTS) via ElevenLabs Multilingual v2. The backend is built on FastAPI with WebSocket-based pipeline orchestration, while the frontend dashboard is developed using Next.js 14 and Tailwind CSS. Session state is managed with Redis and persistent data is stored in PostgreSQL. All services are containerized using Docker and served through an NGINX reverse proxy. The system achieves low-latency end-to-end voice interaction, interrupt handling (barge-in), multi-turn context memory, function calling via LangChain tools, and agent configuration via a web dashboard. Experimental results demonstrate end-to-end response latency of 1.2-1.8 seconds, competitive with commercial voice AI platforms such as Vapi.ai and Retell AI, while remaining fully open-source and self-hostable.},
month = {March},
doi = {https://doi.org/10.64388/IREV9I9-1715544}
}