ChiChieh HuangFOUNDER · AI ENGINEER
Writing Voice & Avatar Hills

Building an Interactive AI Assistant: LLMAvatarTalk

Date2024.06.17
Length257 words
Reading~1 min
Figures3 figures
ChiChieh HuangFounder · AI engineer

Translated from the Chinese original · Read the original

LocatorVoice & Avatar Hills
2 wks
Overview

Introduction

I was recently invited to a competition run by NVIDIA and LangChain, where I built an interactive AI avatar assistant: LLMAvatarTalk: An Interactive AI Assistant. The project brings together advanced AI technologies so you can chat smoothly with the AI assistant you create and see lifelike facial expressions as it talks.

The project is on GitHub, with detailed step-by-step instructions for every stage. If you’re interested in the project or have suggestions, feel free to take a look, open an issue or send a PR. And if it helps you, please give it a star. Thank you!

Since this was an NVIDIA and LangChain competition, I built everything with NVIDIA and LangChain technologies, but you can integrate different technologies to suit your needs.

Features

  1. Speech recognition: NVIDIA RIVA ASR converts the user’s speech into text in real time.
  2. Language processing: NVIDIA NIM APIs and advanced LLMs (like llama3-70b-instruct) provide deep semantic understanding and generate responses.
  3. Text-to-speech: NVIDIA RIVA TTS turns the generated text responses into natural speech.
  4. Facial animation: Audio2Face generates realistic facial expressions and animation from the speech output.
  5. Unreal Engine integration: Unreal Engine’s MetaHuman connects to Audio2Face in real time, making the virtual character more expressive.
  6. LangChain: simplifies integrating NVIDIA RIVA and the NVIDIA NIM APIs, giving AI development a seamless, efficient workflow.

Architecture

Support

If this article helped you, or you’d like to encourage me to keep writing, you can clap for it or buy me a coffee through the link below. Thank you for your support!

#llm#nvidia#avatar#tutorial#ai-avatars

End of the trail

257 words, and you made it to the end.

RSS feed Back to the map
Newsletter

Get the next essay by email.

One email when a new essay goes up, nothing else. Unsubscribe in one click.

← Previous · 2024.06.05RAG Evaluation Frameworks Compared: TruLens, RGAR and RAGAs Next · 2024.07.19 →Knowledge Graph + RAG: Microsoft GraphRAG, Implemented and Visualized
Nearby cairns · the closest essays on the map
RAG Basin32 days apart Knowledge Graph + RAG: Microsoft GraphRAG, Implemented and Visualized What Microsoft GraphRAG is and why combining knowledge graphs with RAG improves answers, plus a step-by-step implementation, two ways to visualize the graph, and what it costs. RAG Basin12 days apart RAG Evaluation Frameworks Compared: TruLens, RGAR and RAGAs As RAG systems grow more complex, they need comprehensive evaluation. The challenges of evaluating retrieval and generation, the RAG triad, how RGAR and RAGAs bring in ground truth, and where RAG evaluation is heading. RAG Basin102 days apart Seven RAG Challenges and How to Solve Them Even with RAG, retrievers return wrong or irrelevant data and LLMs answer from bad information. The seven common failure points of RAG systems and practical fixes for each in LangChain and LlamaIndex.
ChiChieh Huang
Surveyor

ChiChieh Huang

I build generative AI products and write about them, first in Chinese. Lately I’ve been researching agent memory and testing the ideas in Cairn.

About me Invite me to speak