LiveKit Agents + Ai Twin

Use LiveKit Agents for conversation, then speak replies on an Ai Twin with speakText. Turn off agent audio so only the twin talks. Clone the open-source demo to try the full stack locally.

Overview

If you already run LiveKit Agents for speech-to-text and LLM replies, you can render those replies on an Ai Twin avatar. Keep agent audio off, then call speakText() in the browser so Ai Twin handles TTS and lipsync.

Turn off LiveKit agent TTS and audio output. If both the agent and Ai Twin speak, users hear double audio.

Try the demo

Start from the open-source harness instead of wiring everything from scratch. It includes a token server, LiveKit Agents worker (STT + LLM, no agent TTS), and a browser app that calls speakText() on Ai Twin.

github.com/aitwin-me/aitwin-livekit-demo

Mic — user audio → STT → LLM → Ai Twin speaks
Chat — text → LLM → Ai Twin speaks
Speak text — browser calls Ai Twin directly (no LLM)

What you need

Optional: clone aitwin-livekit-demo to run the full local stack.
A LiveKit Agents setup that produces reply text.
Ai Twin Client-Id and Client-Secret from the dashboard → API Keys.
A twin id from My Twins.
The @streamoji/aitwin player in your web app.

Disable agent audio

Configure the agent so it does not synthesize or publish speech. Ai Twin will speak the text in the browser instead.

python

session = AgentSession(
    stt=...,
    llm=...,
    tts=None,  # Ai Twin speaks in the browser
)

await session.start(
    agent=...,
    room=ctx.room,
    room_options=RoomOptions(
        audio_output=False,
    ),
)

Auth token

Mint a short-lived auth token on your backend with Client-Id and Client-Secret. Never expose Client-Secret in the browser.

python

import requests

response = requests.post(
    "https://us-central1-streamoji-265f4.cloudfunctions.net/getAiTwinAuthToken",
    headers={
        "Content-Type": "application/json",
        "Client-Id": CLIENT_ID,
        "Client-Secret": CLIENT_SECRET,
    },
    json={
        "userId": "user-123",
        "userName": "Jane",
        "expiresIn": 3600,
    },
    timeout=20,
)
auth_token = response.json()["authToken"]

Pass the returned authToken to <AiTwin />. See the AI Twin Player guide for full auth and props details.

Speak agent replies

When your agent has a reply string, deliver it to the browser however you already move data in LiveKit (text stream, data packet, or your own channel). Then call speakText.

tsx

import { AiTwin, type AiTwinHandle } from "@streamoji/aitwin";

const twinRef = useRef<AiTwinHandle | null>(null);

// Mount with the auth token from your backend
<AiTwin
  ref={twinRef}
  id={twinId}
  authToken={authToken}
  onReady={() => {
    // Safe to call speakText after onReady
  }}
/>

// When your LiveKit agent finishes a reply, speak it:
await twinRef.current?.speakText(agentReplyText);

Wait for onReady (or queue text) before the first speakText.
Speak only the final reply text. Partial or duplicate streams will make the twin cut itself off or repeat.