Building a Voice Agent with LiveKit and Memobase

Full Code

This tutorial demonstrates how to build a voice agent with long-term memory using Memobase and LiveKit. This combination is ideal for applications like AI companions, customer support bots, and more.

Setup

Get API Keys:
- Memobase: Sign up at Memobase or run a local server.
- LiveKit: Get your LIVEKIT_URL, API_KEY, and API_SECRET from the LiveKit Cloud Console.
- Deepgram: Get your DEEPGRAM_API_KEY from the Deepgram Console.

Environment Variables: Set up your environment variables.

OPENAI_API_KEY="your_openai_api_key"
DEEPGRAM_API_KEY="your_deepgram_api_key"
LIVEKIT_URL="your_livekit_url"
LIVEKIT_API_KEY="your_livekit_api_key"
LIVEKIT_API_SECRET="your_livekit_api_secret"
MEMOBASE_URL="https://api.memobase.io"
MEMOBASE_API_KEY="your_memobase_api_key"

Install Dependencies:
pip install -r requirements.txt

Code Breakdown

The full code is available here. We will be using the LiveKit Agents SDK v1.0. The core of the integration involves subclassing the livekit.agents.Agent class and overriding the llm_node method to inject memory context from Memobase.

Agent Initialization

First, we initialize the Memobase client and define our custom agent class.

import os
from dotenv import load_dotenv
from livekit.agents import Agent, llm
from memobase import AsyncMemoBaseClient, ChatBlob
from memobase.utils import string_to_uuid

load_dotenv()
mb_client = AsyncMemoBaseClient(
    api_key=os.getenv("MEMOBASE_API_KEY"), project_url=os.getenv("MEMOBASE_URL")
)

class RAGEnrichedAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a warm-hearted partner who remembers past interactions.",
        )
        self.user_name = os.getenv("MEMOBASE_USER_NAME", "test_user")
        self.chat_log_index = 1

Injecting Memory

Next, we override the llm_node method. This method is called just before the chat history is sent to the LLM. Here, we will retrieve the user’s memory from Memobase and add it to the system prompt.

async def llm_node(
    self,
    chat_ctx: llm.ChatContext,
    # ... other params
) -> AsyncIterable[llm.ChatChunk]:
    # Ensure Memobase is reachable
    assert await mb_client.ping(), "Memobase is not reachable"
    
    # Get or create the user in Memobase
    user = await mb_client.get_or_create_user(string_to_uuid(self.user_name))

    # Insert new messages into memory
    if len(chat_ctx.items) > self.chat_log_index:
        new_messages = chat_ctx.items[self.chat_log_index : len(chat_ctx.items) - 1]
        if len(new_messages):
            blob = ChatBlob(
                messages=[
                    {"role": m.role, "content": m.content[0]}
                    for m in new_messages
                    if m.role in ["user", "assistant"]
                ]
            )
            await user.insert(blob)
            await user.flush()
            self.chat_log_index = len(chat_ctx.items) - 1

    # Retrieve memory context and add to chat history
    rag_context: str = await user.context(max_token_size=500)
    chat_ctx.add_message(content=rag_context, role="system")
    
    # Call the default LLM node
    return Agent.default.llm_node(self, chat_ctx, tools, model_settings)

The rag_context string will contain the user’s profile and recent events, formatted and ready to be used by the LLM.

Running the Agent

Finally, we set up the entry point to run the agent.

async def entrypoint(ctx: JobContext):
    # ... (connect to room, set up STT, TTS, etc.)
    await session.start(
        agent=RAGEnrichedAgent(),
        # ...
    )

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

To run the code, first download the necessary assets:

python livekit_example.py download-files

Then, start the agent:

python livekit_example.py console

You can now have a conversation with the voice agent. It will remember information you provide across different sessions.

​Setup

​Code Breakdown

​Agent Initialization

​Injecting Memory

​Running the Agent

Setup

Code Breakdown

Agent Initialization

Injecting Memory

Running the Agent