Kwami

Kwami is a browser TypeScript library for 3D AI companions. Drop a canvas on the page, give the companion a soul, and it can speak, listen, remember, and use tools — with a WebGL avatar that reacts in real time.

Avatar · voice · memory · tools · skills

Site: kwami.io · Package: npmjs.com/package/kwami · Support: SUPPORT.md

---

Table of contents

• Overview
• Features
• Architecture
• Quick start
• Usage
• Getting started
• Scripts
• Project structure
• Documentation
• Support
• Contributing
• Code of conduct
• Security
• Releases
• License

---

Overview

Each Kwami instance is an independent agent:

Piece	What it does
Avatar	WebGL renderer on an HTMLCanvasElement (blob, black hole, particles, iris)
Agent	Voice pipeline over LiveKit — STT, LLM, TTS, or realtime
Soul	Name, personality, emotional traits, and system prompt
Memory	Long-term recall (Zep adapter)
Tools	External capabilities via MCP
Skills	Native in-process behaviors

Several instances can share a page. Configuration can change after connect() without tearing the room down.

The avatar is local: new Kwami(canvas) draws immediately. Voice, memory, and tool dispatch need a LiveKit-backed agent and a token endpoint you control — the library does not host those for you.

import { Kwami } from 'kwami';

const luna = new Kwami(canvas, {
  soul: { name: 'Luna', personality: 'friendly and creative' },
  agent: { voice: { llm: { model: 'gpt-4o' } } },
});

await luna.connect('user-123', {
  onStateChange: (state) => console.log(state),
  onUserTranscript: (text) => console.log('user:', text),
  onAgentResponse: (text) => console.log('luna:', text),
});

---

Features

• 3D avatars — blob-xyz, black-hole, particles-face, and eye-iris, with skins, presets, and audio-reactive motion
• Voice pipeline — mix-and-match STT / LLM / TTS, or a realtime provider, dispatched through LiveKit
• Provider catalog — Deepgram, OpenAI, Anthropic, Gemini, ElevenLabs, Groq, Cartesia, AssemblyAI, and more
• Soul presets — ready-made personalities (Kaya, Nexus, Phoenix, Haven, …) plus live trait updates
• Memory — optional long-term context so the companion remembers the user across sessions
• MCP tools — connect Model Context Protocol servers and expose their tools to the agent
• Skills — register native behaviors that run in the page
• Typed ESM — kwami ships as ESM with generated .d.ts; three is a peer dependency

---

Architecture

                         ┌─────────────────────────────────────┐
                         │              Kwami                  │
                         │  id · state · connect / disconnect  │
                         └───────────────┬─────────────────────┘
          ┌──────────┬───────────┬───────┼────────┬──────────┬──────────┐
          ▼          ▼           ▼       ▼        ▼          ▼
       Avatar      Agent        Soul   Memory   Tools      Skills
      (WebGL)    (LiveKit)   (persona)  (Zep)   (MCP)     (native)
          │          │
          │          └── VoiceSession  STT → LLM → TTS
          └── Scene · skins · audio bands · renderer

Concept	Detail
Construct	new Kwami(canvas, config?) creates avatar, agent, soul, memory, tools, and skills
Connect	connect(userId) initializes memory, builds the system prompt, and dispatches a LiveKit agent
Speak / listen	User speech and agent replies update avatar state: idle → listening → thinking → speaking
Update live	Soul, voice, tools, and skills can change while the room is up
Dispose	Call dispose() so the WebGL context, LiveKit room, and registry entry are released

---

Quick start

pnpm add kwami three

three is a peer dependency. Install it beside kwami; do not rely on a transitive copy.

A WebGL2-capable browser is required. The avatar mounts without a backend; connect() needs a token endpoint and an agent identity your server mints — see Connecting.

import { Kwami } from 'kwami';

const canvas = document.querySelector('canvas')!;
const kwami = new Kwami(canvas, {
  soul: { name: 'Kaya', personality: 'A warm, friendly AI companion' },
});

await kwami.connect('user-123');
kwami.sendMessage('Hello — who are you?');

// Later
await kwami.disconnect();
kwami.dispose();

Channels:

pnpm add kwami          # latest stable (main)
pnpm add kwami@rc       # release candidate (stg)
pnpm add kwami@dev      # prerelease (dev)

---

Usage

Soul presets

import { Kwami, getSoulPresetById, toSoulConfig } from 'kwami';

const kaya = getSoulPresetById('friendly');
const kwami = new Kwami(canvas, {
  soul: kaya ? toSoulConfig(kaya) : { name: 'Kaya' },
});

Presets include Kaya, Nexus, Phoenix, Haven, Sage, and others (soulPresets / getSoulPresetsByCategory). Traits can change after connect:

kwami.updateSoul({ emotionalTone: 'enthusiastic' });

Avatar

const kwami = new Kwami(canvas, {
  avatar: { renderer: 'blob-xyz' }, // or 'black-hole' | 'particles-face' | 'eye-iris'
});

Connecting

Pin the agent participant and keep provider keys on the server. The adapter treats that identity as authoritative for audio, transcripts, and tool_call messages.

const kwami = new Kwami(canvas, {
  soul: { name: 'Kaya' },
  agent: {
    livekit: {
      tokenEndpoint: '/api/livekit/token',
      agentIdentity: 'agent-7f3a', // or agentIdentityPrefix: 'kwami-agent:'
    },
  },
});

await kwami.connect('user-123', {
  onStateChange: (state) => {
    /* idle | listening | thinking | speaking */
  },
  onUserTranscript: (text) => {},
  onAgentResponse: (text) => {},
  onError: (error) => console.error(error),
});

Do not put LLM / TTS / STT API keys in the browser. Details: SECURITY.md.

Tools

kwami.registerTool({
  name: 'get_weather',
  description: 'Current weather for a city',
  parameters: {
    type: 'object',
    properties: { city: { type: 'string' } },
    required: ['city'],
  },
  handler: async ({ city }) => fetchWeather(String(city)),
});

MCP servers can be passed in tools.mcp; connect() waits for them before dispatching the agent.

Skills

kwami.skills.register({
  name: 'wave',
  description: 'Play a greeting on the avatar',
  trigger: 'action',
  execute: async ({ kwami: instance }) => {
    // drive the local avatar / UI
    return { success: true, message: 'waved' };
  },
});

Lifecycle

Always dispose() when the canvas leaves the page. Instances stay in a process-wide registry until you do; an unmounted component leaks the WebGL context and the LiveKit room.

await kwami.disconnect();
kwami.dispose();

---

Getting started

This section is for working on the library. Consumers only need Quick start.

Prerequisites

Tool	Requirement
Node.js	Version in .nvmrc (>= 22.14) — nvm use
pnpm	>= 10 — corepack enable picks up packageManager from package.json

pnpm-lock.yaml is the lockfile of record. CI installs with --frozen-lockfile; an npm install here produces a lockfile CI will reject.

Install

git clone https://github.com/kwami-labs/kwami.git
cd kwami
nvm use
corepack enable
pnpm install    # also installs husky hooks

Build & verify

pnpm build
pnpm lint && pnpm typecheck && pnpm test:run

---

Scripts

# Build
pnpm build              # bundle + type declarations → dist/
pnpm dev                # rebuild on change
pnpm clean              # remove dist/, coverage/, and test output

# Quality
pnpm lint
pnpm lint:fix
pnpm format
pnpm format:check
pnpm typecheck
pnpm audit:ci

# Tests — see docs/testing.md
pnpm test               # unit
pnpm test:watch
pnpm test:coverage      # unit + coverage ratchet (CI)
pnpm test:integration
pnpm test:e2e           # built bundle in a real browser (WebGL)
pnpm test:all

# Releases — see docs/releases.md
pnpm release:dry-run    # print next version; change nothing

---

Project structure

kwami/
├── src/                  # Library source
│   ├── agent/            # Voice pipeline, LiveKit adapter, VoiceSession
│   ├── avatar/           # WebGL renderers, scene, skins, audio
│   ├── memory/           # Memory adapters
│   ├── soul/             # Personality and presets
│   ├── skills/           # Native behaviors
│   ├── tools/            # Tool registry (MCP)
│   ├── types/            # TypeScript definitions
│   └── Kwami.ts          # Public entry surface
├── tests/
│   ├── unit/
│   ├── integration/
│   └── e2e/
├── scripts/
│   ├── ci/               # Pipeline gates
│   └── release/          # Baseline tag, back-merge
├── docs/                 # Deep-dive guides
├── .github/              # Workflows, issue/PR templates, CODEOWNERS
├── CHANGELOG.md          # Generated by semantic-release
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── SECURITY.md
├── SUPPORT.md
├── LICENSE               # Apache-2.0
└── package.json

---

Documentation

Document	Description
SUPPORT.md	How to get help, and what belongs in an issue
CONTRIBUTING.md	Setup, branches, commits, PR checklist
CODE_OF_CONDUCT.md	Community standards (Contributor Covenant 2.1)
SECURITY.md	Supported versions & vulnerability reporting
CHANGELOG.md	Released changes (generated)
docs/ci-cd.md	CI pipeline & promotion gate
docs/releases.md	Channels, version bumps, troubleshooting
docs/testing.md	Unit / integration / e2e layers
LICENSE	Apache License 2.0 (full text)

---

Support

Something broken, unclear, or missing? SUPPORT.md is the map — questions, bugs, features, and what we cannot debug from an issue (your LiveKit project, provider keys, page wrappers).

• Questions: open a question
• Bugs: open a bug report
• Ideas: open a feature request

Please search existing issues first. Do not paste tokens or API keys.

---

Contributing

Contributions are welcome. Please read CONTRIBUTING.md and the Code of Conduct before opening a PR.

Branch promotion (enforced in CI):

feature/* ──► dev ──► stg ──► main

• Branch off dev; open PRs against dev
• Use Conventional Commits — versions, tags, changelog, and npm publishes are derived from them
• Feature PRs are squash-merged; the PR title becomes the release subject
• Before push: pnpm lint && pnpm typecheck && pnpm test:unit && pnpm test:integration

Issue tracker: github.com/kwami-labs/kwami/issues

---

Code of conduct

This project follows the Contributor Covenant. By participating you agree to the CODE_OF_CONDUCT.md.

Reports: [email protected]. Do not post accusations in a public issue.

---

Security

Do not open public issues for vulnerabilities.

Report privately via GitHub Security Advisories or see SECURITY.md for supported versions, scope, and response expectations.

---

Releases

Releases are automated with semantic-release. Do not hand-edit CHANGELOG.md or the version field.

Branch	npm tag	Channel
main	latest	Stable
stg	rc	Release candidate
dev	dev	Prerelease

Details: docs/releases.md · History: CHANGELOG.md

---

License

Copyright © 2025–2026 Alex Colls Outumuro

Licensed under the Apache License, Version 2.0. See the LICENSE file for the full text.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.