> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spatialreal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LiveKit Client

> Real-time voice communication with avatars using LiveKit

```mermaid theme={null}
flowchart LR
    A["🎤 Microphone"] --> B["AvatarPlayer"]
    B <-->|WebRTC| C["LiveKit Server"]
    B --> D["AvatarKit SDK"]
    D --> E["🖥️ Avatar Rendering"]
```

<Note>
  **Web Only:** RTC Mode is currently available for Web applications only.
</Note>

<Warning>
  **Do not call `initializeAudioContext()`** — it is not needed in RTC Mode. Avatar audio is delivered as a native WebRTC audio track by the LiveKit client SDK, not through the AvatarKit SDK's internal audio player. The `AvatarPlayer` adapter only feeds **animation data** to the SDK for rendering; audio playback is handled entirely by LiveKit's WebRTC stack.
</Warning>

## Installation

<Warning>
  **Critical:** We are only compatable with `livekit-client` under version `2.17`, since `2.17` introduced single PC by default. We are working on adding support for `2.17`+ in a future release, but for now please ensure you install version `2.16.1` to avoid compatibility issues.
</Warning>

<Tabs>
  <Tab title="pnpm">
    ```bash theme={null}
    pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>

  <Tab title="npm">
    ```bash theme={null}
    npm install @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>

  <Tab title="yarn">
    ```bash theme={null}
    yarn add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>
</Tabs>

You also need to configure your build tool for WASM files — see [Build Tool Configuration](/guide/sdk-mode-web#build-tool-configuration).

## Authentication

| Credential        | How to Obtain                                                 | Notes                             |
| ----------------- | ------------------------------------------------------------- | --------------------------------- |
| **App ID**        | [SpatialReal Studio](https://app.spatialreal.ai) → Create App | For SDK initialization            |
| **Session Token** | Your backend → AvatarKit Server                               | For avatar loading (max 24 hours) |
| **LiveKit Token** | Your backend → LiveKit Server                                 | For RTC room connection           |

## Quick Start

For a complete client + server implementation, see [AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-cascade-voice-agent).

<Steps>
  <Step title="Initialize SDK in Host Mode">
    ```typescript theme={null}
    import { AvatarSDK, AvatarManager, AvatarView, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

    await AvatarSDK.initialize('your-app-id', {
      environment: Environment.intl,
      drivingServiceMode: DrivingServiceMode.host,  // MUST be host for RTC
    })

    AvatarSDK.setSessionToken('your-session-token')
    ```
  </Step>

  <Step title="Load Avatar & Create View">
    ```typescript theme={null}
    const avatar = await AvatarManager.shared.load('avatar-id')
    const container = document.getElementById('avatar-container')!
    const avatarView = new AvatarView(avatar, container)
    ```
  </Step>

  <Step title="Create Player with LiveKit Provider">
    ```typescript theme={null}
    import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'

    const provider = new LiveKitProvider()
    const player = new AvatarPlayer(provider, avatarView, {
      logLevel: 'warning',
    })
    ```
  </Step>

  <Step title="Connect to LiveKit Server">
    ```typescript theme={null}
    await player.connect({
      url: 'wss://your-livekit-server.com',
      token: 'your-livekit-token',
      roomName: 'room-name',
    })
    ```
  </Step>

  <Step title="Start Voice Interaction">
    ```typescript theme={null}
    // Start microphone publishing
    await player.startPublishing()

    // Stop microphone
    await player.stopPublishing()

    // Disconnect when done
    await player.disconnect()
    ```
  </Step>
</Steps>

## AvatarPlayer API

### Constructor

```typescript theme={null}
new AvatarPlayer(provider: LiveKitProvider, avatarView: AvatarView, options?: AvatarPlayerOptions)
```

### AvatarPlayerOptions

```typescript theme={null}
interface AvatarPlayerOptions {
  /** Start speaking transition frames, default 5 (~200ms at 25fps) */
  transitionStartFrameCount?: number

  /** End speaking transition frames, default 40 (~1600ms at 25fps) */
  transitionEndFrameCount?: number

  /** Log level: 'info' | 'warning' | 'error' | 'none', default 'warning' */
  logLevel?: LogLevel
}
```

### Connection

```typescript theme={null}
// Connect to LiveKit server
await player.connect(config: LiveKitConnectionConfig)

// Disconnect and clean up
await player.disconnect()

// Reconnect using last config (useful after stalls)
await player.reconnect()

// Check connection status
player.isConnected  // boolean
player.getConnectionState()  // ConnectionState
```

#### LiveKitConnectionConfig

```typescript theme={null}
interface LiveKitConnectionConfig {
  url: string       // LiveKit server URL (wss://...)
  token: string     // Auth token from your backend
  roomName: string  // Room name
}
```

### Microphone Control

```typescript theme={null}
// Start microphone (requests permission automatically)
await player.startPublishing()

// Stop microphone
await player.stopPublishing()
```

### Custom Audio Publishing

For non-microphone audio sources like audio elements or Web Audio API.

```typescript theme={null}
// Publish a custom audio track
await player.publishAudio(track: MediaStreamTrack)

// Stop custom audio
await player.unpublishAudio()
```

| Audio Source       | How to Obtain Track                                                      |
| ------------------ | ------------------------------------------------------------------------ |
| `<audio>` element  | `audioElement.captureStream().getAudioTracks()[0]`                       |
| Screen share audio | `getDisplayMedia({ audio: true })`                                       |
| Web Audio API      | `audioContext.createMediaStreamDestination().stream.getAudioTracks()[0]` |

<Note>
  `unpublishAudio()` does **not** stop the track. You are responsible for calling `track.stop()` to release the resource.
</Note>

### Native Client Access

Access the underlying LiveKit `Room` instance for advanced features.

```typescript theme={null}
import { LiveKitRoom } from '@spatialwalk/avatarkit-rtc'

// Via provider (recommended — full type safety)
const room = provider.getNativeClient()  // LiveKitRoom | null

// Via player (requires type assertion)
const room = player.getNativeClient() as LiveKitRoom | null

console.log('Remote participants:', room?.remoteParticipants.size)
```

### ConnectionState

```typescript theme={null}
type ConnectionState = 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'
```

## Events

```typescript theme={null}
player.on(event: string, handler: Function)
player.off(event: string, handler: Function)
```

| Event                        | Callback                   | Description                            |
| ---------------------------- | -------------------------- | -------------------------------------- |
| `'connected'`                | `()`                       | Connected to RTC server                |
| `'disconnected'`             | `()`                       | Disconnected from RTC server           |
| `'error'`                    | `(error: Error)`           | An error occurred                      |
| `'connection-state-changed'` | `(state: ConnectionState)` | Connection state changed               |
| `'stalled'`                  | `()`                       | Data stream stalled (no frames for 3s) |

<Note>
  When a `stalled` event fires, the avatar automatically transitions to idle animation. Consider calling `player.reconnect()` to recover.
</Note>

**Stall recovery example:**

```typescript theme={null}
player.on('stalled', async () => {
  console.log('Stream stalled, reconnecting...')
  try {
    await player.reconnect()
  } catch (error) {
    console.error('Reconnection failed:', error)
  }
})
```

## Complete Example

```typescript theme={null}
import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

async function init() {
  // Initialize SDK in host mode
  await AvatarSDK.initialize('your-app-id', {
    environment: Environment.intl,
    drivingServiceMode: DrivingServiceMode.host,
  })
  AvatarSDK.setSessionToken('your-session-token')

  // Load avatar and create view
  const avatar = await AvatarManager.shared.load('character-id')
  const container = document.getElementById('avatar-container')!
  const avatarView = new AvatarView(avatar, container)

  // Create player
  const provider = new LiveKitProvider()
  const player = new AvatarPlayer(provider, avatarView, {
    logLevel: 'info',
    transitionStartFrameCount: 5,
    transitionEndFrameCount: 40,
  })

  // Listen to events
  player.on('connected', () => console.log('Connected!'))
  player.on('disconnected', () => console.log('Disconnected!'))
  player.on('error', (err) => console.error('Error:', err))
  player.on('stalled', async () => {
    console.log('Stream stalled, reconnecting...')
    await player.reconnect()
  })

  // Connect to LiveKit server
  await player.connect({
    url: 'wss://your-livekit-server.com',
    token: 'your-livekit-token',
    roomName: 'my-room',
  })

  // Start microphone
  await player.startPublishing()
}
```

## Browser Compatibility

| Browser | Minimum Version | Notes                           |
| ------- | --------------- | ------------------------------- |
| Chrome  | 94+             | VP8 + RTCRtpScriptTransform     |
| Firefox | 117+            | RTCRtpScriptTransform required  |
| Safari  | 15.4+           | RTCRtpScriptTransform supported |
| Edge    | 94+             | Same as Chrome                  |