# Server API Reference
Source: https://docs.spatialreal.ai/api-reference/api-reference

Server-side API Reference

## Obtain a session token

Call the server endpoint `POST /v1/console/session-tokens`.

### Console API host

In the examples below, replace `<console-api-host>` with the host for your region:

| Region         | `<console-api-host>`                     |
| -------------- | ---------------------------------------- |
| `ap-northeast` | `console.ap-northeast.spatialwalk.cloud` |
| `us-west`      | `console.us-west.spatialwalk.cloud`      |

### Request parameters

* **Body**
  * **expireAt**: int, expiration timestamp in seconds. Range: now \< expireAt \< now + 24 hours
  * **modelVersion**: string, reserved field; omit
* **Header**
  * **X-Api-Key**: your API key

### Request example

```bash theme={null}
curl --location --request POST 'https://<console-api-host>/v1/console/session-tokens' \  # replace with your console API domain
--header 'X-Api-Key: <your-api-key>' \                                                  # replace with your API key
--header 'Content-Type: application/json' \
--data-raw '{
    "expireAt": 1754824283,
    "modelVersion": ""
}'                                                                                      # set expireAt to the desired expiration time
```

### Response example

```json theme={null}
{
  "sessionToken": "..."
}
```


# Authentication
Source: https://docs.spatialreal.ai/api-reference/auth

Authentication Flow

## Before you start

Before you begin, make sure you have your app-id and api-key.
AvatarKit authentication requires a server-side component, so you need to implement an authentication endpoint on your own server.

## Connection flow

1. The client sends an authentication request to your business server.
2. The business server sends a request to SpatialReal's server to generate a session token, including information such as expiration time in the request body, and providing the api-key.
3. The SpatialReal server returns the session token to your business server.
4. The business server returns the session token to the client.
5. The client initializes AvatarKit with the session token.
6. Inside AvatarKit, a connection request is created to the SpatialReal server using the session token.

```mermaid theme={null}
sequenceDiagram
    participant Client as Client
    participant BizServer as Business Server
    participant SW as SpatialReal Server
    participant Kit as AvatarKit

    Client->>BizServer: Authentication request
    BizServer->>SW: Generate session token (with api-key, expiration)
    SW-->>BizServer: Return session token
    BizServer-->>Client: Return session token
    Client->>Kit: Initialize with session token
    Kit->>SW: Create connection request (session token)
    SW-->>Kit: Connection response
    Note over SW,Kit: After token expiration, new connections are rejected<br>Existing connections are not affected
```

## Token expiration

If you attempt to establish a new connection after the token's configured expiration time, it will be rejected.
Existing established connections are not affected.

## Notes

* Avoid leaking your api-key; ensure it is only used on the server.
* The session token is designed to be single-use. Ensure a new token is used for each connection.

> For detailed authentication API docs, see the [API Reference](/api-reference/api-reference)


# Regions
Source: https://docs.spatialreal.ai/api-reference/regions

Available regions, how to configure and how to fallback to other region for higher availability.

## Globally distributed service

Our avatar service runs in multiple regions. You can choose the region that’s closest to your infrastructure to minimize latency.

## Available regions

Use these region names when selecting a deployment:

* **ap-northeast**
* **us-west**

## Console API hosts

When a doc example uses `<console-api-host>`, substitute the host for your region:

| Region         | `<console-api-host>`                     |
| -------------- | ---------------------------------------- |
| `ap-northeast` | `console.ap-northeast.spatialwalk.cloud` |
| `us-west`      | `console.us-west.spatialwalk.cloud`      |

## Console and Ingress endpoint URLs

When a doc example uses console endpoint url or ingress endpoint url, substitute the URL for your region:

| Region         | Console Endpoint URL                                        | Ingress Endpoint URL (v2)                                   |
| -------------- | ----------------------------------------------------------- | ----------------------------------------------------------- |
| `ap-northeast` | `https://console.ap-northeast.spatialwalk.cloud/v1/console` | `wss://api.ap-northeast.spatialwalk.cloud/v2/driveningress` |
| `us-west`      | `https://console.us-west.spatialwalk.cloud/v1/console`      | `wss://api.us-west.spatialwalk.cloud/v2/driveningress`      |

## Cross-region portability

The following identifiers/credentials are **not region-specific** and can be used across regions:

* **App ID**
* **API Key**
* **Avatar ID**
* **Session Token** (generated by the console API)

This means you can switch regions or run multi-region without recreating apps, rotating keys, or re-generating avatars.

## High availability with regional fallback

Services in different regions run on **separate infrastructure**, so you can improve availability by implementing fallback:

* **Primary + backup**: if requests to the primary region fail, retry against another region.
* **Data sync**: since the identifiers/credentials mentioned above are portable, you can keep using them when failing over to another region.


# Audio Format and Sample Rate
Source: https://docs.spatialreal.ai/concepts/audio

Audio input format requirements and sample rate configuration across platforms.

SpatialReal services accept **mono 16-bit PCM (`s16le`)** audio. The sample rate is configured at SDK initialization and must match the audio your application sends.

## Audio Input Requirements

| Property     | Value                                                               |
| ------------ | ------------------------------------------------------------------- |
| Sample Rate  | One of `8000`, `16000`, `22050`, `24000`, `32000`, `44100`, `48000` |
| Channels     | `1` (mono)                                                          |
| Bit Depth    | `16-bit`                                                            |
| Format       | Raw PCM bytes                                                       |
| PCM Encoding | `s16le`                                                             |

<Note>
  If your source audio uses stereo channels, floating point samples, compressed codecs, or an unsupported sample rate, convert it before sending it to SpatialReal.
</Note>

## What Is [`s16le`](https://trac.ffmpeg.org/wiki/audio%20types)?

* `s16` means each audio sample is a signed 16-bit integer.
* `le` means the bytes are stored in little-endian order.
* `mono` means the stream contains a single channel only.

## Supported Sample Rates

| Sample Rate | Common Use Case                                |
| ----------- | ---------------------------------------------- |
| `8000` Hz   | Telephony, low-bandwidth voice                 |
| `16000` Hz  | Speech recognition, voice assistants (default) |
| `22050` Hz  | Low-quality speech synthesis                   |
| `24000` Hz  | Common TTS output (OpenAI, ElevenLabs)         |
| `32000` Hz  | Wideband audio                                 |
| `44100` Hz  | CD-quality audio                               |
| `48000` Hz  | Professional audio, WebRTC default             |

<Warning>
  Sending audio at a sample rate that doesn't match your SDK configuration will produce distorted or silent playback. Always ensure the configured rate matches your audio source.
</Warning>

## How to Choose a Sample Rate

* Use `16000` Hz (default) for most speech-driven integrations — it balances quality and bandwidth well.
* Match your TTS provider's native output rate to avoid resampling. For example, many TTS services output at `24000` Hz.
* In RTC Mode, the sample rate is typically governed by the RTC framework (e.g., LiveKit defaults to `48000` Hz).

<Note>
  The sample rate only affects audio sent to the driving service. The avatar's rendered animation quality is independent of sample rate.
</Note>

## Configuration

Set the sample rate when initializing the client-side SDK through the `AudioFormat` parameter.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    await AvatarSDK.initialize({
      appId: 'YOUR_APP_ID',
      audioFormat: { channelCount: 1, sampleRate: 16000 },
      // ...
    });
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    AvatarSDK.initialize(
        appID: "YOUR_APP_ID",
        configuration: Configuration(
            environment: .intl,
            audioFormat: AudioFormat(sampleRate: 16000),
            drivingServiceMode: .sdk,
            logLevel: .warning
        )
    )
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    AvatarSDK.initialize(
        context = applicationContext,
        appId = "YOUR_APP_ID",
        configuration = Configuration(
            environment = Environment.Intl,
            audioFormat = AudioFormat(16000),
            drivingServiceMode = DrivingServiceMode.SDK,
            logLevel = LogLevel.INFO
        )
    )
    ```
  </Tab>
</Tabs>

## Before Sending Audio

Before sending audio to the SDK or service, make sure your pipeline outputs:

* a supported sample rate
* mono audio
* 16-bit signed PCM samples
* raw PCM bytes rather than WAV headers or compressed audio frames

For a practical overview of common audio encodings and container formats, see the [FFmpeg audio types reference](https://trac.ffmpeg.org/wiki/audio%20types).


# Events
Source: https://docs.spatialreal.ai/concepts/events

Available events to handle in your application, and how to handle them.

AvatarKit client SDKs expose event callbacks so your app can react to connection changes, conversation state transitions, and errors in real time.

## Event Callbacks

| Callback              | Payload             | Location           | Description                                                       |
| --------------------- | ------------------- | ------------------ | ----------------------------------------------------------------- |
| `onFirstRendering`    | —                   | `AvatarView`       | Fires once when the avatar renders its first frame.               |
| `onConnectionState`   | `ConnectionState`   | `AvatarController` | Fires when the WebSocket connection state changes. SDK mode only. |
| `onConversationState` | `ConversationState` | `AvatarController` | Fires when the avatar's playback state changes.                   |
| `onError`             | `AvatarError`       | `AvatarController` | Fires when a runtime error occurs.                                |

## ConnectionState

Indicates the current state of the WebSocket connection to the driving service.

| State          | Description                                                                   |
| -------------- | ----------------------------------------------------------------------------- |
| `disconnected` | No active connection.                                                         |
| `connecting`   | Connection is being established.                                              |
| `connected`    | Connection is active and ready.                                               |
| `failed`       | Connection failed. On iOS and Android, includes `code` and `message` details. |

## ConversationState

Tracks the avatar's current playback state.

| State     | Description                                                |
| --------- | ---------------------------------------------------------- |
| `idle`    | No active conversation — avatar shows breathing animation. |
| `playing` | Avatar is actively playing audio and animation.            |
| `paused`  | Playback is paused.                                        |

## Usage Examples

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    const controller = avatarView.controller;

    // First frame rendered
    avatarView.onFirstRendering = () => {
      console.log('First frame rendered');
    };

    // Connection state (SDK mode only)
    controller.onConnectionState = (state) => {
      console.log('Connection:', state);
    };

    // Conversation state
    controller.onConversationState = (state) => {
      console.log('Conversation:', state);
    };

    // Error handling
    controller.onError = (error) => {
      console.error('Error:', error.code, error.message);
    };
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    let controller = avatarView.controller

    // First frame rendered
    avatarView.onFirstRendering = {
        print("First frame rendered")
    }

    // Connection state (SDK mode only)
    controller.onConnectionState = { state in
        switch state {
        case .connected:
            print("Connected")
        case .failed(let code, let message):
            print("Failed: \(code) \(message)")
        default:
            break
        }
    }

    // Conversation state
    controller.onConversationState = { state in
        print("Conversation: \(state)")
    }

    // Error handling
    controller.onError = { error in
        print("Error: \(error.localizedDescription)")
    }
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    val controller = avatarView.controller

    // First frame rendered
    avatarView.onFirstRendering = {
        Log.d("Avatar", "First frame rendered")
    }

    // Connection state (SDK mode only)
    controller?.onConnectionState = { state ->
        when (state) {
            is ConnectionState.Connected ->
                Log.d("Avatar", "Connected")
            is ConnectionState.Failed ->
                Log.e("Avatar", "Failed: ${state.message}")
            else -> {}
        }
    }

    // Conversation state
    controller?.onConversationState = { state ->
        Log.d("Avatar", "Conversation: $state")
    }

    // Error handling
    controller?.onError = { error ->
        Log.e("Avatar", "Error: ${error.message}")
    }
    ```
  </Tab>
</Tabs>


# Integration Modes
Source: https://docs.spatialreal.ai/concepts/integrations

Choose the right integration approach for your use case

## Choose Your Integration Mode

SpatialReal offers four distinct integration modes to suit different architectural needs, latency requirements, and development preferences.

<CardGroup>
  <Card title="SDK Mode" icon="mobile" href="#sdk-mode">
    Client-centric integration with minimal server-side changes
  </Card>

  <Card title="RTC Mode" icon="tower-broadcast" href="#rtc-mode">
    Ultra-low latency via LiveKit or Agora
  </Card>

  <Card title="Framework Plugin" icon="plug" href="#framework-plugin">
    Seamless integration with LiveKit Agents or TEN Framework
  </Card>

  <Card title="Host Mode" icon="server" href="#host-mode">
    Full control with custom transport layer
  </Card>
</CardGroup>

***

## SDK Mode

In this mode, the client-side application manages the audio input. The developer passes the audio to the SpatialReal Client SDK, which handles the server interaction to retrieve animation data and render the avatar.

<Steps>
  <Step title="Pass Audio">
    Developer passes audio to the SpatialReal SDK on the client.
  </Step>

  <Step title="Inference">
    SDK calls the inference service.
  </Step>

  <Step title="Render">
    SDK receives drive parameters and plays the avatar.
  </Step>
</Steps>

**Best Suited For:**

* **Client-Centric Logic:** Scenarios where the voice agent logic resides primarily on the device.
* **Moderate Latency:** Projects where ultra-low latency is not the absolute priority.
* **Simplified Architecture:** Minimal server-side development required (only for authentication), allowing most logic to remain on the client.

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#64748b"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#111827"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  flowchart LR
      subgraph Client ["🖥️ Client"]
          App["Client App (Voice Logic)"]
          SDK["SpatialReal SDK"]
      end

      Cloud["☁️ SpatialReal Cloud"]

      App -- "Audio Data" --> SDK
      SDK -- "Send Audio" --> Cloud
      Cloud -- "Drive Params" --> SDK
      SDK -- "Render & Play" --> App

      style Client fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000
      style Cloud fill:#fff3e0,stroke:#FF9800,stroke-width:2px,color:#000
  ```
</div>

<Card title="View SDK Mode Guide" icon="arrow-right" href="/guide/sdk-mode" />

***

## RTC Mode

This mode leverages Real-Time Communication (RTC) infrastructure (currently supporting LiveKit and Agora). The developer streams audio to SpatialReal, but instead of returning data to the developer, SpatialReal pushes the drive parameters directly into an RTC room/channel.

<Steps>
  <Step title="Stream Audio">
    Developer streams audio via Server SDK.
  </Step>

  <Step title="Push to RTC">
    SpatialReal Service pushes audio and avatar drive data to an RTC Room.
  </Step>

  <Step title="Subscribe & Play">
    Client joins the room using the SpatialReal RTC Client to subscribe, parse, and play.
  </Step>
</Steps>

<Note>
  The stream contains **binary drive parameters and audio**, not a pre-rendered video feed. Therefore, it cannot be played by standard video players; it requires the SpatialReal Client to render.
</Note>

**Best Suited For:**

* **Existing RTC Users:** Teams already using LiveKit or Agora for voice agents but *not* using their specific agent frameworks (e.g., LiveKit Agents or TEN Framework).
* **Server-Managed State:** Scenarios requiring server-side management of conversation state (e.g., handling interruptions).
* **Ultra-Low Latency:** Leveraging established RTC networks for minimal delay.

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#64748b"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#111827"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  flowchart BT
      subgraph Server ["🔧 Developer Backend"]
          DevServer["Server Logic"]
          ServerSDK["SpatialReal Server SDK"]
          DevServer --> ServerSDK
      end

      subgraph Cloud ["☁️ SpatialReal Cloud"]
          Service["Avatar Service"]
          RTCPub["RTC Egress"]
          Service --> RTCPub
      end

      subgraph RTC ["📡 RTC Provider"]
          Room["LiveKit Room / Agora Channel"]
      end

      subgraph Client ["🖥️ Client"]
          RTCClient["SpatialReal RTC Client"]
      end

      ServerSDK -->|Audio Stream| Service
      RTCPub -->|Drive Params + Audio| Room
      Room -->|Subscribe| RTCClient

      style Client fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000
      style RTC fill:#f3e5f5,stroke:#9C27B0,stroke-width:2px,color:#000
      style Cloud fill:#fff3e0,stroke:#FF9800,stroke-width:2px,color:#000
      style Server fill:#e8eaf6,stroke:#3F51B5,stroke-width:2px,color:#000
  ```
</div>

<Card title="View RTC Mode Guide" icon="arrow-right" href="/guide/rtc-mode" />

***

## Framework Plugin

This is the most streamlined approach for modern voice agent frameworks. Developers use a provided plugin that sits inside the voice agent pipeline (e.g., LiveKit Agents, TEN Framework).

<Steps>
  <Step title="Intercept Audio">
    The Plugin intercepts audio from the agent pipeline.
  </Step>

  <Step title="Send to SpatialReal">
    It sends audio to SpatialReal and configures the RTC push parameters.
  </Step>

  <Step title="Push & Render">
    SpatialReal pushes data to the RTC room (Client side is identical to RTC Mode).
  </Step>
</Steps>

**Best Suited For:**

* **Framework Users:** Teams already using frameworks like LiveKit Agents or TEN Framework.
* **Rapid Integration:** Low implementation cost; the plugin automatically handles signal processing and conversation state (interruption logic).
* **Migration:** Easy to switch if you are currently using other avatar services within these frameworks.

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#64748b"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#111827"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  flowchart BT
      subgraph Agent ["🤖 Voice Agent"]
          Pipeline["Agent Pipeline (LiveKit / TEN)"]
          Plugin["SpatialReal Plugin"]
      end

      subgraph Cloud ["☁️ SpatialReal Cloud"]
          Service["Inference Service"]
          RTCPub["RTC Egress"]
      end

      subgraph RTC ["📡 RTC Provider"]
          Room["LiveKit Room / Agora Channel"]
      end

      subgraph Client ["🖥️ Client"]
          RTCClient["SpatialReal RTC Client"]
      end

      Pipeline -. Audio .-> Plugin
      Plugin -- Audio & Config --> Service
      Service --> RTCPub
      RTCPub -- RTC Push --> Room
      Room -- Subscribe --> RTCClient

      style Client fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000
      style RTC fill:#f3e5f5,stroke:#9C27B0,stroke-width:2px,color:#000
      style Cloud fill:#fff3e0,stroke:#FF9800,stroke-width:2px,color:#000
      style Agent fill:#e8eaf6,stroke:#3F51B5,stroke-width:2px,color:#000
  ```
</div>

<Card title="View Framework Plugin Guide" icon="arrow-right" href="/guide/rtc-mode" />

***

## Host Mode

In Host Mode, the developer acts as the bridge. You use the SpatialReal Server SDK to stream audio to the service and receive streaming drive parameters back. It is then your responsibility to transport this data to the client.

<Steps>
  <Step title="Stream Audio">
    Developer sends streaming audio via Server SDK.
  </Step>

  <Step title="Receive Parameters">
    SpatialReal returns streaming drive parameters to the Developer's Server.
  </Step>

  <Step title="Transport">
    Developer transmits audio and parameters to the Client SDK via a custom transport layer.
  </Step>

  <Step title="Render">
    Client SDK renders the avatar.
  </Step>
</Steps>

<Warning>
  The custom transport layer must ensure data is delivered **without duplication, loss, or disorder**.
</Warning>

**Best Suited For:**

* **Custom Transport:** Teams that already have a reliable, controllable transport layer.
* **Deep Integration:** Developers willing to handle server-side adaptation for maximum control.
* **High Low-Latency Demands:** When you need to optimize the network path manually.

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#64748b"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#111827"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  flowchart LR
      subgraph Client ["🖥️ Client"]
          ClientSDK["SpatialReal Client SDK"]
      end

      subgraph Server ["🔧 Developer Backend"]
          DevServer["Server Logic"]
          ServerSDK["SpatialReal Server SDK"]
      end

      Cloud["☁️ SpatialReal Cloud"]

      DevServer -- "Audio Stream" --> ServerSDK
      ServerSDK -- "Send Audio" --> Cloud
      Cloud -- "Drive Params" --> ServerSDK
      ServerSDK -- "Animation Stream" --> DevServer
      DevServer == "Custom Transport: Audio + Animation" ==> ClientSDK

      style Client fill:#e8f4fd,stroke:#2196F3,stroke-width:2px,color:#000
      style Server fill:#e8eaf6,stroke:#3F51B5,stroke-width:2px,color:#000
      style Cloud fill:#fff3e0,stroke:#FF9800,stroke-width:2px,color:#000
  ```
</div>

<Card title="View Host Mode Guide" icon="arrow-right" href="/guide/host-mode" />

***

## Still Not Sure?

Check integration overview for a side-by-side comparison and runnable example projects.

<Card title="Open Integrations Overview" icon="arrow-right" href="/overview/integration_overview" />

***

## Want a Quick Taste?

Check this quick start guide to get a simple demo up and running in minutes.

<Card title="Open Quickstart Guide" icon="arrow-right" href="/overview/speech-to-avatar" />

***

## Next Steps

<CardGroup>
  <Card title="SDK Mode Guide" icon="book-open" href="/guide/sdk-mode">
    Get started with the simplest integration approach
  </Card>

  <Card title="Host Mode Guide" icon="book-open" href="/guide/host-mode">
    Learn how to implement custom transport
  </Card>

  <Card title="RTC Mode & Framework Plugin Guide" icon="book-open" href="/guide/rtc-mode">
    Set up real-time communication with LiveKit or Agora
  </Card>

  <Card title="Demo Projects" icon="code" href="/overview/demo-projects">
    Explore sample implementations
  </Card>
</CardGroup>


# Client Lifecycle
Source: https://docs.spatialreal.ai/concepts/lifecycle

SDK initialization, avatar loading, connection management, and resource cleanup.

AvatarKit follows a four-stage lifecycle: **Initialize → Load → Connect → Cleanup**. Each stage maps to a core component.

## Overview

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#64748b"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#111827"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  graph LR
      A["<b>INITIALIZE</b><br/>AvatarSDK.initialize()"]
      B["<b>LOAD</b><br/>AvatarManager.load()"]
      C["<b>RENDER</b><br/>AvatarView(avatar)"]
      D["<b>CONNECT</b><br/>controller.start()"]

      A -->B
      B -->C
      C -->D


  ```
</div>

| Stage      | Component          | What happens                                                                                              |
| ---------- | ------------------ | --------------------------------------------------------------------------------------------------------- |
| Initialize | `AvatarSDK`        | Configures app ID, environment, audio format, and driving mode. Must be called once before any other API. |
| Load       | `AvatarManager`    | Downloads and caches avatar assets. Returns an `Avatar` instance.                                         |
| Render     | `AvatarView`       | Creates the rendering surface and its associated `AvatarController`.                                      |
| Connect    | `AvatarController` | Opens a WebSocket to the driving service. The avatar begins responding to audio.                          |

## Stage 1: Initialize

Call `AvatarSDK.initialize()` once at app startup. This sets global configuration that all subsequent operations depend on.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    await AvatarSDK.initialize({
      appId: 'YOUR_APP_ID',
      audioFormat: { channelCount: 1, sampleRate: 16000 },
      drivingServiceMode: 'sdk',
    });
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    AvatarSDK.initialize(
        appID: "YOUR_APP_ID",
        configuration: Configuration(
            environment: .intl,
            audioFormat: AudioFormat(sampleRate: 16000),
            drivingServiceMode: .sdk,
            logLevel: .warning
        )
    )
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    AvatarSDK.initialize(
        context = applicationContext,
        appId = "YOUR_APP_ID",
        configuration = Configuration(
            environment = Environment.Intl,
            audioFormat = AudioFormat(16000),
            drivingServiceMode = DrivingServiceMode.SDK,
            logLevel = LogLevel.INFO
        )
    )
    ```
  </Tab>
</Tabs>

Set `AvatarSDK.sessionToken` before connecting. The token authenticates your session with the driving service.

## Stage 2: Load Avatar

Use `AvatarManager` to download and cache avatar assets. Loading is asynchronous and supports progress tracking.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    const avatar = await AvatarManager.load('AVATAR_ID', (progress) => {
      console.log('Loading:', progress);
    });
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    let avatar = try await AvatarManager.shared.load(id: "AVATAR_ID") { progress in
        print("Loading: \(progress)")
    }
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    val avatar = AvatarManager.load("AVATAR_ID") { progress ->
        Log.d("Avatar", "Loading: $progress")
    }
    ```
  </Tab>
</Tabs>

Loaded assets are cached locally. Subsequent loads for the same avatar ID skip the download. Use `AvatarManager.clear(id:)` or `clearAll()` to manage cache.

## Stage 3: Render

Create an `AvatarView` with the loaded avatar. The view automatically creates an `AvatarController` you can access to manage the connection and send audio.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    const avatarView = new AvatarView(avatar, containerElement);
    const controller = avatarView.controller;
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    let avatarView = AvatarView(avatar: avatar)
    let controller = avatarView.controller
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    val avatarView = AvatarView(context)
    avatarView.init(avatar, lifecycleScope)
    val controller = avatarView.controller
    ```
  </Tab>
</Tabs>

## Stage 4: Connect and Interact

Call `controller.start()` to open the WebSocket connection. Once connected, send audio with `controller.send()` (SDK Mode) or `controller.yield()` (Host Mode).

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#000000"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#000000"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  sequenceDiagram
      autonumber
      participant C as Controller
      participant S as Connection/Voice State

      C->>S: controller.start()
      Note right of S: onConnectionState: connecting
      S-->>C: onConnectionState: connected
      S-->>C: onConversationState: idle
      
      rect rgb(240, 240, 240)
          Note over C,S: Audio Streaming
          C->>S: send(audio, end: false)
          S-->>C: onConversationState: playing
          C->>S: ... multiple chunks ...
          C->>S: send(audio, end: false)
          S-->>C: onConversationState: playing
          C->>S: send(audio chunk, end: true)
      end

      S-->>C: onConversationState: playing
      S-->>C: onConversationState: idle
  ```
</div>

Call `controller.interrupt()` to stop the current playback immediately. Call `controller.close()` when done.

## Cleanup

Clean up resources when the avatar is no longer needed.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    avatarView.dispose();
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    // AvatarView automatically releases resources when removed from the view hierarchy.
    // No explicit cleanup call needed.
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    avatarView.dispose()
    ```
  </Tab>
</Tabs>

<Warning>
  On Web and Android, always call `avatarView.dispose()` when the view is no longer needed. On iOS, resources are released automatically when the view is deallocated.
</Warning>


# Agent Quickstart Walkthrough
Source: https://docs.spatialreal.ai/guide/agent-quickstart-walkthrough

Step-by-step code walkthrough for the LiveKit Agent + SpatialReal quickstart

This page explains the quickstart by backend/frontend steps and keeps each key file in copyable code blocks.

<Card title="Run Quickstart First" icon="rocket" href="/overview/agent">
  Use the clone-and-run flow first, then come back here for implementation details.
</Card>

<Steps>
  <Step title="Configure backend environment and dependencies">
    The backend uses two processes: token server and agent worker.

    <div>
      ```bash title="backend/.env" theme={null}
      LIVEKIT_URL=wss://your-project.livekit.cloud # https://cloud.livekit.io
      LIVEKIT_API_KEY=your_api_key # https://cloud.livekit.io
      LIVEKIT_API_SECRET=your_api_secret # https://cloud.livekit.io

      GOOGLE_API_KEY=your_google_api_key # https://aistudio.google.com/api-keys
      E2E_GOOGLE_MODEL=gemini-2.5-flash-native-audio-preview-12-2025
      E2E_GOOGLE_VOICE=Puck

      SPATIALREAL_API_KEY=your_api_key # https://app.spatialreal.ai/apps
      SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
      SPATIALREAL_AVATAR_ID=6aed28f9-674c-4ffb-89ee-b447b28aa3ed # https://app.spatialreal.ai/avatars/library
      ```

      ```toml title="backend/pyproject.toml" theme={null}
      [project]
      name = "spatialreal-agent-quickstart-backend"
      version = "0.1.0"
      requires-python = ">=3.10,<3.15"
      dependencies = [
        "flask>=3.0.0",
        "flask-cors>=4.0.0",
        "python-dotenv>=1.0.0",
        "livekit-api>=1.1.0",
        "livekit-agents==1.4.5",
        "livekit-plugins-google==1.4.5",
        "livekit-plugins-spatialreal==1.4.5",
      ]
      ```
    </div>
  </Step>

  <Step title="Implement token endpoint and dispatch logic">
    `token_server.py` issues the browser token and dispatches the LiveKit agent.

    <div>
      ```python title="backend/token_server.py" theme={null}
      import asyncio
      import os
      from datetime import timedelta
      from uuid import uuid4

      from dotenv import load_dotenv
      from flask import Flask, jsonify, request
      from flask_cors import CORS
      from livekit import api

      load_dotenv()

      app = Flask(__name__)
      CORS(app)

      LIVEKIT_URL = os.getenv("LIVEKIT_URL")
      LIVEKIT_API_KEY = os.getenv("LIVEKIT_API_KEY")
      LIVEKIT_API_SECRET = os.getenv("LIVEKIT_API_SECRET")


      async def create_room_and_dispatch(room_name: str) -> None:
          lkapi = api.LiveKitAPI(LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET)
          try:
              try:
                  await lkapi.room.create_room(api.CreateRoomRequest(name=room_name))
              except Exception:
                  pass

              await lkapi.agent_dispatch.create_dispatch(
                  api.CreateAgentDispatchRequest(room=room_name, agent_name="voice-assistant")
              )
          finally:
              await lkapi.aclose()


      @app.route("/token", methods=["POST"])
      def token():
          if not LIVEKIT_API_KEY or not LIVEKIT_API_SECRET:
              return jsonify({"error": "LiveKit credentials not configured"}), 500

          body = request.get_json() or {}
          room_name = body.get("room", "voice-agent-room")
          requested_identity = body.get("identity")
          identity = (
              requested_identity.strip()
              if isinstance(requested_identity, str) and requested_identity.strip()
              else f"browser-{uuid4().hex[:8]}"
          )

          jwt = (
              api.AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET)
              .with_identity(identity)
              .with_name(identity)
              .with_ttl(timedelta(hours=1))
              .with_grants(
                  api.VideoGrants(
                      room_join=True,
                      room=room_name,
                      can_publish=True,
                      can_subscribe=True,
                      can_publish_data=True,
                  )
              )
              .to_jwt()
          )

          try:
              asyncio.run(create_room_and_dispatch(room_name))
          except Exception as exc:
              print(f"Warning: Failed to dispatch agent: {exc}")

          return jsonify(
              {
                  "token": jwt,
                  "url": LIVEKIT_URL,
                  "room": room_name,
                  "identity": identity,
              }
          )


      if __name__ == "__main__":
          app.run(host="0.0.0.0", port=8080, debug=True)
      ```
    </div>
  </Step>

  <Step title="Implement agent worker + AvatarSession">
    `agent.py` runs the realtime LLM and starts SpatialReal avatar publishing in the same room.

    <div>
      ```python title="backend/agent.py" theme={null}
      import os

      from dotenv import load_dotenv
      from livekit.agents import Agent, AgentSession, AutoSubscribe, JobContext, WorkerOptions, cli
      from livekit.plugins import google
      from livekit.plugins.spatialreal import AvatarSession

      load_dotenv()


      class VoiceAssistant(Agent):
          def __init__(self) -> None:
              super().__init__(
                  instructions="You are a helpful voice assistant. Keep replies short and natural."
              )


      async def entrypoint(ctx: JobContext) -> None:
          await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

          session = AgentSession(
              llm=google.realtime.RealtimeModel(
                  model=os.getenv("E2E_GOOGLE_MODEL", "gemini-2.5-flash"),
                  voice=os.getenv("E2E_GOOGLE_VOICE", "Puck"),
                  api_key=os.getenv("GOOGLE_API_KEY"),
              )
          )

          avatar = AvatarSession()
          await avatar.start(session, room=ctx.room)

          await session.start(agent=VoiceAssistant(), room=ctx.room)


      if __name__ == "__main__":
          cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, agent_name="voice-assistant"))
      ```
    </div>
  </Step>

  <Step title="Configure frontend env and Vite">
    Frontend reads token endpoint and room name from env and proxies `/token` locally.

    <div>
      ```bash title="frontend/.env" theme={null}
      VITE_SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
      VITE_SPATIALREAL_AVATAR_ID=6aed28f9-674c-4ffb-89ee-b447b28aa3ed # https://app.spatialreal.ai/avatars/library
      VITE_TOKEN_ENDPOINT=http://localhost:8080/token
      VITE_ROOM_NAME=voice-agent-room
      ```

      ```ts title="frontend/vite.config.ts" theme={null}
      import { defineConfig } from 'vite'
      import react from '@vitejs/plugin-react'
      import { avatarkitVitePlugin } from '@spatialwalk/avatarkit/vite'

      export default defineConfig({
        plugins: [react(), avatarkitVitePlugin()],
        server: {
          port: 3000,
          proxy: {
            '/token': {
              target: 'http://localhost:8080',
              changeOrigin: true,
            },
          },
        },
      })
      ```
    </div>
  </Step>

  <Step title="AvatarKit UI setup">
    This frontend uses AvatarKit UI components (`SpatialRealAvatarProvider`, `SpatialRealAvatarCanvas`, etc.).

    To set up the same UI stack:

    1. Complete shadcn base setup: [shadcn manual installation](https://ui.shadcn.com/docs/installation/manual)
    2. Install dependencies and component files:

    <div>
      ```bash theme={null}
      pnpm add @spatialwalk/avatarkit @livekit/components-react @livekit/components-styles
      npx shadcn@latest add https://ui.spatialreal.ai/r/spatialreal-avatar.json
      ```
    </div>
  </Step>

  <Step title="Implement AvatarKit UI frontend page">
    This frontend is implemented with [AvatarKit UI](/sdk-reference/web-sdk/avatarkit-ui).
    `App.tsx` requests token, mounts `SpatialRealAvatarProvider`, and controls microphone state.

    <div>
      ```tsx title="frontend/src/App.tsx" theme={null}
      import { useState } from 'react'
      import '@livekit/components-styles'
      import { Button } from '@/components/ui/button'
      import { Track } from 'livekit-client'

      import {
        SpatialRealAvatarCanvas,
        SpatialRealAvatarError,
        SpatialRealAvatarFrame,
        SpatialRealAvatarLoading,
        SpatialRealAvatarProvider,
        SpatialRealAvatarStatus,
        useSpatialRealAvatarContext,
      } from '@/components/spatialreal-avatar'

      type AvatarConnection = {
        url: string
        token: string
        roomName: string
      }

      type TokenResponse = {
        url: string
        token: string
        room: string
      }

      function AvatarPanel({ onExit }: { onExit: () => void }) {
        const avatar = useSpatialRealAvatarContext()
        const [pending, setPending] = useState(false)
        const micPublication = avatar.room?.localParticipant.getTrackPublication(Track.Source.Microphone)
        const hasPublishedMic = Boolean(micPublication?.track)
        const isMicMuted = micPublication?.isMuted ?? false

        const toggleMicrophone = async () => {
          if (pending || !avatar.isConnected) return
          setPending(true)

          try {
            if (!hasPublishedMic) {
              await avatar.startPublishingMicrophone()
            } else if (isMicMuted) {
              await micPublication?.unmute()
            } else {
              await micPublication?.mute()
            }
          } finally {
            setPending(false)
          }
        }

        const disconnect = async () => {
          if (pending) return
          setPending(true)

          try {
            await avatar.disconnect()
          } finally {
            setPending(false)
            onExit()
          }
        }

        return (
          <div className="flex w-full max-w-[760px] flex-col gap-3">
            <SpatialRealAvatarFrame className="overflow-hidden">
              <SpatialRealAvatarCanvas minHeight={420} />
              <SpatialRealAvatarLoading />
              <SpatialRealAvatarError />
            </SpatialRealAvatarFrame>

            <div className="flex flex-wrap items-center gap-2">
              <Button disabled={!avatar.isConnected || pending} onClick={() => void toggleMicrophone()} type="button">
                {!hasPublishedMic ? 'Enable Mic' : isMicMuted ? 'Unmute Mic' : 'Mute Mic'}
              </Button>

              <Button disabled={pending} onClick={() => void disconnect()} type="button" variant="outline">
                Disconnect
              </Button>

              <SpatialRealAvatarStatus />

              <span className="text-sm">
                {avatar.error
                  ? avatar.error.message
                  : avatar.isConnected
                    ? !hasPublishedMic
                      ? 'Connected. Mic is off, enable mic to talk.'
                      : isMicMuted
                        ? 'Connected. Mic is muted.'
                        : 'Connected. Mic is on, start speaking.'
                    : 'Connecting...'}
              </span>
            </div>
          </div>
        )
      }

      export default function App() {
        const appId = import.meta.env.VITE_SPATIALREAL_APP_ID
        const avatarId = import.meta.env.VITE_SPATIALREAL_AVATAR_ID
        const tokenEndpoint = import.meta.env.VITE_TOKEN_ENDPOINT || '/token'
        const roomName = import.meta.env.VITE_ROOM_NAME || 'voice-agent-room'

        const [connection, setConnection] = useState<AvatarConnection | null>(null)
        const [connecting, setConnecting] = useState(false)
        const [status, setStatus] = useState('Click Connect to start')

        const requestConnection = async () => {
          if (connecting || connection) return
          if (!appId || !avatarId) {
            setStatus('Missing VITE_SPATIALREAL_APP_ID or VITE_SPATIALREAL_AVATAR_ID in .env')
            return
          }

          setConnecting(true)
          setStatus('Requesting token...')

          try {
            const response = await fetch(tokenEndpoint, {
              method: 'POST',
              headers: { 'Content-Type': 'application/json' },
              body: JSON.stringify({ room: roomName }),
            })
            if (!response.ok) throw new Error('Failed to fetch token')

            const payload = (await response.json()) as TokenResponse
            if (!payload.url || !payload.token || !payload.room) {
              throw new Error('Token response is missing url, token, or room')
            }

            setConnection({
              url: payload.url,
              token: payload.token,
              roomName: payload.room,
            })
            setStatus('Connecting avatar...')
          } catch (error) {
            setStatus(error instanceof Error ? error.message : 'Failed to request token')
          } finally {
            setConnecting(false)
          }
        }

        if (!appId || !avatarId) {
          return <div className="p-4">Missing required environment variables. Check `.env`.</div>
        }

        return (
          <div className="flex min-h-screen items-center justify-center p-4">
            {connection ? (
              <SpatialRealAvatarProvider
                appId={appId}
                avatarId={avatarId}
                connection={connection}
                onConnected={() => setStatus('Connected. Click Enable Mic to talk.')}
                onDisconnected={() => setStatus('Disconnected')}
                onAvatarError={(error) => setStatus(error.message)}
              >
                <AvatarPanel
                  onExit={() => {
                    setConnection(null)
                    setStatus('Disconnected')
                  }}
                />
              </SpatialRealAvatarProvider>
            ) : (
              <div className="flex w-full max-w-[560px] flex-col gap-2.5">
                <Button disabled={connecting} onClick={() => void requestConnection()} type="button">
                  {connecting ? 'Connecting...' : 'Connect'}
                </Button>
                <span className="text-sm">{status}</span>
              </div>
            )}
          </div>
        )
      }
      ```
    </div>
  </Step>

  <Step title="Understand runtime sequence">
    Use this sequence to trace issues across backend, agent, and frontend.

    <div>
      ```text theme={null}
      frontend -> /token -> token_server creates JWT + dispatches agent
      agent worker joins room -> starts Gemini session + AvatarSession
      frontend connects with token -> avatar renders -> mic publish/unpublish drives conversation
      ```
    </div>
  </Step>
</Steps>


# Host Mode
Source: https://docs.spatialreal.ai/guide/host-mode

Self-managed networking with AvatarKit Server SDK

## What is Host Mode?

In Host Mode, **your application** manages the network connection to AvatarKit's server-side SDK. Your server sends encoded messages to your client, and the client SDK receives and **decodes them internally** for synchronized playback and rendering.

```mermaid theme={null}
flowchart LR
    A["🖥️ AvatarKit<br/>Server SDK"] -->|Encoded Messages| B["Your App<br/>(Network Layer)"]
    B -->|yieldAudioData| C["AvatarKit<br/>Client SDK"]
    B -->|yieldFramesData| C
    C -->|Decode & Render| D["🖥️ Avatar Rendering"]
```

<Warning>
  Host Mode **requires** AvatarKit's server-side SDK to generate the encoded messages. The data passed to `yieldAudioData()` and `yieldFramesData()` are encoded messages from the server SDK — not raw audio or animation data you create yourself.
</Warning>

## When to Use

* **Custom network layer** — you manage the connection between your client and AvatarKit's server SDK yourself
* **RTC integration** — messages are relayed through a real-time communication server (LiveKit, Agora, etc.)
* **Proxy architecture** — your backend acts as a relay between the client and AvatarKit server

## Requirements

| Requirement              | Description                                                                  |
| ------------------------ | ---------------------------------------------------------------------------- |
| **App ID**               | Obtained from [SpatialReal Studio](https://app.spatialreal.ai)               |
| **Session Token**        | **Not required** on the client side                                          |
| **AvatarKit Server SDK** | Your backend must integrate with AvatarKit's server SDK to generate messages |

## SDK Mode vs Host Mode

| Aspect               | SDK Mode                                         | Host Mode                                          |
| -------------------- | ------------------------------------------------ | -------------------------------------------------- |
| **Network**          | Client SDK connects to AvatarKit server directly | Your app relays messages from AvatarKit Server SDK |
| **Message Decoding** | Handled internally                               | Handled internally (same)                          |
| **Session Token**    | Required (client-side)                           | Not required (client-side)                         |
| **Server SDK**       | Not needed                                       | **Required** on your backend                       |
| **Key Methods**      | `send()`, `start()`, `close()`                   | `yieldAudioData()`, `yieldFramesData()`            |
| **Use Case**         | Simplest integration                             | Custom networking / RTC relay                      |

## Key Concepts

### ConversationId Management

ConversationId links audio and animation messages for a single conversation session:

1. Call `yieldAudioData()` — returns a `conversationId`
2. Use that `conversationId` when calling `yieldFramesData()`
3. Messages with a **mismatched** conversationId will be **discarded**
4. Use `getCurrentConversationId()` to retrieve the current active session ID

<Warning>
  **Important:** Always use the conversationId returned by `yieldAudioData()` when sending animation messages. Mismatched IDs cause messages to be silently dropped.
</Warning>

### Fallback Mechanism

If you provide empty animation data (empty array or undefined), the SDK automatically enters **audio-only mode** for that session. Once in audio-only mode, any subsequent animation data for that session is ignored — only audio continues playing.

## Get Started

<CardGroup>
  <Card title="Web Demo" icon="globe" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    GitHub demo repository
  </Card>

  <Card title="iOS Demo" icon="apple" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    GitHub demo repository
  </Card>

  <Card title="Android Demo" icon="android" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    GitHub demo repository
  </Card>
</CardGroup>


# Introduction
Source: https://docs.spatialreal.ai/guide/introduction

Step-by-step guides for integrating SpatialReal avatars into your application

The **Guide** walks you through integrating SpatialReal avatars by **integration mode**. Choose the mode that matches your architecture, then follow the linked guides for your platform (Web, iOS, Android) or backend (LiveKit, Server SDK).

## Integration Modes

<CardGroup>
  <Card title="SDK Mode" icon="code" href="/guide/sdk-mode">
    Client sends audio; SDK handles networking and rendering. Best for simple, client-centric apps.
  </Card>

  <Card title="RTC Mode & Framework Plugins" icon="tower-broadcast" href="/guide/rtc-mode">
    Real-time voice via LiveKit. Use framework plugins or Server SDK with egress.
  </Card>

  <Card title="Host Mode" icon="server" href="/guide/host-mode">
    You control the transport. Server SDK sends audio and relays animation to the client SDK.
  </Card>

  <Card title="Agent Mode" icon="robot" href="/guide/agent-mode">
    Fully managed voice agent (coming soon).
  </Card>
</CardGroup>

## Where to Start

* **New to SpatialReal?** Start with [Overview → Speech-to-Avatar Quickstart](/overview/speech-to-avatar), then open the guide for your chosen mode.
* **Already chose a mode?** Use the sidebar: pick **SDK Mode**, **RTC Mode**, or **Host Mode**, then follow the platform-specific quickstarts (Web, iOS, Android) or server guides (LiveKit).
* **Want working code?** Check out our [Demo Projects](/overview/demo-projects) for complete examples you can run immediately.

## Quickstart Code Walkthroughs

<CardGroup>
  <Card title="Speech-to-Avatar Walkthrough" icon="book-open" href="/guide/speech-to-avatar-walkthrough">
    Understand the SDK Mode quickstart code structure and runtime flow.
  </Card>

  <Card title="Agent Quickstart Walkthrough" icon="book-open" href="/guide/agent-quickstart-walkthrough">
    Understand the LiveKit agent quickstart backend and frontend architecture.
  </Card>
</CardGroup>

<Card title="AvatarKit Voice Agent Demo" icon="github" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
  A full reference repository with implementation details, including different frontend UI options and multiple backend agent patterns.
</Card>

## Quick Links

| Mode     | Client / Platform                                                                            | Server                                                                                                                    |
| -------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **SDK**  | [Web](/guide/sdk-mode-web) · [iOS](/ios-sdk/quickstart) · [Android](/android-sdk/quickstart) | Session token from your backend only                                                                                      |
| **RTC**  | [LiveKit Client](/guide/rtc-livekit-client)                                                  | [LiveKit Server](/guide/rtc-livekit-server)                                                                               |
| **Host** | [Web](/web-sdk/host-mode) · [iOS](/ios-sdk/host-mode) · [Android](/android-sdk/host-mode)    | [Python](/server/python-sdk/python-host-mode) · [Golang](/server/go-sdk/go-host-mode) · [JS](/server/js-sdk/js-host-mode) |


# LiveKit Client
Source: https://docs.spatialreal.ai/guide/rtc-livekit-client

Real-time voice communication with avatars using LiveKit

```mermaid theme={null}
flowchart LR
    A["🎤 Microphone"] --> B["AvatarPlayer"]
    B <-->|WebRTC| C["LiveKit Server"]
    B --> D["AvatarKit SDK"]
    D --> E["🖥️ Avatar Rendering"]
```

<Note>
  **Web Only:** RTC Mode is currently available for Web applications only.
</Note>

<Warning>
  **Do not call `initializeAudioContext()`** — it is not needed in RTC Mode. Avatar audio is delivered as a native WebRTC audio track by the LiveKit client SDK, not through the AvatarKit SDK's internal audio player. The `AvatarPlayer` adapter only feeds **animation data** to the SDK for rendering; audio playback is handled entirely by LiveKit's WebRTC stack.
</Warning>

## Installation

<Warning>
  **Critical:** We are only compatable with `livekit-client` under version `2.17`, since `2.17` introduced single PC by default. We are working on adding support for `2.17`+ in a future release, but for now please ensure you install version `2.16.1` to avoid compatibility issues.
</Warning>

<Tabs>
  <Tab title="pnpm">
    ```bash theme={null}
    pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>

  <Tab title="npm">
    ```bash theme={null}
    npm install @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>

  <Tab title="yarn">
    ```bash theme={null}
    yarn add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
    ```
  </Tab>
</Tabs>

You also need to configure your build tool for WASM files — see [Build Tool Configuration](/guide/sdk-mode-web#build-tool-configuration).

## Authentication

| Credential        | How to Obtain                                                 | Notes                             |
| ----------------- | ------------------------------------------------------------- | --------------------------------- |
| **App ID**        | [SpatialReal Studio](https://app.spatialreal.ai) → Create App | For SDK initialization            |
| **Session Token** | Your backend → AvatarKit Server                               | For avatar loading (max 24 hours) |
| **LiveKit Token** | Your backend → LiveKit Server                                 | For RTC room connection           |

## Quick Start

For a complete client + server implementation, see [AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-cascade-voice-agent).

<Steps>
  <Step title="Initialize SDK in Host Mode">
    ```typescript theme={null}
    import { AvatarSDK, AvatarManager, AvatarView, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

    await AvatarSDK.initialize('your-app-id', {
      environment: Environment.intl,
      drivingServiceMode: DrivingServiceMode.host,  // MUST be host for RTC
    })

    AvatarSDK.setSessionToken('your-session-token')
    ```
  </Step>

  <Step title="Load Avatar & Create View">
    ```typescript theme={null}
    const avatar = await AvatarManager.shared.load('avatar-id')
    const container = document.getElementById('avatar-container')!
    const avatarView = new AvatarView(avatar, container)
    ```
  </Step>

  <Step title="Create Player with LiveKit Provider">
    ```typescript theme={null}
    import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'

    const provider = new LiveKitProvider()
    const player = new AvatarPlayer(provider, avatarView, {
      logLevel: 'warning',
    })
    ```
  </Step>

  <Step title="Connect to LiveKit Server">
    ```typescript theme={null}
    await player.connect({
      url: 'wss://your-livekit-server.com',
      token: 'your-livekit-token',
      roomName: 'room-name',
    })
    ```
  </Step>

  <Step title="Start Voice Interaction">
    ```typescript theme={null}
    // Start microphone publishing
    await player.startPublishing()

    // Stop microphone
    await player.stopPublishing()

    // Disconnect when done
    await player.disconnect()
    ```
  </Step>
</Steps>

## AvatarPlayer API

### Constructor

```typescript theme={null}
new AvatarPlayer(provider: LiveKitProvider, avatarView: AvatarView, options?: AvatarPlayerOptions)
```

### AvatarPlayerOptions

```typescript theme={null}
interface AvatarPlayerOptions {
  /** Start speaking transition frames, default 5 (~200ms at 25fps) */
  transitionStartFrameCount?: number

  /** End speaking transition frames, default 40 (~1600ms at 25fps) */
  transitionEndFrameCount?: number

  /** Log level: 'info' | 'warning' | 'error' | 'none', default 'warning' */
  logLevel?: LogLevel
}
```

### Connection

```typescript theme={null}
// Connect to LiveKit server
await player.connect(config: LiveKitConnectionConfig)

// Disconnect and clean up
await player.disconnect()

// Reconnect using last config (useful after stalls)
await player.reconnect()

// Check connection status
player.isConnected  // boolean
player.getConnectionState()  // ConnectionState
```

#### LiveKitConnectionConfig

```typescript theme={null}
interface LiveKitConnectionConfig {
  url: string       // LiveKit server URL (wss://...)
  token: string     // Auth token from your backend
  roomName: string  // Room name
}
```

### Microphone Control

```typescript theme={null}
// Start microphone (requests permission automatically)
await player.startPublishing()

// Stop microphone
await player.stopPublishing()
```

### Custom Audio Publishing

For non-microphone audio sources like audio elements or Web Audio API.

```typescript theme={null}
// Publish a custom audio track
await player.publishAudio(track: MediaStreamTrack)

// Stop custom audio
await player.unpublishAudio()
```

| Audio Source       | How to Obtain Track                                                      |
| ------------------ | ------------------------------------------------------------------------ |
| `<audio>` element  | `audioElement.captureStream().getAudioTracks()[0]`                       |
| Screen share audio | `getDisplayMedia({ audio: true })`                                       |
| Web Audio API      | `audioContext.createMediaStreamDestination().stream.getAudioTracks()[0]` |

<Note>
  `unpublishAudio()` does **not** stop the track. You are responsible for calling `track.stop()` to release the resource.
</Note>

### Native Client Access

Access the underlying LiveKit `Room` instance for advanced features.

```typescript theme={null}
import { LiveKitRoom } from '@spatialwalk/avatarkit-rtc'

// Via provider (recommended — full type safety)
const room = provider.getNativeClient()  // LiveKitRoom | null

// Via player (requires type assertion)
const room = player.getNativeClient() as LiveKitRoom | null

console.log('Remote participants:', room?.remoteParticipants.size)
```

### ConnectionState

```typescript theme={null}
type ConnectionState = 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'
```

## Events

```typescript theme={null}
player.on(event: string, handler: Function)
player.off(event: string, handler: Function)
```

| Event                        | Callback                   | Description                            |
| ---------------------------- | -------------------------- | -------------------------------------- |
| `'connected'`                | `()`                       | Connected to RTC server                |
| `'disconnected'`             | `()`                       | Disconnected from RTC server           |
| `'error'`                    | `(error: Error)`           | An error occurred                      |
| `'connection-state-changed'` | `(state: ConnectionState)` | Connection state changed               |
| `'stalled'`                  | `()`                       | Data stream stalled (no frames for 3s) |

<Note>
  When a `stalled` event fires, the avatar automatically transitions to idle animation. Consider calling `player.reconnect()` to recover.
</Note>

**Stall recovery example:**

```typescript theme={null}
player.on('stalled', async () => {
  console.log('Stream stalled, reconnecting...')
  try {
    await player.reconnect()
  } catch (error) {
    console.error('Reconnection failed:', error)
  }
})
```

## Complete Example

```typescript theme={null}
import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

async function init() {
  // Initialize SDK in host mode
  await AvatarSDK.initialize('your-app-id', {
    environment: Environment.intl,
    drivingServiceMode: DrivingServiceMode.host,
  })
  AvatarSDK.setSessionToken('your-session-token')

  // Load avatar and create view
  const avatar = await AvatarManager.shared.load('character-id')
  const container = document.getElementById('avatar-container')!
  const avatarView = new AvatarView(avatar, container)

  // Create player
  const provider = new LiveKitProvider()
  const player = new AvatarPlayer(provider, avatarView, {
    logLevel: 'info',
    transitionStartFrameCount: 5,
    transitionEndFrameCount: 40,
  })

  // Listen to events
  player.on('connected', () => console.log('Connected!'))
  player.on('disconnected', () => console.log('Disconnected!'))
  player.on('error', (err) => console.error('Error:', err))
  player.on('stalled', async () => {
    console.log('Stream stalled, reconnecting...')
    await player.reconnect()
  })

  // Connect to LiveKit server
  await player.connect({
    url: 'wss://your-livekit-server.com',
    token: 'your-livekit-token',
    roomName: 'my-room',
  })

  // Start microphone
  await player.startPublishing()
}
```

## Browser Compatibility

| Browser | Minimum Version | Notes                           |
| ------- | --------------- | ------------------------------- |
| Chrome  | 94+             | VP8 + RTCRtpScriptTransform     |
| Firefox | 117+            | RTCRtpScriptTransform required  |
| Safari  | 15.4+           | RTCRtpScriptTransform supported |
| Edge    | 94+             | Same as Chrome                  |


# Serverside Setup With LiveKit
Source: https://docs.spatialreal.ai/guide/rtc-livekit-server

Server-side setup for LiveKit + SpatialReal avatar

This guide covers how to send avatar audio and animation to a LiveKit room from your backend. Choose the path that matches how you build your voice agent.

***

## 1. Using the LiveKit Agents Framework

If you already use [LiveKit Agents](https://docs.livekit.io/agents/) to build your voice agent, the simplest option is [livekit-plugins-spatialreal](https://github.com/spatialwalk/livekit-plugins-spatialreal). The plugin hooks into your agent pipeline, sends TTS audio to SpatialReal, and publishes the lip-synced avatar stream into your LiveKit room—so you don’t manage the Server SDK or egress config yourself.

For a working end-to-end reference, see [AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-cascade-voice-agent).

### How it works

1. Your agent runs as usual (VAD, STT, LLM, TTS or Realtime), your current AgentSession setup.
2. The plugin intercepts TTS audio from the agent and sends it to SpatialReal.
3. SpatialReal generates the avatar stream and publishes it to the same LiveKit room.
4. Your client join the room and use the SpatialReal RTC client to render the avatar.

Interruption and conversation state are handled inside the plugin.

### Python: livekit-plugins-spatialreal

**Install**

```bash theme={null}
pip install livekit-plugins-spatialreal
```

**Configure**

| Variable                       | Required | Description               |
| ------------------------------ | -------- | ------------------------- |
| `SPATIALREAL_API_KEY`          | Yes      | Your SpatialReal API key  |
| `SPATIALREAL_APP_ID`           | Yes      | Your SpatialReal app ID   |
| `SPATIALREAL_AVATAR_ID`        | Yes      | Avatar to use             |
| `SPATIALREAL_CONSOLE_ENDPOINT` | No       | Override console endpoint |
| `SPATIALREAL_INGRESS_ENDPOINT` | No       | Override ingress endpoint |
| `LIVEKIT_URL`                  | Yes      | Your LiveKit server URL   |
| `LIVEKIT_API_KEY`              | Yes      | LiveKit API key           |
| `LIVEKIT_API_SECRET`           | Yes      | LiveKit API secret        |

**Use in your agent**

Create an `AvatarSession` and start it with your agent session and room. The plugin will attach to the pipeline and publish the avatar to the room.

```python theme={null}
from livekit.agents import Agent, AgentSession, JobContext, cli, WorkerOptions
from livekit.plugins import spatialreal

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    session = AgentSession(vad=vad, stt=stt, llm=llm, tts=tts)
    avatar = spatialreal.AvatarSession()
    await avatar.start(session, room=ctx.room)

    await session.start(agent=YourAgent(), room=ctx.room)

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

Full API and options: **[livekit-plugins-spatialreal](https://github.com/spatialwalk/livekit-plugins-spatialreal)** (PyPI: [livekit-plugins-spatialreal](https://pypi.org/project/livekit-plugins-spatialreal/)).

### JavaScript plugin (coming soon)

JavaScript/TypeScript plugin for the LiveKit Agents framework will be available soon.

***

## 2. Using the Server SDK with LiveKit Egress

If you are **not** using LiveKit Agents (e.g. custom voice agent server), you can still send avatar output to a LiveKit room by using the **SpatialReal AvatarKit Server SDK** with **LiveKit egress** enabled.

You can also use the [AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-cascade-voice-agent) as a concrete reference for wiring a LiveKit-based voice agent.

In this setup:

* Your server sends audio to the avatar service and passes **LiveKit egress config** (room URL, credentials, room name, publisher id).
* The avatar service streams **audio + animation** directly into the LiveKit room (no need for your server to relay that data).
* Clients join the room and use [@spatialwalk/avatarkit-rtc](/guide/rtc-livekit-client) to render the avatar.

You keep full control over when and how you send audio (and optional interrupt signals; see below), while transport and sync are handled by the service and the RTC client.

### Configuration

| Field          | Description                                                |
| -------------- | ---------------------------------------------------------- |
| `url`          | LiveKit server URL (e.g., `wss://your-livekit-server.com`) |
| `api_key`      | LiveKit API key                                            |
| `api_secret`   | LiveKit API secret                                         |
| `room_name`    | LiveKit room name to publish to                            |
| `publisher_id` | Publisher identity in the room                             |

### Golang Example

```go theme={null}
session := avatarsdkgo.NewAvatarSession(
	avatarsdkgo.WithAPIKey("your-api-key"),
	avatarsdkgo.WithAppID("your-app-id"),
	avatarsdkgo.WithAvatarID("your-avatar-id"),
	avatarsdkgo.WithConsoleEndpointURL("https://console.us-west.spatialwalk.cloud/v1/console"),
	avatarsdkgo.WithIngressEndpointURL("wss://api.us-west.spatialwalk.cloud/v2/driveningress"),
	avatarsdkgo.WithExpireAt(time.Now().Add(5 * time.Minute)),
	avatarsdkgo.WithLiveKitEgress(&avatarsdkgo.LiveKitEgressConfig{
		URL:         "wss://your-livekit-server.com",
		APIKey:      "livekit-api-key",
		APISecret:   "livekit-api-secret",
		RoomName:    "your-room-name",
		PublisherID: "avatar-publisher",
	}),
	avatarsdkgo.WithOnError(func(err error) {
		// handle error
	}),
	avatarsdkgo.WithOnClose(func() {
		// handle close
	}),
)
```

### Python Example

```python theme={null}
from avatarkit import new_avatar_session, LiveKitEgressConfig

session = new_avatar_session(
    api_key="your-api-key",
    app_id="your-app-id",
    avatar_id="your-avatar-id",
    expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
    console_endpoint_url="https://console.us-west.spatialwalk.cloud/v1/console",
    ingress_endpoint_url="wss://api.us-west.spatialwalk.cloud/v2/driveningress",
    livekit_egress=LiveKitEgressConfig(
        url="wss://your-livekit-server.com",
        api_key="livekit-api-key",
        api_secret="livekit-api-secret",
        room_name="your-room-name",
        publisher_id="avatar-publisher",
    ),
    on_error=lambda err: print(f"Error: {err}"),
    on_close=lambda: print("Session closed"),
)
```

After the session is created with LiveKit egress, send audio as in normal Server SDK usage; the service will publish to the configured room automatically.

### Important Notes

When LiveKit egress is enabled:

* The `TransportFrames` / `transport_frames` callback will **not** be invoked
* Audio and animation data are published directly to the specified LiveKit room
* Your client must use the **@spatialwalk/avatarkit-rtc** package to render the avatar (standard video players won't work). See [LiveKit Client Guide](/guide/rtc-livekit-client) for client setup.

### Interrupt

You can interrupt the avatar (e.g. when the user asks a new question). The interrupt uses the most recent request ID, even after `end=true` was sent.

**Golang**

```go theme={null}
// Send audio
requestID, err := session.SendAudio(audioData, true)

// Later, interrupt if needed
interruptedID, err := session.Interrupt()
```

**Python**

```python theme={null}
# Send audio
request_id = await session.send_audio(audio_data, end=True)

# Later, interrupt if needed
interrupted_id = await session.interrupt()
```

***

## Summary

| Approach                        | Best for                                                           | Where to go                                                                                                      |
| ------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| **LiveKit Agents + Plugin**     | Python (or future JS) agents using the LiveKit Agents framework    | [livekit-plugins-spatialreal](https://github.com/spatialwalk/livekit-plugins-spatialreal) (this page, section 1) |
| **Server SDK + LiveKit egress** | Custom backends, any language with a Server SDK (Go, Python, etc.) | This page, section 2                                                                                             |

For client-side setup (joining the room and rendering the avatar), see [LiveKit client guide](/guide/rtc-livekit-client).


# RTC Mode & Framework Plugin
Source: https://docs.spatialreal.ai/guide/rtc-mode

Use LiveKit as the transport layer for SpatialReal avatars

## What is RTC Mode?

RTC Mode enables **real-time voice communication** with avatars through LiveKit. It builds on top of [Host Mode](/guide/host-mode) — the `@spatialwalk/avatarkit-rtc` adapter package handles the RTC connection and feeds audio/animation data to the SDK automatically.

```mermaid theme={null}
flowchart LR
    A["🎤 Microphone"] --> B["AvatarPlayer<br/>(RTC Adapter)"]
    B <-->|WebRTC| C["RTC Server<br/>(LiveKit)"]
    B --> D["AvatarKit SDK"]
    D --> E["🖥️ Avatar Rendering"]
```

<Note>
  **Web Only:** RTC Mode is currently available for Web applications only. iOS and Android support is planned for a future release. In the meantime, native mobile platforms can use [Host Mode](/guide/host-mode) with your own RTC implementation.
</Note>

<Card title="AvatarKit Voice Agent Demo" icon="github" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
  A full reference repository with implementation details, including different frontend UI options and multiple backend agent patterns.
</Card>

## When to Use

* **Real-time voice conversation** — users talk to an avatar via microphone
* **Low-latency interaction** — WebRTC provides sub-second latency
* **Server-side AI** — your RTC server processes audio and generates responses

## Packages Required

| Package                      | Purpose              | Required |
| ---------------------------- | -------------------- | -------- |
| `@spatialwalk/avatarkit`     | Avatar rendering SDK | Yes      |
| `@spatialwalk/avatarkit-rtc` | RTC adapter          | Yes      |
| `livekit-client@2.16.1`      | LiveKit RTC SDK      | Yes      |

<Warning>
  **Critical:** We are only compatable with `livekit-client` under version `2.17`, since `2.17` introduced single PC by default. We are working on adding support for `2.17`+ in a future release, but for now please ensure you install version `2.16.1` to avoid compatibility issues.
</Warning>

## LiveKit Browser Compatibility

| Browser     | Minimum Version |
| ----------- | --------------- |
| **Chrome**  | 94+             |
| **Firefox** | 117+            |
| **Safari**  | 15.4+           |
| **Edge**    | 94+             |

## How It Works

RTC Mode uses the SDK in **Host Mode** internally. The `AvatarPlayer` acts as a bridge:

1. Initializes the avatar SDK with `DrivingServiceMode.host`
2. Connects to the RTC server via the chosen provider
3. Publishes your microphone audio to the RTC server
4. Receives animation and audio data from the RTC server
5. Feeds **animation data** into the avatar SDK for rendering; **audio is played through the native WebRTC audio track** by the RTC provider

You don't need to call `yieldAudioData()` or `yieldFramesData()` manually — the adapter handles this.

### What RTC Mode Does NOT Use

Although RTC Mode builds on Host Mode internally, the following SDK/Host Mode features are **not used** in RTC Mode:

| Feature                                  | Used in SDK/Host Mode                        | Used in RTC Mode                             |
| ---------------------------------------- | -------------------------------------------- | -------------------------------------------- |
| `initializeAudioContext()`               | Yes — required for Web Audio API playback    | **No** — audio is played via WebRTC tracks   |
| Internal audio player                    | Yes — SDK decodes and plays audio internally | **No** — RTC provider handles audio playback |
| `yieldAudioData()` / `yieldFramesData()` | Yes — you call these manually                | **No** — the adapter calls them internally   |
| `start()` / `send()` / `close()`         | Yes (SDK Mode only)                          | **No**                                       |

<Warning>
  **Audio path difference:** In SDK Mode and Host Mode, the SDK plays avatar audio through its internal audio player (Web Audio API). In RTC Mode, avatar audio arrives as a **native WebRTC audio track** and is played by the browser's WebRTC stack directly — the SDK's internal audio player is not involved. This distinction matters for audio processing (e.g., echo cancellation, noise suppression).
</Warning>

## Server-Side Setup

Your backend is responsible for sending audio to the avatar service and having the resulting avatar stream published to your RTC room. Two approaches are available:

| Platform    | Framework plugin                                                                                                                               | Server SDK + egress                                                                                           |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **LiveKit** | [LiveKit Agents plugin](https://github.com/spatialwalk/livekit-plugins-spatialreal) — hooks into your agent pipeline and publishes to the room | [LiveKit Server](/guide/rtc-livekit-server) (section 2) — use AvatarKit Server SDK with LiveKit egress config |

* **Framework plugin**: Best if you already use LiveKit Agents. The plugin handles audio → SpatialReal → RTC publish for you.
* **Server SDK + egress**: Use the [Golang](/server/go-sdk/go-sdk) or [Python](/server/python-sdk/python-sdk) Server SDK, create a session with LiveKit egress config, and send audio. The avatar service publishes audio and animation directly to the RTC room; your server does not relay that data.

Client-side setup is the same either way: the client joins the room and uses `@spatialwalk/avatarkit-rtc` to render the avatar. See the client guides below.

## Get Started

**Client (browser)**

<CardGroup>
  <Card title="LiveKit Client" icon="bolt" href="/guide/rtc-livekit-client">
    Connect with LiveKit RTC
  </Card>
</CardGroup>

**Server (backend)**

<CardGroup>
  <Card title="LiveKit Server" icon="bolt" href="/guide/rtc-livekit-server">
    Plugin or Server SDK with LiveKit egress
  </Card>
</CardGroup>


# SDK Mode
Source: https://docs.spatialreal.ai/guide/sdk-mode

Server-driven avatar animation via WebSocket

## What is SDK Mode?

SDK Mode is the **default** integration mode. The SDK handles all network communication with AvatarKit servers automatically — you just send audio data, and the SDK takes care of fetching animation data, synchronizing playback, and rendering.

```mermaid theme={null}
flowchart LR
    A["🎤 Audio Input<br/>(PCM16 mono)"] -->|send| B["AvatarKit SDK"]
    B -->|WebSocket| C["AvatarKit Server"]
    C -->|Animation Data| B
    B --> D["🖥️ Avatar Rendering"]
```

## When to Use

* **Real-time audio input** — microphone or audio file playback driving an avatar
* **Simplest integration** — minimal code, SDK handles networking
* **Server-side processing** — animation data is generated by AvatarKit backend

## Requirements

| Requirement       | Description                                                    |
| ----------------- | -------------------------------------------------------------- |
| **App ID**        | Obtained from [SpatialReal Studio](https://app.spatialreal.ai) |
| **Session Token** | Obtained from your backend server (max 24 hours validity)      |
| **Audio Format**  | PCM16, mono, configurable sample rate (default 16000 Hz)       |

<Note>
  **Authentication Flow:**

  ```
  Your Client → Your Backend → AvatarKit Server → Session Token (24 hours max)
  ```

  The Session Token must be set before calling `start()`. See the platform-specific guides below for details.
</Note>

## Platform Comparison

| Feature          | Web                                        | iOS                           | Android                        |
| ---------------- | ------------------------------------------ | ----------------------------- | ------------------------------ |
| **Package**      | `@spatialwalk/avatarkit`                   | `AvatarKit.xcframework` / SPM | `ai.spatialwalk:avatarkit`     |
| **Rendering**    | WebGL / WebGPU                             | Metal                         | Vulkan                         |
| **UI Framework** | DOM Canvas                                 | UIKit + SwiftUI wrapper       | Android View + Compose wrapper |
| **Audio Init**   | `initializeAudioContext()` in user gesture | Automatic                     | Automatic                      |
| **Build Config** | Vite plugin / Next.js wrapper required     | Xcode linker flags            | Gradle dependency              |

## Key Concepts

### Fallback Mechanism

If the WebSocket connection fails within 15 seconds, the SDK automatically enters **audio-only fallback mode** — audio continues to play normally without animation. This ensures uninterrupted audio playback even when the server is unreachable.

### ConversationId

Every `send()` call returns a `conversationId` that identifies the current conversation round. When `end: true` is passed, it marks the end of audio input — the avatar will continue playing remaining animation until finished, then automatically return to idle (notified via `onConversationState`). Sending new audio after that starts a new round and interrupts any ongoing playback.

## Get Started

<CardGroup>
  <Card title="Web" icon="globe" href="/guide/sdk-mode-web">
    JavaScript / TypeScript
  </Card>

  <Card title="iOS Demo" icon="apple" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    Use the GitHub demo as the iOS guide
  </Card>

  <Card title="Android Demo" icon="android" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    Use the GitHub demo as the Android guide
  </Card>
</CardGroup>


# SDK Mode
Source: https://docs.spatialreal.ai/guide/sdk-mode-ios&android

Server-driven avatar animation via WebSocket for iOS and Android applications

Use the official demo repositories as the primary integration guides.

<CardGroup>
  <Card title="iOS Demo" icon="apple" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    Swift / SwiftUI reference implementation.
  </Card>

  <Card title="Android Demo" icon="android" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    Kotlin / Compose reference implementation.
  </Card>
</CardGroup>

## SDK Package Setup

<Tabs>
  <Tab title="iOS">
    <Steps>
      <Step title="Download Framework">
        Download the latest [AvatarKit.xcframework](https://github.com/spatialwalk/avatar-kit-ios-release/releases/download/v1.0.0-beta.28/AvatarKit_202604142117.zip).
      </Step>

      <Step title="Add to Project">
        Unzip and drag `AvatarKit.xcframework` into your Xcode project. In your target's **General** tab, ensure it appears under **Frameworks, Libraries, and Embedded Content** with **Embed & Sign**.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Android">
    Add the dependency to your `build.gradle.kts`:

    ```kotlin title="build.gradle.kts" theme={null}
    dependencies {
        implementation("ai.spatialwalk:avatarkit:1.0.0-beta49")
    }
    ```
  </Tab>
</Tabs>

Need full API details? Use the SDK references below.

<CardGroup>
  <Card title="iOS SDK Reference" icon="book" href="/sdk-reference/ios-sdk/api-reference">
    Complete iOS API documentation.
  </Card>

  <Card title="Android SDK Reference" icon="book" href="/sdk-reference/android-sdk/api-reference">
    Complete Android API documentation.
  </Card>
</CardGroup>


# SDK Mode
Source: https://docs.spatialreal.ai/guide/sdk-mode-web

Server-driven avatar animation via WebSocket for Web applications

```mermaid theme={null}
flowchart LR
    A["🎤 Audio Input"] -->|send| B["AvatarKit SDK<br/>(Web)"]
    B -->|WebSocket| C["AvatarKit Server"]
    C -->|Animation Data| B
    B --> D["🖥️ WebGL / WebGPU"]
```

<CardGroup>
  <Card title="Web Demo Repository" icon="github" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo">
    Complete working examples for Vanilla JS, Vue, and React — the fastest way to get started.
  </Card>
</CardGroup>

## Installation

<Tabs>
  <Tab title="pnpm">
    ```bash theme={null}
    pnpm add @spatialwalk/avatarkit
    ```
  </Tab>

  <Tab title="npm">
    ```bash theme={null}
    npm install @spatialwalk/avatarkit
    ```
  </Tab>

  <Tab title="yarn">
    ```bash theme={null}
    yarn add @spatialwalk/avatarkit
    ```
  </Tab>
</Tabs>

## Build Tool Configuration

<Warning>
  **Required:** The SDK uses WASM files that need special build configuration. You **must** configure your build tool before using the SDK.
</Warning>

<Tabs>
  <Tab title="Vite">
    Add the official plugin to `vite.config.ts`:

    ```typescript title="vite.config.ts" theme={null}
    import { defineConfig } from 'vite'
    import { avatarkitVitePlugin } from '@spatialwalk/avatarkit/vite'

    export default defineConfig({
      plugins: [
        avatarkitVitePlugin(),
      ],
    })
    ```

    <Accordion title="Plugin Features">
      The plugin automatically handles:

      * **Development Server**: Sets correct MIME type for WASM files
      * **Build Time**: Copies WASM files to `dist/assets/`
      * **Cloudflare Pages**: Generates `_headers` file
      * **Vite Configuration**: Configures `optimizeDeps`, `assetsInclude`, etc.
    </Accordion>

    <Accordion title="Manual Configuration (Without Plugin)">
      ```typescript title="vite.config.ts" theme={null}
      import { defineConfig } from 'vite'

      export default defineConfig({
        optimizeDeps: {
          exclude: ['@spatialwalk/avatarkit'],
        },
        assetsInclude: ['**/*.wasm'],
        build: {
          assetsInlineLimit: 0,
          rollupOptions: {
            output: {
              assetFileNames: (assetInfo) => {
                if (assetInfo.name?.endsWith('.wasm')) {
                  return 'assets/[name][extname]'
                }
                return 'assets/[name]-[hash][extname]'
              },
            },
          },
        },
        configureServer(server) {
          server.middlewares.use((req, res, next) => {
            if (req.url?.endsWith('.wasm')) {
              res.setHeader('Content-Type', 'application/wasm')
            }
            next()
          })
        },
      })
      ```
    </Accordion>
  </Tab>

  <Tab title="Next.js">
    Wrap your config in `next.config.mjs`:

    ```javascript title="next.config.mjs" theme={null}
    import { withAvatarkit } from '@spatialwalk/avatarkit/next'

    export default withAvatarkit({
      // ...your existing Next.js config
    })
    ```

    <Accordion title="Wrapper Features">
      The wrapper automatically handles:

      * **Emscripten Fix**: Patches `scriptDirectory` so the WASM glue file correctly resolves assets at `/_next/static/chunks/`
      * **WASM Copying**: Copies `.wasm` files into `static/chunks/` via a custom webpack plugin (client build only)
      * **Content-Type Headers**: Adds `application/wasm` response header for `/_next/static/chunks/*.wasm`
      * **Config Chaining**: Preserves your existing `webpack` and `headers` configurations
    </Accordion>

    <Accordion title="Multiple Config Wrappers">
      If you have multiple config wrappers, `withAvatarkit` must wrap your entire config:

      ```javascript title="next.config.mjs" theme={null}
      import { withAvatarkit } from '@spatialwalk/avatarkit/next'
      import withOtherPlugin from 'other-plugin'

      export default withAvatarkit(withOtherPlugin({
        // ...your config
      }))
      ```
    </Accordion>
  </Tab>
</Tabs>

## Authentication

SDK Mode requires an **App ID** and a **Session Token**.

| Credential        | How to Obtain                                                 | Notes                  |
| ----------------- | ------------------------------------------------------------- | ---------------------- |
| **App ID**        | [SpatialReal Studio](https://app.spatialreal.ai) → Create App | Required for all modes |
| **Session Token** | Your backend server requests it from AvatarKit Server         | Max 24 hours validity  |

<Note>
  **Authentication Flow:**

  ```
  Your Client → Your Backend → AvatarKit Server → Session Token (24 hours max)
  ```

  The Session Token must be set before calling `start()`. Never expose your token generation logic in client-side code.
</Note>

## Quick Start

<Steps>
  <Step title="Initialize SDK">
    ```typescript theme={null}
    import {
      AvatarSDK,
      Environment,
      DrivingServiceMode,
    } from '@spatialwalk/avatarkit'

    await AvatarSDK.initialize('your-app-id', {
      environment: Environment.intl,  // or Environment.cn
      drivingServiceMode: DrivingServiceMode.sdk,  // Default
    })

    // Set session token (required before start())
    AvatarSDK.setSessionToken('your-session-token')
    ```
  </Step>

  <Step title="Load Avatar">
    ```typescript theme={null}
    import { AvatarManager } from '@spatialwalk/avatarkit'

    const avatar = await AvatarManager.shared.load('avatar-id', (progress) => {
      console.log(`Loading: ${progress.progress}%`)
    })
    ```
  </Step>

  <Step title="Create View">
    ```typescript theme={null}
    import { AvatarView } from '@spatialwalk/avatarkit'

    // Container MUST have non-zero width and height
    const container = document.getElementById('avatar-container')!
    const avatarView = new AvatarView(avatar, container)
    ```
  </Step>

  <Step title="Initialize Audio Context">
    <Warning>
      **Critical:** `initializeAudioContext()` **must** be called inside a user gesture handler (e.g., `click`, `touchstart`). This is a browser security requirement — calling it outside a user gesture will fail silently.
    </Warning>

    ```typescript theme={null}
    button.addEventListener('click', async () => {
      await avatarView.controller.initializeAudioContext()
    })
    ```
  </Step>

  <Step title="Connect and Send Audio">
    ```typescript theme={null}
    // Start WebSocket connection to AvatarKit server
    // Note: start() initiates the connection asynchronously.
    // Wait for onConnectionState === 'connected' before calling send().
    await avatarView.controller.start()

    // Wait for connection to be ready
    await new Promise<void>((resolve, reject) => {
      avatarView.controller.onConnectionState = (state) => {
        if (state === ConnectionState.connected) resolve()
        else if (state === ConnectionState.failed) reject(new Error('Connection failed'))
      }
    })

    // Send audio data (PCM16, mono, matching configured sample rate)
    const audioData: ArrayBuffer = /* your PCM16 audio data */
    avatarView.controller.send(audioData, false)  // Continue sending

    // Mark end of audio input for current conversation round
    // The avatar will continue playing remaining animation until finished,
    // then automatically return to idle (notified via onConversationState).
    avatarView.controller.send(lastChunk, true)
    ```
  </Step>

  <Step title="Cleanup">
    ```typescript theme={null}
    avatarView.controller.close()  // Close WebSocket connection
    avatarView.dispose()           // Release all resources
    ```
  </Step>
</Steps>

## Core API

### AvatarSDK

SDK initialization and global configuration.

```typescript theme={null}
// Initialize
await AvatarSDK.initialize(appId: string, configuration: Configuration)

// Properties (read-only)
AvatarSDK.isInitialized   // boolean
AvatarSDK.appId           // string
AvatarSDK.configuration   // Configuration
AvatarSDK.version         // string
AvatarSDK.sessionToken    // string

// Methods
AvatarSDK.setSessionToken(token: string)  // Set auth token
AvatarSDK.setUserId(userId: string)       // Set user ID (for telemetry)
AvatarSDK.cleanup()                       // Release all SDK resources
```

<Note>
  `setSessionToken()` can be called before or after `initialize()`. If called before, the token is applied automatically during initialization.
</Note>

### AvatarManager

Avatar resource loading and caching. Access via the singleton `AvatarManager.shared`.

```typescript theme={null}
const manager = AvatarManager.shared

// Load avatar (downloads and caches)
const avatar = await manager.load(
  id: string,
  onProgress?: (progress: LoadProgressInfo) => void
)

// Clear all cached resources
manager.clearAll()
```

| Parameter    | Type                 | Description                                           |
| ------------ | -------------------- | ----------------------------------------------------- |
| `id`         | `string`             | Avatar character ID                                   |
| `onProgress` | `(progress) => void` | Progress callback with `{ progress: number }` (0–100) |

### AvatarView

3D rendering view. Automatically creates a Canvas element and an `AvatarController`.

```typescript theme={null}
// Create view — Canvas is added to container automatically
const avatarView = new AvatarView(avatar: Avatar, container: HTMLElement)
```

| Property / Method  | Type               | Description                       |
| ------------------ | ------------------ | --------------------------------- |
| `controller`       | `AvatarController` | Playback controller (read-only)   |
| `transform`        | `{ x, y, scale }`  | Avatar position and scale         |
| `onFirstRendering` | `() => void`       | Callback when first frame renders |
| `dispose()`        | `void`             | Release all resources             |

**Transform coordinates:**

| Field   | Range   | Description                                          |
| ------- | ------- | ---------------------------------------------------- |
| `x`     | -1 to 1 | Horizontal offset (-1 = left, 0 = center, 1 = right) |
| `y`     | -1 to 1 | Vertical offset (-1 = bottom, 0 = center, 1 = top)   |
| `scale` | > 0     | Scale factor (1.0 = original size)                   |

<Warning>
  **Container requirement:** The container element **must** have non-zero `width` and `height`. The Canvas fills the container and auto-resizes via `ResizeObserver`.
</Warning>

### AvatarController — SDK Mode Methods

These methods are only available when `drivingServiceMode` is `DrivingServiceMode.sdk`.

```typescript theme={null}
// Initialize audio (MUST be in user gesture handler)
await controller.initializeAudioContext()

// Connect to AvatarKit server
await controller.start()

// Send audio data — returns conversationId
const conversationId = controller.send(
  audioData: ArrayBuffer,  // PCM16, mono
  end: boolean             // true = end of conversation round
)

// Close connection
controller.close()
```

**`send()` behavior:**

* `end: false` — continues the current conversation round
* `end: true` — marks the end of audio input for the current round. The avatar will continue playing remaining animation until finished, then automatically return to idle (notified via `onConversationState`). Sending new audio after this starts a new round and interrupts any ongoing playback

### AvatarController — Common Methods

Available in both SDK Mode and Host Mode.

```typescript theme={null}
// Playback control
controller.pause()       // Pause audio + animation
controller.resume()      // Resume playback
controller.interrupt()   // Stop current playback, clear data

// Data management
controller.clear()       // Clear all data and resources

// Conversation
controller.getCurrentConversationId()  // string | null

// Volume (affects avatar audio only, not system volume)
controller.volume = 0.5   // Set volume (0.0 to 1.0)
controller.volume          // Get current volume
```

### AvatarController — Event Callbacks

```typescript theme={null}
// Connection state changes (SDK Mode only)
controller.onConnectionState = (state: ConnectionState) => {
  console.log('Connection:', state)
}

// Conversation state changes
controller.onConversationState = (state: ConversationState) => {
  console.log('Conversation:', state)
}

// Error handler
controller.onError = (error: AvatarError) => {
  console.error('Error:', error.code, error.message)
}
```

## Audio Format

The SDK requires audio in **mono PCM16** format:

| Property        | Value                                                                            |
| --------------- | -------------------------------------------------------------------------------- |
| **Format**      | PCM16 (16-bit signed integer, little-endian)                                     |
| **Channels**    | Mono (1 channel)                                                                 |
| **Sample Rate** | Configurable: 8000, 16000, 22050, 24000, 32000, 44100, 48000 Hz (default: 16000) |
| **Data Type**   | `ArrayBuffer` or `Uint8Array`                                                    |

**Data size:** 1 second at 16 kHz = 16,000 samples × 2 bytes = 32,000 bytes.

<Accordion title="Converting MP3 to PCM16">
  ```typescript theme={null}
  async function mp3ToPcm16(mp3File: File, targetSampleRate: number): Promise<ArrayBuffer> {
    const arrayBuffer = await mp3File.arrayBuffer()
    const audioContext = new AudioContext({ sampleRate: targetSampleRate })
    const audioBuffer = await audioContext.decodeAudioData(arrayBuffer.slice(0))

    const length = audioBuffer.length
    const channels = audioBuffer.numberOfChannels
    const pcm16Buffer = new ArrayBuffer(length * 2)
    const pcm16View = new DataView(pcm16Buffer)

    // Mix to mono if stereo
    const mono = channels === 1
      ? audioBuffer.getChannelData(0)
      : (() => {
          const mixed = new Float32Array(length)
          const left = audioBuffer.getChannelData(0)
          const right = audioBuffer.getChannelData(1)
          for (let i = 0; i < length; i++) mixed[i] = (left[i] + right[i]) / 2
          return mixed
        })()

    // Float32 → Int16
    for (let i = 0; i < length; i++) {
      const s = Math.max(-1, Math.min(1, mono[i]))
      pcm16View.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7FFF, true)
    }

    audioContext.close()
    return pcm16Buffer
  }
  ```
</Accordion>

## Configuration

### Configuration Interface

```typescript theme={null}
interface Configuration {
  environment: Environment
  drivingServiceMode?: DrivingServiceMode  // Default: DrivingServiceMode.sdk
  logLevel?: LogLevel                      // Default: LogLevel.off
  audioFormat?: AudioFormat                // Default: { channelCount: 1, sampleRate: 16000 }
}
```

### Environment

```typescript theme={null}
enum Environment {
  cn = 'cn',      // China region
  intl = 'intl',  // International region
}
```

### DrivingServiceMode

```typescript theme={null}
enum DrivingServiceMode {
  sdk = 'sdk',    // Server-driven (default)
  host = 'host',  // Client-driven
}
```

### AudioFormat

```typescript theme={null}
interface AudioFormat {
  readonly channelCount: 1   // Fixed to mono
  readonly sampleRate: number // 8000 | 16000 | 22050 | 24000 | 32000 | 44100 | 48000
}
```

### LogLevel

```typescript theme={null}
enum LogLevel {
  off = 'off',          // No logging (default)
  error = 'error',      // Errors only
  warning = 'warning',  // Errors + warnings
  all = 'all',          // All logs
}
```

## State Management

### ConnectionState

Reported via `onConnectionState` callback (SDK Mode only).

```typescript theme={null}
enum ConnectionState {
  disconnected = 'disconnected',
  connecting = 'connecting',
  connected = 'connected',
  failed = 'failed',
}
```

### ConversationState

Reported via `onConversationState` callback.

```typescript theme={null}
enum ConversationState {
  idle = 'idle',        // Breathing animation, waiting for input
  playing = 'playing',  // Active conversation playback
  paused = 'paused',    // Paused during playback
}
```

<Note>
  State transitions are notified immediately when the transition starts, not when the animation completes. For example, `playing` is reported as soon as the transition from `idle` begins.
</Note>

## Error Handling

### AvatarError

```typescript theme={null}
import { AvatarError } from '@spatialwalk/avatarkit'

try {
  await avatarView.controller.start()
} catch (error) {
  if (error instanceof AvatarError) {
    console.error('SDK error:', error.message, error.code)
  }
}
```

### Error Callback

```typescript theme={null}
avatarView.controller.onError = (error: AvatarError) => {
  console.error('Controller error:', error.code, error.message)
}
```

## Lifecycle Management

### Avatar Switching

```typescript theme={null}
// 1. Dispose current view
currentAvatarView.dispose()

// 2. Load new avatar
const newAvatar = await AvatarManager.shared.load('new-avatar-id')

// 3. Create new view (reuse same container)
currentAvatarView = new AvatarView(newAvatar, container)

// 4. Reconnect
await currentAvatarView.controller.initializeAudioContext()
await currentAvatarView.controller.start()
```

### Resource Cleanup

`dispose()` automatically cleans up all resources:

* WebSocket connections
* Audio playback data and animation resources
* Canvas elements and render system
* Event listeners and callbacks

<Warning>
  Always call `dispose()` when the view is no longer needed. Failing to do so may cause memory leaks.
</Warning>

### Fallback Mechanism

If the WebSocket connection fails within 15 seconds, the SDK automatically enters **audio-only fallback mode** — audio continues playing without animation. This ensures uninterrupted audio playback when the server is unreachable.

* The fallback mode is interruptible like normal playback
* `onConnectionState` reports `failed` when the connection times out

## Browser Compatibility

| Browser        | Minimum Version | Rendering          |
| -------------- | --------------- | ------------------ |
| Chrome / Edge  | 90+             | WebGPU (preferred) |
| Firefox        | 90+             | WebGL              |
| Safari         | 14+             | WebGL              |
| iOS Safari     | 14+             | WebGL              |
| Android Chrome | 90+             | WebGL              |

## Common Issues

| Issue                       | Cause                                          | Solution                                                              |
| --------------------------- | ---------------------------------------------- | --------------------------------------------------------------------- |
| Audio not working           | `initializeAudioContext()` not in user gesture | Call it inside a `click` or `touchstart` handler                      |
| Avatar not rendering        | Container has zero dimensions                  | Set explicit `width` and `height` on the container                    |
| WASM MIME type error        | Build tool misconfiguration                    | Use Vite plugin or Next.js wrapper                                    |
| Session token invalid       | Token expired or not set                       | Refresh token from backend, call `setSessionToken()` before `start()` |
| WebSocket connection failed | Network or auth issue                          | Check network connectivity and token validity                         |


# Speech-to-Avatar Walkthrough
Source: https://docs.spatialreal.ai/guide/speech-to-avatar-walkthrough

Step-by-step code walkthrough for the Speech-to-Avatar quickstart

This page explains the quickstart in steps and keeps each section copy-friendly.

<Card title="Run Quickstart First" icon="rocket" href="/overview/speech-to-avatar">
  Use the clone-and-run flow first, then come back here for code details.
</Card>

<Steps>
  <Step title="Configure environment variables">
    Set the required app/avatar/token values used by the frontend runtime.

    <div>
      ```bash title=".env" theme={null}
      VITE_SPATIALREAL_AVATAR_ID=6aed28f9-674c-4ffb-89ee-b447b28aa3ed # https://app.spatialreal.ai/avatars/library
      VITE_SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
      VITE_SPATIALREAL_SESSION_TOKEN=your_temporary_session_token # https://app.spatialreal.ai/apps -> details -> Generate Temporary Token
      ```
    </div>
  </Step>

  <Step title="Enable AvatarKit Vite plugin">
    The plugin is required so AvatarKit WASM assets are correctly handled in dev/build.

    <div>
      ```ts title="vite.config.ts" theme={null}
      import { defineConfig } from 'vite'
      import vue from '@vitejs/plugin-vue'
      import { avatarkitVitePlugin } from '@spatialwalk/avatarkit/vite'

      export default defineConfig({
        plugins: [vue(), avatarkitVitePlugin()],
        server: { port: 3000 },
      })
      ```
    </div>
  </Step>

  <Step title="Implement connect + send flow in App.vue">
    This file handles SDK init, avatar loading, connection, and one-shot PCM send.

    <div>
      ```vue title="src/App.vue" theme={null}
      <script setup lang="ts">
      import { nextTick, onBeforeUnmount, ref } from 'vue'
      import {
        AvatarManager,
        AvatarSDK,
        AvatarView,
        DrivingServiceMode,
        Environment,
      } from '@spatialwalk/avatarkit'

      const container = ref<HTMLDivElement | null>(null)
      const status = ref('Click Connect Avatar to start')
      const connecting = ref(false)
      const sending = ref(false)
      const connected = ref(false)

      let avatarView: AvatarView | null = null

      const appId = import.meta.env.VITE_SPATIALREAL_APP_ID
      const avatarId = import.meta.env.VITE_SPATIALREAL_AVATAR_ID
      const sessionToken = import.meta.env.VITE_SPATIALREAL_SESSION_TOKEN

      if (!sessionToken) throw new Error('Missing VITE_SPATIALREAL_SESSION_TOKEN')

      async function downloadPcmAudio(url: string): Promise<ArrayBuffer> {
        const response = await fetch(url)
        if (!response.ok) throw new Error('Failed to load demo audio URL')
        return response.arrayBuffer()
      }

      async function disposeAvatar(): Promise<void> {
        connected.value = false
        avatarView?.controller.close()
        avatarView?.dispose()
        avatarView = null
      }

      async function connectAvatar(): Promise<void> {
        if (connecting.value || connected.value) return
        connecting.value = true

        try {
          status.value = 'Using session token from .env...'

          if (!AvatarSDK.isInitialized) {
            await AvatarSDK.initialize(appId, {
              environment: Environment.intl,
              drivingServiceMode: DrivingServiceMode.sdk,
            })
          }
          AvatarSDK.setSessionToken(sessionToken)

          await nextTick()
          const mountEl = container.value
          if (!mountEl) throw new Error('Avatar container is not ready')

          if (!avatarView) {
            status.value = 'Loading avatar...'
            const avatar = await AvatarManager.shared.load(avatarId)
            avatarView = new AvatarView(avatar, mountEl)
            avatarView.controller.onConnectionState = (state) => {
              connected.value = state === 'connected'
            }
          }

          status.value = 'Connecting to SpatialReal...'
          await avatarView.controller.initializeAudioContext()
          await avatarView.controller.start()
          await new Promise((resolve) => setTimeout(resolve, 300))
          if (!connected.value) throw new Error('Failed to connect to animation channel')

          status.value = 'Avatar connected. Click Send Audio.'
        } catch (error) {
          status.value = error instanceof Error ? error.message : 'Failed to run demo'
        } finally {
          connecting.value = false
        }
      }

      async function sendAudio(): Promise<void> {
        if (sending.value) return
        if (!connected.value || !avatarView) {
          status.value = 'Please click Connect Avatar first.'
          return
        }

        sending.value = true

        try {
          status.value = 'Downloading demo PCM audio...'
          const audioData = await downloadPcmAudio('https://cdn.spatialwalk.cloud/public/website/quickstart_voice.pcm')

          status.value = 'Sending audio...'
          avatarView.controller.send(audioData, true)
          status.value = `Audio sent (${audioData.byteLength} bytes)`
        } catch (error) {
          status.value = error instanceof Error ? error.message : 'Failed to send audio'
        } finally {
          sending.value = false
        }
      }

      onBeforeUnmount(async () => {
        await disposeAvatar()
      })
      </script>

      <template>
        <div style="min-height:100vh; display:flex; align-items:center; justify-content:center; padding:16px;">
          <div style="width:min(720px, 100%); display:flex; flex-direction:column; gap:10px;">
            <div ref="container" style="width:100%; aspect-ratio:16/10; min-height:320px; border-radius:12px; overflow:hidden; border:1px solid;" />

            <div style="display:flex; gap:8px; flex-wrap:wrap;">
              <button :disabled="connecting || connected" @click="connectAvatar">
                {{ connecting ? 'Connecting...' : connected ? 'Avatar Connected' : 'Connect Avatar' }}
              </button>
              <button :disabled="sending || !connected" @click="sendAudio">
                {{ sending ? 'Sending...' : 'Send Audio' }}
              </button>
            </div>

            <div style="font-size:14px;">{{ status }}</div>
          </div>
        </div>
      </template>
      ```
    </div>
  </Step>

  <Step title="Understand runtime sequence">
    Use this as a mental model when debugging connection or audio send issues.

    <div>
      ```text theme={null}
      initialize SDK -> set session token -> load avatar -> create AvatarView -> start controller
      -> fetch pcm -> controller.send(audio, true)
      ```
    </div>
  </Step>
</Steps>


# Agent
Source: https://docs.spatialreal.ai/overview/agent

Run the full LiveKit + SpatialReal agent demo directly from the official repository

Use this quickstart to run the **LiveKit Agent + SpatialReal Avatar** demo quickly.
This page focuses on clone-and-run. The frontend is built with [AvatarKit UI](/sdk-reference/web-sdk/avatarkit-ui), and code-level explanation is in a separate Guide page.

<CardGroup>
  <Card title="GitHub Repository" icon="github" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/spatialreal-agent-quickstart">
    Source code for this quickstart.
  </Card>

  <Card title="Code Walkthrough" icon="book-open" href="/guide/agent-quickstart-walkthrough">
    Backend and frontend architecture walkthrough.
  </Card>
</CardGroup>

## How LiveKit + SpatialReal Work Together

<div>
  ```mermaid actions={false} theme={null}
  ---
  config:
    "look": "handDrawn"
    "theme": "base"
    "themeVariables":
      "background": "#ffffff"
      "textColor": "#111827"
      "lineColor": "#000000"
      "primaryColor": "#e8f4fd"
      "primaryTextColor": "#111827"
      "primaryBorderColor": "#2196F3"
      "secondaryColor": "#f3e5f5"
      "secondaryTextColor": "#000000"
      "secondaryBorderColor": "#9C27B0"
      "tertiaryColor": "#fff3e0"
      "tertiaryTextColor": "#111827"
      "tertiaryBorderColor": "#FF9800"
      "clusterBkg": "#f8fafc"
      "clusterBorder": "#cbd5e1"
      "edgeLabelBackground": "#ffffff"
      "actorBkg": "#e8f4fd"
      "actorBorder": "#2196F3"
      "actorTextColor": "#111827"
      "noteBkgColor": "#fff7ed"
      "noteTextColor": "#111827"
      "signalColor": "#64748b"
      "signalTextColor": "#111827"
  ---
  sequenceDiagram
      participant FE as Frontend (React)
      participant TS as token_server.py
      participant LK as LiveKit
      participant AW as agent.py worker
      participant GM as Gemini Live
      participant SR as SpatialReal Avatar Service

      Note over AW,LK: agent.py is already running & <br/> registered before any room dispatch
      AW->>LK: Start worker and wait for jobs

      FE->>TS: POST /token { room }
      TS->>LK: create room <br/> dispatch agent
      TS-->>FE: { token, url, room, identity }

      FE->>LK: Join room (client token)
      LK->>AW: Dispatch job to running worker
      AW-->>LK: Join room (worker)

      AW->>GM: User audio
      GM-->>AW: AI response audio
      AW->>SR: Forward response audio
      SR-->>LK: Publish avatar audio + animation stream
      LK-->>FE: Deliver room tracks/data
      FE->>FE: SpatialRealAvatarProvider renders avatar
  ```
</div>

In this architecture, each system has a clear responsibility:

* **LiveKit** handles room lifecycle, client/worker connectivity, microphone transport, and agent dispatch.
* **SpatialReal** takes agent response audio, generates lip-sync + avatar animation, and publishes avatar output back into the room.
* **Frontend** only needs to join LiveKit and render with [AvatarKit UI](/sdk-reference/web-sdk/avatarkit-ui)

## Prerequisites

* Node.js 18+
* Python 3.10+
* `pnpm`
* `uv`
* SpatialReal credentials (`SPATIALREAL_API_KEY`, `SPATIALREAL_APP_ID`, `SPATIALREAL_AVATAR_ID`)
* LiveKit credentials (`LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`)
* Gemini key (`GOOGLE_API_KEY`)

<Steps>
  <Step title="Clone the repo and enter the quickstart folder">
    ```bash theme={null}
    git clone https://github.com/spatialwalk/avatarkit-voice-agent-demo.git
    cd avatarkit-voice-agent-demo/spatialreal-agent-quickstart
    ```
  </Step>

  <Step title="Create env files">
    ```bash theme={null}
    cp backend/.env.example backend/.env
    cp frontend/.env.example frontend/.env
    ```

    Fill `backend/.env`:

    ```bash title="backend/.env" theme={null}
    LIVEKIT_URL=wss://your-project.livekit.cloud # https://cloud.livekit.io
    LIVEKIT_API_KEY=your_api_key # https://cloud.livekit.io
    LIVEKIT_API_SECRET=your_api_secret # https://cloud.livekit.io

    GOOGLE_API_KEY=your_google_api_key # https://aistudio.google.com/api-keys
    E2E_GOOGLE_MODEL=gemini-2.5-flash-native-audio-preview-12-2025
    E2E_GOOGLE_VOICE=Puck

    SPATIALREAL_API_KEY=your_api_key # https://app.spatialreal.ai/apps
    SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
    SPATIALREAL_AVATAR_ID=6aed28f9-674c-4ffb-89ee-b447b28aa3ed # https://app.spatialreal.ai/avatars/library
    ```

    Fill `frontend/.env`:

    ```bash title="frontend/.env" theme={null}
    VITE_SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
    VITE_SPATIALREAL_AVATAR_ID=6aed28f9-674c-4ffb-89ee-b447b28aa3ed # https://app.spatialreal.ai/avatars/library
    VITE_TOKEN_ENDPOINT=http://localhost:8080/token
    VITE_ROOM_NAME=voice-agent-room
    ```
  </Step>

  <Step title="Install dependencies">
    ```bash theme={null}
    # backend
    cd backend
    uv sync

    # frontend
    cd ../frontend
    pnpm install
    ```
  </Step>

  <Step title="Run the stack (3 terminals)">
    ```bash theme={null}
    # Terminal 1: token server
    cd avatarkit-voice-agent-demo/spatialreal-agent-quickstart/backend
    uv run token_server.py
    ```

    ```bash theme={null}
    # Terminal 2: LiveKit agent worker
    cd avatarkit-voice-agent-demo/spatialreal-agent-quickstart/backend
    uv run agent.py dev
    ```

    ```bash theme={null}
    # Terminal 3: frontend
    cd avatarkit-voice-agent-demo/spatialreal-agent-quickstart/frontend
    pnpm dev
    ```

    Open `http://localhost:3000`, click **Connect**, then **Enable Mic** to start speaking.
  </Step>
</Steps>

## Next

* Want to understand the backend dispatch flow and AvatarKit UI frontend logic? See [Agent Quickstart Walkthrough](/guide/agent-quickstart-walkthrough).


# Changelog
Source: https://docs.spatialreal.ai/overview/changelog



<Update label="1.0.0-beta.80" description="2025-01">
  **Next.js Plugin** - Added `withAvatarkit` wrapper for Next.js projects to automate WASM file configuration with webpack

  * Patches Emscripten `scriptDirectory` for correct asset resolution
  * Copies WASM files to `static/chunks/` automatically
  * Adds `application/wasm` Content-Type headers
  * Preserves existing webpack and headers configurations
</Update>

<Update label="1.0.0-beta.75" description="2025-01">
  **Local Log Upload** - Automatic local log upload for debugging and troubleshooting
</Update>

<Update label="1.0.0-beta.71" description="2024-12">
  **API Refinement** - Improved API structure by marking internal classes with `@internal` for better code organization

  **Type Safety** - Enhanced type definitions by hiding internal implementation details from public API
</Update>

<Update label="1.0.0-beta.64" description="2024-12">
  **Vite Plugin** - Added `avatarkitVitePlugin` to automate WASM file configuration

  * Automatically handles WASM MIME types in dev server
  * Copies WASM files during build
  * Generates `_headers` file for Cloudflare Pages
</Update>

<Update label="1.0.0-beta.63" description="2024-12">
  **Audio Context API** - Added `initializeAudioContext()` method that must be called in user gesture context

  * Ensures AudioContext is created properly to satisfy browser security policies
  * All audio operations now require prior initialization
</Update>

<Update label="1.0.0-beta.55" description="2024-11">
  **Eye Tracking Control** - Added `eyeTrackingEnabled` parameter to `PostProcessingConfig` for real-time control

  **Point Count API** - Added `getPointCount()` method to `AvatarController` to get avatar point cloud count
</Update>

<Update label="1.0.0-beta.51" description="2024-11">
  **Post-Processing Enhancement** - Added `translation` and `eyePose` fields to `PostProcessingConfig`

  **Bugfix** - Fixed avatar settings update issue when using cached avatars
</Update>

<Update label="1.0.0-beta.45" description="2024-10">
  **Immediate Rendering** - Post-processing and camera updates apply immediately when playback is paused

  **API Change** - Replaced `ready` Promise with `onFirstRendering` callback for better async handling
</Update>

<Update label="1.0.0-beta.36" description="2024-10">
  **Audio Configuration** - Added configurable audio sample rate support (8000-48000 Hz, default: 16000)

  * Audio recording and playback automatically use configured sample rate
</Update>

<Update label="1.0.0-beta.35" description="2024-10">
  **API Change** - Changed `setTransform()` method to `transform` getter/setter property
</Update>

<Update label="1.0.0-beta.29" description="2024-09">
  **Breaking Changes** - Removed `Environment.test`, changed default log level to `off`

  **New Features** - Added avatar caching methods (`retrieve`, `clear`, `clearAll`)

  **Background Image** - Added background image control in `AvatarView`
</Update>

<Update label="1.0.0-beta.28" description="2024-09">
  **Breaking Change** - Renamed `AvatarKit` to `AvatarSDK` for better consistency

  * Update imports: `AvatarKit` → `AvatarSDK`
</Update>

<Update label="1.0.0-beta.22" description="2024-08">
  **API Changes** - Renamed `onAvatarState` to `onConversationState`, `Environment.us` to `Environment.intl`

  **Volume Control** - Added `setVolume()` and `getVolume()` methods for audio playback control
</Update>

<Update label="1.0.0-beta.17" description="2024-08">
  **Fallback Mechanism** - Added automatic audio-only fallback when animation data is unavailable

  **API Change** - Moved playback mode configuration to `AvatarSDK.initialize()`
</Update>

<Update label="1.0.0-beta.3" description="2024-07">
  **Architecture** - Refactored to three-layer architecture (Rendering, Playback, Network)

  **Dual Mode** - Added SDK mode and Host mode support
</Update>

<Update label="1.0.0-beta.97" description="2026-04">
  **Breaking Change** — `avatarView.transform` renamed to `avatarView.avatarTransform` (cross-SDK alignment).

  **Avatar transform from character settings** — Automatically applies position and scale from character settings.

  **Rendering quality** — Improved color accuracy for avatar rendering.
</Update>

<Update label="1.0.0-beta.28" description="2026-04">
  **Rendering quality** — Improved color accuracy for avatar rendering.

  **Avatar transform from character settings** — Automatically applies position and scale from character settings.

  **WebSocket close codes** — Added handling for auth failure, insufficient balance, session timeout, concurrent limit.
</Update>

<Update label="1.0.0-beta48" description="2026-04">
  **Rendering quality** — Improved color accuracy for avatar rendering.

  **Avatar transform from character settings** — Automatically applies position and scale from character settings.

  **Device support expansion** — More devices supported through benchmark scoring instead of deny-list.

  **WebSocket close codes** — Added handling for auth failure, insufficient balance, session timeout, concurrent limit.
</Update>


# Demo Projects
Source: https://docs.spatialreal.ai/overview/demo-projects

Complete working examples for every platform

Get started quickly with our demo projects. The repository contains fully functional applications for all platforms that you can run immediately.

**Repository:** [github.com/spatialwalk/avatarkit-voice-agent-demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo)

```bash theme={null}
git clone https://github.com/spatialwalk/avatarkit-voice-agent-demo.git
cd avatarkit-voice-agent-demo
```

## What's Included

A collection of voice agent + avatar demo projects covering Web, iOS, and Android:

* End-to-end voice agent + avatar pipeline
* Multiple runnable example projects per platform
* Works out of the box after filling required API keys in `.env`

### Web

| Example | Framework               | Build Tool     |
| ------- | ----------------------- | -------------- |
| Vanilla | Plain TypeScript        | Vite           |
| Vue     | Vue 3 + Composition API | Vite           |
| React   | React 18+               | Vite / Next.js |

### iOS

* Swift 6.2 / SwiftUI
* iOS 16+
* Metal rendering

<Warning>
  Metal rendering is not supported in the iOS Simulator. Use a physical device for testing.
</Warning>

### Android

* Kotlin / Jetpack Compose
* API 24+
* Vulkan rendering


# FAQ
Source: https://docs.spatialreal.ai/overview/faq

Frequently asked questions

### Common Issues

<AccordionGroup>
  <Accordion title="Supported Audio Format">
    Currently only support PCM 16-bit mono audio at multiple sample rates:
    8000, 16000, 22050, 24000, 32000, 44100, 48000.

    See [Audio Sample Rate](/concepts/audio) for configuration details.
  </Accordion>

  <Accordion title="Avatar Playback Fallback Strategy">
    If the avatar animation drive data is abnormal after passing in audio, the fallback playback strategy will be triggered:

    * Audio plays normally
    * Animation plays idle breathing state / Animation stops
  </Accordion>

  <Accordion title="Session token expired or invalid">
    Session tokens have a limited lifetime. If you receive `sessionTokenExpired` or `sessionTokenInvalid`, request a new token from your server and set it via `AvatarSDK.sessionToken` before reconnecting.
  </Accordion>

  <Accordion title="Avatar assets fail to download">
    Check your network connectivity and ensure the avatar ID is correct. If the issue persists, clear the local cache with `AvatarManager.clearAll()` and retry. On mobile, verify that your app has network permissions.
  </Accordion>
</AccordionGroup>

### Platform-Specific

<AccordionGroup>
  <Accordion title="Web: WASM module fails to load">
    AvatarKit Web uses WebAssembly for rendering. Your build tool must be configured to handle `.wasm` files:

    * **Vite**: Use the AvatarKit Vite plugin.
    * **Next.js**: Use the AvatarKit Next.js wrapper component.

    See the [Web SDK guide](/guide/sdk-mode-web) for setup instructions.
  </Accordion>

  <Accordion title="Web: WebGL not supported">
    AvatarKit requires WebGL 2.0. Ensure the browser supports it and hardware acceleration is enabled. Most modern browsers support WebGL 2.0 by default.
  </Accordion>

  <Accordion title="iOS: Metal rendering requirements">
    AvatarKit iOS uses Metal for GPU rendering. Metal is available on all devices running iOS 16+. The simulator does not support Metal — test on a physical device.
  </Accordion>

  <Accordion title="Android: Vulkan rendering requirements">
    AvatarKit Android uses Vulkan for GPU rendering. Ensure the target device supports Vulkan 1.1+ (API 24+). Most devices from 2018 onwards support Vulkan.
  </Accordion>

  <Accordion title="RTC: LiveKit version compatibility">
    RTC Mode requires `livekit-client` at exactly version `2.16.1`. Other versions may cause connection or audio routing issues.
  </Accordion>
</AccordionGroup>

## Performance Issues

<AccordionGroup>
  <Accordion title="Avatar Rendering Resolution Support">
    Our avatar models don't use the concept of resolution.

    We can reconstruct 4K resolution 3D avatar models, but this depends on the model's point count.

    Fewer points result in less detail restoration, but all models can be rendered to the corresponding resolution on the client side.

    Metrics such as PSNR and LPIPS are typically used to measure the quality of detail restoration.
  </Accordion>

  <Accordion title="Can the SDK dynamically switch between different models based on device performance?">
    Not currently supported. Developers need to implement their own strategy for switching between different models based on device performance.
  </Accordion>
</AccordionGroup>


# Get Your API Keys
Source: https://docs.spatialreal.ai/overview/get-apikeys

To integrate with SpatialReal, first create your API Key.

Follow these steps to get your `App ID` and `API Key`.

<Steps>
  <Step title="Open API Keys page">
    Go to [https://app.spatialreal.ai/apps](https://app.spatialreal.ai/apps).
  </Step>

  <Step title="Click Create API Key">
    On the API key page, click **Create API Key**.

    <Frame>
      <img alt="API Keys page" />
    </Frame>
  </Step>

  <Step title="Enter application name">
    Type your **Application Name** and click **Create**.

    <Frame>
      <img alt="Click Create API Key" />
    </Frame>
  </Step>

  <Step title="Copy App ID and API Key">
    In the popup, copy both **App ID** and **API Key**.

    <Frame>
      <img alt="Create app and view popup" />
    </Frame>
  </Step>
</Steps>


# How to integrate
Source: https://docs.spatialreal.ai/overview/integration_overview

Choose an integration mode and start from runnable example projects

<Info>
  SpatialReal currently provides **avatar-only** services, focusing on generating and rendering real-time avatar animations based on audio input. Voice conversation logic, speech synthesis, and other agent functionalities are managed by your application or third-party services.
</Info>

## Compare Integration Modes

| Mode                                                            | Characteristic               | Latency     | Dev Effort | Ideal Scenario                                                                                |
| --------------------------------------------------------------- | ---------------------------- | ----------- | ---------- | --------------------------------------------------------------------------------------------- |
| **[SDK Mode](/concepts/integrations#sdk-mode)**                 | Client-centric               | Moderate ⏱️ | Low 🟢     | Easy client-centric integration.                                                              |
| **[Framework Plugin](/concepts/integrations#framework-plugin)** | Voice agent framework plugin | Ultra-Low ⚡ | Low 🟢     | Use [LiveKit Agents](https://docs.livekit.io/agents/) or [TEN Framework](https://theten.ai/). |
| **[RTC Mode](/concepts/integrations#rtc-mode)**                 | Transport via Agora/LiveKit  | Ultra-Low ⚡ | Medium 🟡  | You want to optimize latency, and use LiveKit or Agora as transport layer                     |
| **[Host Mode](/concepts/integrations#host-mode)**               | Use your own transport layer | Low 💨      | High 🔴    | Full control over data transport.                                                             |

## Start from Runnable Examples

<Card title="AvatarKit Voice Agent Demo" icon="code" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-agents">
  A collection of example projects that can work out of the box after API keys are filled in `.env`.
</Card>

```bash theme={null}
git clone https://github.com/spatialwalk/avatarkit-voice-agent-demo.git
cd avatarkit-voice-agent-demo
```

## Guides

<CardGroup>
  <Card title="SDK Mode Guide" icon="book-open" href="/guide/sdk-mode">
    Start with client-centric integration.
  </Card>

  <Card title="Host Mode Guide" icon="book-open" href="/guide/host-mode">
    Build custom transport and server orchestration.
  </Card>
</CardGroup>

<Card title="RTC Mode & Framework Plugin Guide" icon="book-open" href="/guide/rtc-mode">
  Use LiveKit or Agora for real-time delivery, including LiveKit Agents and TEN Framework integration.
</Card>

## Want to Learn More?

Read the full [Integration Modes](/concepts/integrations) doc to understand the technical details, service architecture, and best practices for each integration mode.


# Welcome to SpatialReal Documentation
Source: https://docs.spatialreal.ai/overview/introduction



SpatialReal is the next-generation AI infrastructure for empathetic human-AI interactions. We build **real-time**, **high-fidelity**, and **cost-effective** digital avatar interactions through on-device rendering technology.

<Frame>
  <img alt="pipeline" />
</Frame>

<p>
  In the pipeline diagram: ASR = Automatic Speech Recognition, LLM = Large Language Model, TTS = Text-to-Speech, S2S = Speech-to-Speech (e.g. Gemini Live).
</p>

## Why SpatialReal?

Traditional digital avatar solutions force painful trade-offs between cost, visual quality, and real-time performance. SpatialReal solves this trilemma:

| Metric            | Traditional Cloud Rendering  | SpatialReal On-Device          |
| ----------------- | ---------------------------- | ------------------------------ |
| **Bandwidth**     | 1-2 MB/s (Video Stream)      | 10-20 KB/s (Driver Data)       |
| **Cost**          | High GPU rental costs        | Ultra-low edge compute         |
| **Latency**       | > 3s                         | \< 1.5s                        |
| **Compatibility** | High-speed network dependent | 99% of Android/iOS/Web devices |

## Use Cases

* **Online Education** - Immersive 1-on-1 tutoring with precise lip-sync for language learning and K12
* **Recruitment & Training** - 7x24h AI interviewers and roleplay training simulations
* **Customer Service** - Brand ambassadors with visual eye contact that builds emotional connection

## Creating an account

### Sign Up

Visit [SpatialReal Studio](https://app.spatialreal.ai/) to create your free account.

### SpatialReal Studio

[SpatialReal Studio](https://app.spatialreal.ai/) is your control center where you can:

* Free access to a large avatar library
* Manage your API keys
* Monitor your usage
* Control your subscription

## Quick Start

<CardGroup>
  <Card title="Use LiveKit(recommended)" icon="play" href="/overview/agent">
    LiveKit is a realtime communication platform that allows you to create voice and video chat applications. Deploying your own LiveKit server is quick and easy.
  </Card>

  <Card title="Use Avatarkit SDKs" icon="key" href="/overview/speech-to-avatar">
    For when you need full control of your tech stack (e.g., custom voice bots).
  </Card>
</CardGroup>

<Card title="Start building in 5 minutes" icon="rocket" href="/overview/integration_overview">
  Compare integration modes and start from runnable example projects.
</Card>

## About Us

Learn more about SpatialReal and our mission at [spatialreal.ai](https://www.spatialreal.ai/).


# Speech-to-Avatar
Source: https://docs.spatialreal.ai/overview/speech-to-avatar

Run the Speech-to-Avatar demo directly from the official repository

Use this quickstart to run a ready-made **Speech-to-Avatar** demo in a few minutes.
This page focuses on running the demo fast. Code-level explanation is in the Guide page.

<CardGroup>
  <Card title="GitHub Repository" icon="github" href="https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/spatialreal-speech-to-avatar-quickstart">
    Source code for this quickstart.
  </Card>

  <Card title="Code Walkthrough" icon="book-open" href="/guide/speech-to-avatar-walkthrough">
    File-by-file explanation of the implementation.
  </Card>
</CardGroup>

## Prerequisites

* Node.js 18+
* `pnpm`
* SpatialReal credentials:
  * `VITE_SPATIALREAL_APP_ID`
  * `VITE_SPATIALREAL_AVATAR_ID`
  * `VITE_SPATIALREAL_SESSION_TOKEN` ([how to generate](/studio/aki-key#temporary-session-token))

<Steps>
  <Step title="Clone the repo and enter the quickstart folder">
    ```bash theme={null}
    git clone https://github.com/spatialwalk/avatarkit-voice-agent-demo.git
    cd avatarkit-voice-agent-demo/spatialreal-speech-to-avatar-quickstart
    ```
  </Step>

  <Step title="Create env file">
    ```bash theme={null}
    cp .env.example .env
    ```

    Update `.env` with your own values:

    ```bash title=".env" theme={null}
    VITE_SPATIALREAL_APP_ID=your_app_id # https://app.spatialreal.ai/apps
    VITE_SPATIALREAL_AVATAR_ID=your_avatar_id # https://app.spatialreal.ai/avatars/library
    VITE_SPATIALREAL_SESSION_TOKEN=your_temporary_session_token # https://app.spatialreal.ai/apps -> details -> Generate Temporary Token
    ```
  </Step>

  <Step title="Install and run">
    ```bash theme={null}
    pnpm install
    pnpm dev
    ```

    Open `http://localhost:3000`, click **Connect Avatar**, then click **Send Audio**.
  </Step>
</Steps>

## Next

* Want to understand the code structure and runtime flow? See [Speech-to-Avatar Walkthrough](/guide/speech-to-avatar-walkthrough).


# Client Error Codes
Source: https://docs.spatialreal.ai/resources/client-error

iOS, Android, and Web SDK error codes with recovery guidance

All client SDKs surface errors through the `AvatarController.onError` callback as `AvatarError`.

<Tabs>
  <Tab title="Web">
    ```typescript theme={null}
    controller.onError = (error: AvatarError) => {
      console.error('Error:', error.code, error.message)
    }
    ```
  </Tab>

  <Tab title="iOS">
    ```swift theme={null}
    controller.onError = { error in
        print("Error: \(error)")
    }
    ```
  </Tab>

  <Tab title="Android">
    ```kotlin theme={null}
    controller?.onError = { error ->
        Log.e("Avatar", "Error: ${error.message}")
    }
    ```
  </Tab>
</Tabs>

## Web — ErrorCode

| Code                           | Description                                 | Recovery                                               |
| ------------------------------ | ------------------------------------------- | ------------------------------------------------------ |
| `appIDUnrecognized`            | App ID not recognized.                      | Verify your app ID in the dashboard.                   |
| `sessionTokenInvalid`          | Session token is invalid or appId mismatch. | Verify token generation on your server.                |
| `sessionTokenExpired`          | Session token has expired.                  | Request a new token from your server.                  |
| `insufficientBalance`          | Account balance is insufficient.            | Top up your account in the dashboard.                  |
| `sessionTimeout`               | Session timed out.                          | Reconnect by calling `start()` again.                  |
| `concurrentLimitExceeded`      | Concurrent connection limit exceeded.       | Close other active sessions before starting a new one. |
| `avatarIDUnrecognized`         | Avatar ID not found.                        | Check the avatar ID exists and is published.           |
| `failedToFetchAvatarMetadata`  | Failed to fetch avatar metadata.            | Check network connectivity and retry.                  |
| `failedToDownloadAvatarAssets` | Failed to download avatar assets.           | Check network connectivity and retry.                  |
| `serverError`                  | Server-side error.                          | Check the error message for details.                   |

## iOS — AvatarError

| Error                          | Description                                 | Recovery                                               |
| ------------------------------ | ------------------------------------------- | ------------------------------------------------------ |
| `appIDUnrecognized`            | App ID not recognized.                      | Verify your app ID in the dashboard.                   |
| `sessionTokenInvalid`          | Session token is invalid or appId mismatch. | Verify token generation on your server.                |
| `sessionTokenExpired`          | Session token has expired.                  | Request a new token from your server.                  |
| `insufficientBalance`          | Account balance is insufficient.            | Top up your account in the dashboard.                  |
| `sessionTimeout`               | Session timed out.                          | Reconnect by calling `start()` again.                  |
| `concurrentLimitExceeded`      | Concurrent connection limit exceeded.       | Close other active sessions before starting a new one. |
| `avatarIDUnrecognized`         | Avatar ID not found.                        | Check the avatar ID exists and is published.           |
| `avatarAssetMissing`           | Avatar asset is missing locally.            | Re-download with `AvatarManager.load()`.               |
| `failedToFetchAvatarMetadata`  | Failed to fetch avatar metadata.            | Check network connectivity and retry.                  |
| `failedToDownloadAvatarAssets` | Failed to download avatar assets.           | Check network connectivity and retry.                  |
| `serverError(code, message)`   | Server-side error.                          | Check the error message for details.                   |

## Android — AvatarError

| Error                          | Description                                 | Recovery                                               |
| ------------------------------ | ------------------------------------------- | ------------------------------------------------------ |
| `AppIDUnrecognized`            | App ID not recognized.                      | Verify your app ID in the dashboard.                   |
| `SessionTokenInvalid`          | Session token is invalid or appId mismatch. | Verify token generation on your server.                |
| `SessionTokenExpired`          | Session token has expired.                  | Request a new token from your server.                  |
| `InsufficientBalance`          | Account balance is insufficient.            | Top up your account in the dashboard.                  |
| `SessionTimeout`               | Session timed out.                          | Reconnect by calling `start()` again.                  |
| `ConcurrentLimitExceeded`      | Concurrent connection limit exceeded.       | Close other active sessions before starting a new one. |
| `AvatarIDUnrecognized`         | Avatar ID not found.                        | Check the avatar ID exists and is published.           |
| `AvatarAssetMissing`           | Avatar asset is missing locally.            | Re-download with `AvatarManager.load()`.               |
| `FailedToFetchAvatarMetadata`  | Failed to fetch avatar metadata.            | Check network connectivity and retry.                  |
| `FailedToDownloadAvatarAssets` | Failed to download avatar assets.           | Check network connectivity and retry.                  |
| `ServerError(message, code)`   | Server-side error.                          | Check the error message for details.                   |

## Recommended Handling Flow

1. Log the error code and current lifecycle phase.
2. Retry transient network/download failures with backoff.
3. Refresh session token for auth-related errors.
4. Surface configuration errors to developers/operators.

For platform-specific API details, see:

* [Web SDK API reference](/sdk-reference/web-sdk/api-reference)
* [iOS SDK API reference](/sdk-reference/ios-sdk/api-reference)
* [Android SDK API reference](/sdk-reference/android-sdk/api-reference)


# Server Error Codes
Source: https://docs.spatialreal.ai/resources/server-error

Stable server-side SDK error codes and handling fields

For backend integrations, use your server SDK's structured error fields as the source of truth.

* [Server SDK reference](/sdk-reference/python-sdk/python-sdk)

Server-side integrations should account for session token failures, WebSocket handshake failures, runtime server errors, and unexpected connection closures.

## Common Server SDK Error Codes

These stable error codes are useful when building retry, logging, and alerting flows on your backend.

| Error code                | Meaning                                                 |
| ------------------------- | ------------------------------------------------------- |
| `sessionTokenExpired`     | Session token expired or is no longer authorized        |
| `sessionTokenInvalid`     | Session token is invalid or empty                       |
| `appIDUnrecognized`       | App ID is not recognized by the server                  |
| `appIDMismatch`           | Session token belongs to a different app                |
| `avatarNotFound`          | Requested avatar does not exist                         |
| `billingRequired`         | Session is blocked by billing requirements              |
| `creditsExhausted`        | Runtime or connect-time credits are exhausted           |
| `sessionDurationExceeded` | Session hit a billing-enforced duration limit           |
| `unsupportedSampleRate`   | Audio sample rate is not accepted by the handshake      |
| `invalidEgressConfig`     | LiveKit or Agora egress configuration is invalid        |
| `egressUnavailable`       | Egress service is unavailable or not configured         |
| `idleTimeout`             | Session was closed because no input was received        |
| `upstreamError`           | Upstream internal service failed                        |
| `protocolError`           | Invalid protobuf payload or unexpected message sequence |
| `connectionFailed`        | Transport-level connection setup failed                 |
| `connectionClosed`        | WebSocket closed unexpectedly                           |
| `serverError`             | Server returned an unclassified error                   |
| `invalidRequest`          | Client request payload or parameters are invalid        |
| `unknown`                 | Fallback code when the SDK cannot classify the failure  |

## Server SDK Error Fields

When handling server SDK errors, check these fields to determine the correct recovery path:

* `code`: stable SDK error code for programmatic handling
* `phase`: where the failure happened, such as `session_token`, `websocket_connect`, or `websocket_runtime`
* `http_status`: HTTP status for token or upgrade failures
* `server_code`, `server_title`, `server_detail`: normalized details returned by the server
* `connection_id`, `req_id`: correlation IDs for tracing requests in logs
* `close_code`, `close_reason`: WebSocket close details for unexpected disconnects

## Recommended Handling Flow

1. Log `code`, `phase`, and correlation IDs (`connection_id`, `req_id`).
2. Retry only transient failures (for example `connectionClosed`, `connectionFailed`, `idleTimeout`).
3. Surface configuration/auth failures (`sessionTokenInvalid`, `appIDMismatch`) to operators.
4. Alert on recurring `upstreamError`, `serverError`, or `protocolError`.


# API Reference
Source: https://docs.spatialreal.ai/sdk-reference/android-sdk/api-reference

Browse the complete API reference

## Installation

```kotlin title="build.gradle.kts" theme={null}
dependencies {
    implementation("ai.spatialwalk:avatarkit:1.0.0-beta49")
}
```

### AvatarSDK

The core management class of the SDK, responsible for initialization and global configuration.

```kotlin theme={null}
object AvatarSDK
```

##### Properties

<Accordion title="sessionToken">
  The session token used to authenticate avatars with the AvatarKit Server.

  ```kotlin theme={null}
  var sessionToken: String
  ```
</Accordion>

<Accordion title="userId">
  The user identifier.

  ```kotlin theme={null}
  var userId: String
  ```
</Accordion>

<Accordion title="version">
  Returns the version of AvatarKit.

  ```kotlin theme={null}
  val version: String
  ```
</Accordion>

##### Methods

<Accordion title="initialize(context, appId, configuration)">
  Initialize AvatarKit.

  ```kotlin theme={null}
  fun initialize(
      context: Context,
      appId: String,
      configuration: Configuration
  )
  ```

  **Parameters:**

  * `context`: Application context
  * `appId`: Your application identifier
  * `configuration`: The configuration for AvatarKit
</Accordion>

<Accordion title="isDeviceSupported()">
  Returns whether the current device meets the minimum requirements for avatar rendering.

  ```kotlin theme={null}
  suspend fun isDeviceSupported(): Boolean
  ```
</Accordion>

<Accordion title="deviceScore()">
  Measures the device's computational performance for avatar rendering.

  ```kotlin theme={null}
  suspend fun deviceScore(): DeviceScore
  ```

  **Returns:** A `DeviceScore` containing `cpuScore` and `gpuScore`.
</Accordion>

### AvatarManager

Avatar resource manager, responsible for downloading, caching, and loading avatar data.

```kotlin theme={null}
object AvatarManager
```

##### Methods

<Accordion title="initialize(context)">
  Initialize AvatarManager. Must be called before use.

  ```kotlin theme={null}
  fun initialize(context: Context)
  ```

  **Parameters:**

  * `context`: Application context
</Accordion>

<Accordion title="load(id, onProgress)">
  Loads an avatar by ID.

  ```kotlin theme={null}
  suspend fun load(
      id: String,
      onProgress: ((LoadProgress) -> Unit)? = null
  ): Avatar
  ```

  **Parameters:**

  * `id`: The avatar identifier
  * `onProgress`: Optional progress callback

  **Returns:** The loaded `Avatar` instance.
</Accordion>

<Accordion title="clear(id)">
  Clears cached data for a specific avatar.

  ```kotlin theme={null}
  suspend fun clear(id: String)
  ```

  **Parameters:**

  * `id`: The avatar identifier to clear
</Accordion>

<Accordion title="clearAll()">
  Clears all cached avatar data.

  ```kotlin theme={null}
  suspend fun clearAll()
  ```
</Accordion>

<Accordion title="getCacheSize(id)">
  Gets the cache size for a specific avatar.

  ```kotlin theme={null}
  suspend fun getCacheSize(id: String): Long
  ```

  **Parameters:**

  * `id`: The avatar identifier

  **Returns:** The cache size in bytes.
</Accordion>

<Accordion title="getAllCacheSize()">
  Gets the total cache size for all avatars.

  ```kotlin theme={null}
  suspend fun getAllCacheSize(): Long
  ```

  **Returns:** The total cache size in bytes.
</Accordion>

<Accordion title="getCacheStats()">
  Get cache statistics.

  ```kotlin theme={null}
  suspend fun getCacheStats(): CacheStats
  ```

  **Returns:** A `CacheStats` object containing total entries, total size, and max size.
</Accordion>

### AvatarController

Real-time communication controller that handles WebSocket connections and audio/video data.

```kotlin theme={null}
class AvatarController
```

##### Properties

<Accordion title="onConnectionState">
  Callback for connection state changes.

  ```kotlin theme={null}
  var onConnectionState: ((ConnectionState) -> Unit)?
  ```
</Accordion>

<Accordion title="onConversationState">
  Callback for conversation state changes.

  ```kotlin theme={null}
  var onConversationState: ((ConversationState) -> Unit)?
  ```
</Accordion>

<Accordion title="onError">
  Callback for error events.

  ```kotlin theme={null}
  var onError: ((AvatarError) -> Unit)?
  ```
</Accordion>

<Accordion title="volume">
  The volume of playback (0.0 to 1.0).

  ```kotlin theme={null}
  var volume: Float
  ```
</Accordion>

##### Methods

<Accordion title="start()">
  Starts the avatar driving service connection.

  ```kotlin theme={null}
  fun start()
  ```
</Accordion>

<Accordion title="close()">
  Close connection.

  ```kotlin theme={null}
  fun close()
  ```
</Accordion>

<Accordion title="interrupt()">
  Stops playback and terminates the current conversation.

  ```kotlin theme={null}
  fun interrupt()
  ```
</Accordion>

<Accordion title="send(audioData, end)">
  Sends audio to the avatar driving service.

  ```kotlin theme={null}
  suspend fun send(audioData: ByteArray, end: Boolean = false): String
  ```

  **Parameters:**

  * `audioData`: PCM16 (S16LE) mono audio data. Byte length **must** be even (2 bytes per sample).
  * `end`: Whether this is the end of the audio stream

  **Returns:** A conversation ID string.

  <Warning>
    Audio data byte length must be even (2 bytes per sample). Odd-length data will cause a server-side validation error and WebSocket disconnect.
  </Warning>
</Accordion>

<Accordion title="yield(audioData, end, audioFormat)">
  Yields audio from the host.

  ```kotlin theme={null}
  suspend fun yield(
      audioData: ByteArray,
      end: Boolean = false,
      audioFormat: AudioFormat? = null
  ): String
  ```

  **Parameters:**

  * `audioData`: The audio data
  * `end`: Whether this is the end of the audio stream
  * `audioFormat`: The audio format (optional)

  **Returns:** A conversation ID string.
</Accordion>

<Accordion title="yield(animations, conversationId)">
  Yields animations from the host. Returns whether the server has sent all animation data for this conversation.

  ```kotlin theme={null}
  fun yield(animations: List<ByteArray>, conversationId: String): Boolean
  ```

  **Parameters:**

  * `animations`: List of animation data
  * `conversationId`: The request ID returned from yield audio

  **Returns:** `true` if the server has sent all animation data (end signal received), `false` otherwise.
</Accordion>

### AvatarView

3D rendering view that automatically creates and manages AvatarController.

```kotlin theme={null}
class AvatarView : FrameLayout
```

##### Initializers

<Accordion title="constructor(context)">
  Creates a new avatar view.

  ```kotlin theme={null}
  constructor(context: Context)
  ```
</Accordion>

##### Properties

<Accordion title="controller">
  The controller for the avatar.

  ```kotlin theme={null}
  val controller: AvatarController?
  ```
</Accordion>

<Accordion title="avatarTransform">
  Transform for avatar content position and scale within the view.

  ```kotlin theme={null}
  var avatarTransform: Transform
  ```
</Accordion>

<Accordion title="onFirstRendering">
  Callback when the first frame is rendered.

  ```kotlin theme={null}
  var onFirstRendering: (() -> Unit)?
  ```
</Accordion>

##### Methods

<Accordion title="init(avatar, scope)">
  Initialize view with avatar.

  ```kotlin theme={null}
  fun init(avatar: Avatar, scope: CoroutineScope)
  ```

  **Parameters:**

  * `avatar`: The avatar to display
  * `scope`: Coroutine scope, usually obtained via `activity.lifecycleScope`
</Accordion>

<Accordion title="pauseRendering()">
  Pause avatar rendering.

  ```kotlin theme={null}
  fun pauseRendering()
  ```
</Accordion>

<Accordion title="resumeRendering()">
  Resume avatar rendering.

  ```kotlin theme={null}
  fun resumeRendering()
  ```
</Accordion>

<Accordion title="dispose()">
  Release all resources. Call when the avatar view is no longer needed (e.g., in onDestroy).

  ```kotlin theme={null}
  fun dispose()
  ```
</Accordion>

***

### Avatar

Avatar data class containing core avatar information.

```kotlin theme={null}
class Avatar
```

| Property     | Type     | Description             |
| ------------ | -------- | ----------------------- |
| `id`         | `String` | The avatar identifier.  |
| `pointCount` | `Int`    | The avatar point count. |

### Configuration

SDK configuration class.

```kotlin theme={null}
data class Configuration
```

##### Initializers

<Accordion title="Configuration(environment, audioFormat, drivingServiceMode, logLevel)">
  Creates a new configuration with the specified parameters.

  ```kotlin theme={null}
  data class Configuration(
      val environment: Environment,
      val audioFormat: AudioFormat = AudioFormat(16000),
      val drivingServiceMode: DrivingServiceMode = DrivingServiceMode.SDK,
      val logLevel: LogLevel = LogLevel.INFO
  )
  ```

  **Parameters:**

  * `environment`: The environment to use
  * `audioFormat`: The audio format configuration
  * `drivingServiceMode`: The driving service mode
  * `logLevel`: The log level
</Accordion>

### Environment

Environment enum.

```kotlin theme={null}
enum class Environment
```

| Case   | Description                |
| ------ | -------------------------- |
| `intl` | International environment. |
| `cn`   | China environment.         |
| `test` | Test environment.          |

### AudioFormat

Audio format configuration for AvatarKit.

```kotlin theme={null}
class AudioFormat
```

##### Initializers

<Accordion title="AudioFormat(sampleRate:)">
  Creates a new audio format with the specified sample rate.

  ```kotlin theme={null}
  class AudioFormat(val sampleRate: Int)
  ```

  **Parameters:**

  * `sampleRate`: The audio sample rate in Hz. Supported sample rates are: 8000, 16000, 22050, 24000, 32000, 44100, 48000.
</Accordion>

### DrivingServiceMode

Driving service modes for AvatarKit.

```kotlin theme={null}
enum class DrivingServiceMode
```

| Case   | Description                                   |
| ------ | --------------------------------------------- |
| `SDK`  | The SDK handles driving service internally.   |
| `HOST` | The host application handles driving service. |

### LogLevel

Log levels for AvatarKit.

```kotlin theme={null}
enum class LogLevel
```

| Case      | Description           |
| --------- | --------------------- |
| `ALL`     | Log all messages.     |
| `VERBOSE` | Log verbose messages. |
| `DEBUG`   | Log debug messages.   |
| `INFO`    | Log info messages.    |
| `WARNING` | Log warning messages. |
| `ERROR`   | Log error messages.   |
| `OFF`     | Disable logging.      |

### ConnectionState

Connection state sealed class.

```kotlin theme={null}
sealed class ConnectionState
```

| Case                | Description                              |
| ------------------- | ---------------------------------------- |
| `Connecting`        | The connection is being established.     |
| `Connected`         | The connection is active.                |
| `Disconnected`      | The connection has been closed.          |
| `Failed(Exception)` | The connection failed with an exception. |

### ConversationState

Avatar conversation state enum.

```kotlin theme={null}
enum class ConversationState
```

| Case      | Description                              |
| --------- | ---------------------------------------- |
| `Idle`    | Idle state, showing breathing animation. |
| `Playing` | The avatar is playing audio/animation.   |
| `Paused`  | Playback is paused.                      |

### LoadProgress

Load progress sealed class.

```kotlin theme={null}
sealed class LoadProgress
```

| Case                 | Description                      |
| -------------------- | -------------------------------- |
| `Downloading(Float)` | Downloading with progress (0-1). |
| `Completed`          | Loading completed.               |
| `Failed(Throwable)`  | Loading failed with error.       |


# Changelog
Source: https://docs.spatialreal.ai/sdk-reference/android-sdk/changelog

AvatarKit Android SDK version update history

## 1.0.0-beta48

### ✨ Features

* **Rendering quality improvement** — Improved color accuracy for avatar rendering.
* **Avatar transform from character settings** — Avatar automatically applies position and scale from character settings. User-set `avatarTransform` stacks on top of the base values.

## 1.0.0-beta47

### ✨ Features

* **Device support expansion** — Previously rejected devices now use benchmark scoring instead of being blocked outright, expanding supported device range.

## 1.0.0-beta46

### 🐛 Bugfixes

* **Render stability** — Fixed a crash on certain devices during view lifecycle transitions.

### ✨ Features

* **WebSocket close code handling** — Added handling for server custom close codes: auth failure, insufficient balance, session timeout, concurrent limit exceeded. Non-recoverable errors are now properly recognized.
* **HTTP error mapping improvement** — Non-mapped HTTP errors now return descriptive error information instead of being silently ignored.

## 1.0.0-beta45

### 🐛 Bugfixes

* **Android \< 12 device support fix** — Fixed a crash that prevented \~30% of devices (Android 11 and below) from completing device support detection.

## 1.0.0-beta44

### 🐛 Bugfixes

* **Pre-init crash fix** — `isDeviceSupported()` no longer crashes when called before `initialize()`.

### 🔧 Improvements

* **Telemetry event queue** — Events sent before SDK initialization are now queued and flushed after `initialize()`.

## 1.0.0-beta43

### ✨ Features

* **Host mode animation end signal** — `yield(animations, conversationId)` now returns a `Boolean` indicating whether the server has sent all animation data for the current conversation.

### 🐛 Bugfixes

* **Background task cleanup** — Fixed background tasks not being properly cancelled when app enters background.

## 1.0.0-beta42

### 🐛 Bugfixes

* **Release build fix** — Fixed crashes in release builds when code minification is enabled.

## 1.0.0-beta41

### 🐛 Bugfixes

* **Release build support** — Fixed crashes in consumer apps when code minification is enabled.

## 1.0.0-beta40

### 🐛 Bugfixes

* **Initialization stability** — Fixed a crash that could occur during SDK initialization on some devices.

## 1.0.0-beta39

### 🐛 Bugfixes

* **Huawei device rendering fix** — Fixed partial flickering on Huawei Mate 70 Pro and similar devices.
* **Close callback timing** — `close()` now triggers `Disconnected` callback synchronously, aligned with iOS/Web behavior.
* **Network reconnect stability** — Improved stability during rapid close/restart cycles.

### ✨ Features

* **Device support threshold** — Lowered benchmark threshold, expanding supported device range from 37% to 61%.

## 1.0.0-beta38

### 🐛 Bugfixes

* **Compressed model cache** — Fixed cache not differentiating between standard and compressed models.
* **Deprecated API removal** — Removed `supportsCurrentDevice()` and `benchmark()` in favor of `isDeviceSupported()` and `deviceScore()`.

## 1.0.0-beta37

### ⚡ Breaking Changes

* **FrameRateInfo** — Unified as pre-computed fields for cross-SDK consistency.

### ✨ Features

* **Frame rate monitor** — Enable/disable switch for frame rate monitoring.
* **Device benchmark** — `deviceScore` and `isDeviceSupported` APIs for runtime device capability assessment.
* **Compressed model (Beta)** — `load()` accepts optional `useCompressedModel` parameter for loading compressed models (\~30% smaller). Please verify rendering quality before using in production.

### 🐛 Bugfixes

* **Detach/attach rendering** — Fixed rendering failure when AvatarView is detached and re-attached to window hierarchy.
* **Duplicate initialize guard** — Prevented duplicate `initialize()` calls from causing unexpected state.
* **AppId validation** — Added `appId` validation on `initialize()` with guidance for obtaining credentials.


# Golang SDK
Source: https://docs.spatialreal.ai/sdk-reference/go-sdk/go-sdk

Using the Golang SDK

## Repository

The Golang SDK is available on GitHub: [spatialwalk/avatar-sdk-go](https://github.com/spatialwalk/avatar-sdk-go).

```bash theme={null}
go get github.com/spatialwalk/avatar-sdk-go
```

## Quick start

```go theme={null}
import (
	avatarsdkgo "github.com/spatialwalk/avatar-sdk-go"
)

func main() {
	session := avatarsdkgo.NewAvatarSession(
		avatarsdkgo.WithAPIKey("your-api-key"),
		avatarsdkgo.WithAppID("your-app-id"),
		avatarsdkgo.WithAvatarID("your-avatar-id"),
		avatarsdkgo.WithConsoleEndpointURL("https://console.us-west.spatialwalk.cloud/v1/console"),
		avatarsdkgo.WithIngressEndpointURL("wss://api.us-west.spatialwalk.cloud/v2/driveningress"),
		avatarsdkgo.WithExpireAt(time.Now().Add(5 * time.Minute)),
		avatarsdkgo.WithTransportFrames(func(data []byte, last bool) {
			// handle animation frame
		}),
		avatarsdkgo.WithOnError(func(err error) {
			// handle error
		}),
		avatarsdkgo.WithOnClose(func() {
			// handle close
		}),
	)

	if err := session.Init(ctx); err != nil {
		// handle init error
	}

	connectionID, err := session.Start(ctx)
	if err != nil {
		// handle start error
	}
	defer session.Close()

	// Send audio
	requestID, err := session.SendAudio(audioData, true)
	if err != nil {
		// handle error
	}
}
```

## Egress Mode

For real-time applications, you can configure egress to stream avatar output directly to a real-time platform like [**LiveKit** (rooms)](https://docs.livekit.io/intro/overview/). See [LiveKit Agent server side guide](/guide/rtc-livekit-server) for full configuration and code examples.

Quick example with LiveKit:

```go theme={null}
session := avatarsdkgo.NewAvatarSession(
	// ... other options ...
	avatarsdkgo.WithLiveKitEgress(&avatarsdkgo.LiveKitEgressConfig{
		URL:         "wss://your-livekit-server.com",
		APIKey:      "livekit-api-key",
		APISecret:   "livekit-api-secret",
		RoomName:    "your-room-name",
		PublisherID: "avatar-publisher",
	}),
)
```

## Changelog

[https://github.com/spatialwalk/avatar-sdk-go/releases](https://github.com/spatialwalk/avatar-sdk-go/releases)


# API Reference
Source: https://docs.spatialreal.ai/sdk-reference/ios-sdk/api-reference

Browse the complete API reference

## Installation

Download the latest [AvatarKit.xcframework](https://github.com/spatialwalk/avatar-kit-ios-release/releases/download/v1.0.0-beta.28/AvatarKit_202604142117.zip), unzip, and drag it into your Xcode project. Set **Embed & Sign** in your target's **Frameworks, Libraries, and Embedded Content**.

### AvatarSDK

Main initialization and configuration interface for AvatarKit.

```swift theme={null}
@MainActor enum AvatarSDK
```

##### Type Properties

<Accordion title="appID">
  The app identifier.

  ```swift theme={null}
  static var appID: String { get }
  ```
</Accordion>

<Accordion title="configuration">
  The app configuration.

  ```swift theme={null}
  static var configuration: Configuration { get }
  ```
</Accordion>

<Accordion title="sessionToken">
  The session token used to authenticate avatars with the AvatarKit Server.

  ```swift theme={null}
  static var sessionToken: String
  ```
</Accordion>

<Accordion title="userID">
  The user identifier provided by the developer to identify the user in the AvatarKit Log Service.

  ```swift theme={null}
  static var userID: String
  ```
</Accordion>

<Accordion title="isDeviceSupported()">
  Returns whether the current device meets the minimum requirements for avatar rendering.

  ```swift theme={null}
  static func isDeviceSupported() async -> Bool
  ```
</Accordion>

<Accordion title="version">
  Returns the version of AvatarKit.

  ```swift theme={null}
  static var version: String { get }
  ```
</Accordion>

##### Type Methods

<Accordion title="initialize(appID:configuration:)">
  Initialize AvatarKit.

  ```swift theme={null}
  static func initialize(appID: String, configuration: Configuration)
  ```

  **Parameters:**

  * `appID`: The app identifier
  * `configuration`: The configuration for AvatarKit
</Accordion>

<Accordion title="deviceScore()">
  Measures the device's computational performance for avatar rendering.

  ```swift theme={null}
  static func deviceScore() async -> DeviceScore
  ```

  **Returns:** A `DeviceScore` containing `cpuScore: Int` and `gpuScore: Int`.
</Accordion>

### AvatarManager

Manage avatar asset loading, caching, and retrieval.

```swift theme={null}
final class AvatarManager
```

##### Type Properties

<Accordion title="shared">
  The shared avatar manager instance.

  ```swift theme={null}
  static let shared: AvatarManager
  ```
</Accordion>

##### Instance Methods

<Accordion title="load(id:onProgress:)">
  Loads an avatar by ID.

  ```swift theme={null}
  func load(id: String, onProgress: (ProgressHandler)?) async throws -> Avatar
  ```

  **Parameters:**

  * `id`: The avatar identifier
  * `onProgress`: Optional progress callback

  **Returns:** The loaded `Avatar` instance.
</Accordion>

<Accordion title="cancelLoading(id:)">
  Cancels the loading of avatar by ID.

  ```swift theme={null}
  func cancelLoading(id: String) async
  ```

  **Parameters:**

  * `id`: The avatar identifier
</Accordion>

<Accordion title="cancelAllLoading()">
  Cancels all loading of avatars.

  ```swift theme={null}
  func cancelAllLoading() async
  ```
</Accordion>

<Accordion title="retrieve(id:)">
  Retrieves a cached avatar by ID.

  ```swift theme={null}
  func retrieve(id: String) -> Avatar?
  ```

  **Parameters:**

  * `id`: The avatar identifier

  **Returns:** The cached `Avatar` if available, otherwise `nil`.
</Accordion>

<Accordion title="derive(assetPath:)">
  Derives an avatar from a local asset path.

  ```swift theme={null}
  func derive(assetPath: String) throws -> Avatar
  ```

  **Parameters:**

  * `assetPath`: The path to the avatar asset

  **Returns:** The derived `Avatar` instance.
</Accordion>

<Accordion title="clear(id:)">
  Clears cached data for a specific avatar.

  ```swift theme={null}
  func clear(id: String) throws
  ```

  **Parameters:**

  * `id`: The avatar identifier to clear
</Accordion>

<Accordion title="clearAll()">
  Clears all cached avatar data.

  ```swift theme={null}
  func clearAll() throws
  ```
</Accordion>

<Accordion title="clearLRU(keepCount:)">
  Clears least recently used avatars, keeping the specified count.

  ```swift theme={null}
  func clearLRU(keepCount: Int) throws
  ```

  **Parameters:**

  * `keepCount`: Number of most recently used avatars to keep
</Accordion>

<Accordion title="getCacheSize(id:)">
  Gets the cache size for a specific avatar.

  ```swift theme={null}
  func getCacheSize(id: String) throws -> Int
  ```

  **Parameters:**

  * `id`: The avatar identifier

  **Returns:** The cache size in bytes.
</Accordion>

<Accordion title="getAllCacheSize()">
  Gets the total cache size for all avatars.

  ```swift theme={null}
  func getAllCacheSize() throws -> Int
  ```

  **Returns:** The total cache size in bytes.
</Accordion>

### AvatarController

The main controller for managing avatar driving service connections and interactions.

```swift theme={null}
@MainActor final class AvatarController
```

##### Instance Properties

<Accordion title="onFirstRendering">
  Callback when the avatar is first rendered.

  ```swift theme={null}
  var onFirstRendering: (() -> Void)?
  ```
</Accordion>

<Accordion title="onConnectionState">
  Callback for connection state changes.

  ```swift theme={null}
  var onConnectionState: ((ConnectionState) -> Void)?
  ```
</Accordion>

<Accordion title="onConversationState">
  Callback for conversation state changes.

  ```swift theme={null}
  var onConversationState: ((ConversationState) -> Void)?
  ```
</Accordion>

<Accordion title="onError">
  Callback for error events.

  ```swift theme={null}
  var onError: ((AvatarError) -> Void)?
  ```
</Accordion>

<Accordion title="start()">
  Starts the avatar driving service connection.

  ```swift theme={null}
  func start()
  ```
</Accordion>

<Accordion title="close()">
  Closes the avatar driving service.

  ```swift theme={null}
  func close()
  ```
</Accordion>

<Accordion title="pause()">
  Pauses audio and animation playback.

  ```swift theme={null}
  func pause()
  ```
</Accordion>

<Accordion title="resume()">
  Resumes audio and animation playback.

  ```swift theme={null}
  func resume()
  ```
</Accordion>

<Accordion title="isRendering">
  Whether the current avatar is actually rendering.

  ```swift theme={null}
  var isRendering: Bool { get }
  ```
</Accordion>

<Accordion title="interrupt()">
  Stops playback and terminates the current conversation.

  ```swift theme={null}
  func interrupt()
  ```
</Accordion>

<Accordion title="send(_:end:)">
  Sends audio to the avatar driving service.

  ```swift theme={null}
  func send(_ data: Data, end: Bool) -> String
  ```

  **Parameters:**

  * `data`: PCM16 (S16LE) mono audio data. Byte length **must** be even (2 bytes per sample).
  * `end`: Whether this is the end of the audio stream

  **Returns:** A conversation ID string.

  <Warning>
    Audio data byte length must be even (2 bytes per sample). Odd-length data will cause a server-side validation error and WebSocket disconnect.
  </Warning>
</Accordion>

<Accordion title="yield(_:end:audioFormat:)">
  Yields audio from the host.

  ```swift theme={null}
  func yield(_ data: Data, end: Bool, audioFormat: AudioFormat?) -> String
  ```

  **Parameters:**

  * `data`: The audio data
  * `end`: Whether this is the end of the audio stream
  * `audioFormat`: The audio format (optional)

  **Returns:** A conversation ID string.
</Accordion>

<Accordion title="yield(_:conversationID:)">
  Yields animations from the host.

  ```swift theme={null}
  func yield(_ animations: [Data], conversationID: String)
  ```

  **Parameters:**

  * `animations`: Array of animation data
  * `conversationID`: The conversation identifier
</Accordion>

<Accordion title="pointCount">
  The point count of the current avatar.

  ```swift theme={null}
  var pointCount: Int { get }
  ```
</Accordion>

<Accordion title="volume">
  The volume of playback.

  ```swift theme={null}
  var volume: Float
  ```
</Accordion>

### AvatarView

A UIView subclass for rendering avatars.

```swift theme={null}
@MainActor final class AvatarView: UIView
```

##### Initializers

<Accordion title="init(avatar:)">
  Creates a new avatar view with the specified avatar.

  ```swift theme={null}
  init(avatar: Avatar)
  ```

  **Parameters:**

  * `avatar`: The avatar to display
</Accordion>

##### Instance Properties

<Accordion title="controller">
  The controller for the avatar.

  ```swift theme={null}
  var controller: AvatarController { get }
  ```
</Accordion>

<Accordion title="avatarTransform">
  Transform for avatar content position and scale within the view.

  ```swift theme={null}
  var avatarTransform: Transform
  ```
</Accordion>

<Accordion title="isOpaque">
  Whether the view is opaque.

  ```swift theme={null}
  var isOpaque: Bool
  ```
</Accordion>

##### Instance Methods

<Accordion title="pauseRendering()">
  Pause avatar rendering.

  ```swift theme={null}
  func pauseRendering()
  ```
</Accordion>

<Accordion title="resumeRendering()">
  Resume avatar rendering.

  ```swift theme={null}
  func resumeRendering()
  ```
</Accordion>

***

### Avatar

Represents an avatar instance.

```swift theme={null}
struct Avatar
```

| Instance Property | Type     | Description                                   |
| ----------------- | -------- | --------------------------------------------- |
| `id`              | `String` | The avatar identifier.                        |
| `isFromCache`     | `Bool`   | Whether the instance of avatar is from cache. |

### Configuration

Configuration for AvatarKit.

```swift theme={null}
struct Configuration
```

##### Initializers

<Accordion title="init(environment:audioFormat:drivingServiceMode:logLevel:)">
  Creates a new configuration with the specified parameters.

  ```swift theme={null}
  init(
      environment: Environment,
      audioFormat: AudioFormat,
      drivingServiceMode: DrivingServiceMode,
      logLevel: LogLevel
  )
  ```

  **Parameters:**

  * `environment`: The environment to use
  * `audioFormat`: The audio format configuration
  * `drivingServiceMode`: The driving service mode
  * `logLevel`: The log level
</Accordion>

| Instance Property    | Type                 | Description                         |
| -------------------- | -------------------- | ----------------------------------- |
| `environment`        | `Environment`        | Environment for AvatarKit.          |
| `audioFormat`        | `AudioFormat`        | Audio format for AvatarKit.         |
| `drivingServiceMode` | `DrivingServiceMode` | Driving service mode for AvatarKit. |
| `logLevel`           | `LogLevel`           | Log level for AvatarKit.            |

### Environment

Environments for AvatarKit.

```swift theme={null}
enum Environment
```

| Case   | Description                |
| ------ | -------------------------- |
| `intl` | International environment. |
| `cn`   | China environment.         |

### AudioFormat

Audio format configuration for AvatarKit.

```swift theme={null}
struct AudioFormat
```

##### Initializers

<Accordion title="init(sampleRate:)">
  Creates a new audio format with the specified sample rate.

  ```swift theme={null}
  init(sampleRate: Int)
  ```

  **Parameters:**

  * `sampleRate`: The audio sample rate in Hz
</Accordion>

| Instance Property | Type  | Description                                        |
| ----------------- | ----- | -------------------------------------------------- |
| `sampleRate`      | `Int` | The audio sample rate in Hz.                       |
| `channelCount`    | `Int` | The number of audio channels. Fixed to 1 for mono. |

### DrivingServiceMode

Driving service modes for AvatarKit.

```swift theme={null}
enum DrivingServiceMode
```

| Case   | Description                                   |
| ------ | --------------------------------------------- |
| `sdk`  | The SDK handles driving service internally.   |
| `host` | The host application handles driving service. |

### LogLevel

Log levels for AvatarKit.

```swift theme={null}
enum LogLevel
```

| Case      | Description                   |
| --------- | ----------------------------- |
| `all`     | Log all messages.             |
| `warning` | Log warnings and errors only. |
| `error`   | Log errors only.              |
| `off`     | Disable logging.              |

### ConnectionState

Connection states for AvatarKit.

```swift theme={null}
enum ConnectionState
```

| Case            | Description                          |
| --------------- | ------------------------------------ |
| `connecting`    | The connection is being established. |
| `connected`     | The connection is active.            |
| `disconnected`  | The connection has been closed.      |
| `failed(Error)` | The connection failed with an error. |

### ConversationState

Conversation states for AvatarKit.

```swift theme={null}
enum ConversationState
```

| Case      | Description                            |
| --------- | -------------------------------------- |
| `idle`    | No active conversation.                |
| `playing` | The avatar is playing audio/animation. |
| `paused`  | The conversation is paused.            |

### Transform

Transform for avatar content rendering.

```swift theme={null}
struct Transform
```

##### Initializers

<Accordion title="init(x:y:scale:)">
  Creates a new transform with the specified translation and scale.

  ```swift theme={null}
  init(x: Float, y: Float, scale: Float)
  ```

  **Parameters:**

  * `x`: The x-axis translation
  * `y`: The y-axis translation
  * `scale`: The scale factor
</Accordion>

| Type Property | Type        | Description             |
| ------------- | ----------- | ----------------------- |
| `identity`    | `Transform` | The identity transform. |

| Instance Property | Type    | Description             |
| ----------------- | ------- | ----------------------- |
| `x`               | `Float` | The x-axis translation. |
| `y`               | `Float` | The y-axis translation. |
| `scale`           | `Float` | The scale factor.       |

### AvatarError

Avatar errors for AvatarKit.

```swift theme={null}
enum AvatarError: LocalizedError
```

| Case                           | Description                              |
| ------------------------------ | ---------------------------------------- |
| `appIDUnrecognized`            | The app ID is not recognized.            |
| `avatarIDUnrecognized`         | The avatar ID is not recognized.         |
| `avatarAssetMissing`           | The avatar asset is missing.             |
| `failedToDownloadAvatarAssets` | Failed to download avatar assets.        |
| `failedToFetchAvatarMetadata`  | Failed to fetch avatar metadata.         |
| `sessionTokenExpired`          | The session token has expired.           |
| `sessionTokenInvalid`          | The session token is invalid.            |
| `insufficientBalance`          | Account balance is insufficient.         |
| `sessionTimeout`               | Session timed out.                       |
| `concurrentLimitExceeded`      | Concurrent connection limit exceeded.    |
| `serverError(code:message:)`   | Server-side error with code and message. |

| Instance Property  | Type      | Description                           |
| ------------------ | --------- | ------------------------------------- |
| `errorDescription` | `String?` | A localized description of the error. |
| `failureReason`    | `String?` | The reason for the failure.           |


# Changelog
Source: https://docs.spatialreal.ai/sdk-reference/ios-sdk/changelog

AvatarKit iOS SDK version update history

## 1.0.0-beta.28

### ✨ Features

* **Rendering quality improvement** — Improved color accuracy for avatar rendering.
* **Avatar transform from character settings** — Avatar automatically applies position and scale from character settings. User-set `avatarTransform` stacks on top of the base values.

## 1.0.0-beta.27

### ✨ Features

* **WebSocket close code handling** — Added handling for server custom close codes: auth failure, insufficient balance, session timeout, concurrent limit exceeded. Non-recoverable errors are now properly recognized.

## 1.0.0-beta.26

### 🔧 Improvements

* **Telemetry event queue** — Events sent before SDK initialization are now queued and flushed after `initialize()`.

## 1.0.0-beta.25

### ✨ Features

* **Host mode animation end signal** — `yield(_:conversationID:)` now returns a `Bool` indicating whether the server has sent all animation data for the current conversation.

## 1.0.0-beta.24

### 🐛 Bugfixes

* **Stream observer lifecycle** — Fixed state updates being lost across close/restart cycles, preventing missed callbacks after reconnect.
* **Fallback mode idle callback** — Fixed missing `onConversationState(.idle)` callback after audio completes in fallback mode.
* **Reconnect state fix** — Fixed `onConversationState(.playing)` not firing after close→reconnect→send.

### ✨ Features

* **Device support threshold** — Lowered benchmark threshold based on production telemetry showing all iOS devices run at stable 24.9fps.

## 1.0.0-beta.23

### ⚡ Breaking Changes

* Removed deprecated `supportsCurrentDevice()` and `benchmark()` APIs — use `isDeviceSupported()` and `deviceScore()` instead.

### ✨ Features

* **Compressed model support** — Added `useCompressedModel` parameter to `AvatarManager.load()` for loading compressed models (\~30% smaller).
* **Idle animation while waiting** — Render idle animation while waiting for first server frames, instead of showing a static avatar.

### 🐛 Bugfixes

* **Release build crash fix** — Fixed a crash that occurred in release builds.

## 1.0.0-beta.22

### ⚡ Breaking Changes

* **FrameRateInfo** — Unified as pre-computed fields for cross-SDK consistency.

### ✨ Features

* **Frame rate monitor** — Enable/disable switch for frame rate monitoring.
* **Device benchmark** — `deviceScore` and `isDeviceSupported` APIs for runtime device capability assessment.
* **Compressed model (Beta)** — `load()` accepts optional `useCompressedModel` parameter for loading compressed models (\~30% smaller). Please verify rendering quality before using in production.

### 🐛 Bugfixes

* **Detach/attach rendering** — Fixed renderer not rebuilt after AvatarView removed from and re-added to window hierarchy.
* **Close→Start reconnection** — Fixed connection state changes lost after `close()` then `start()` cycle.
* **Pause/resume stability** — Fixed pause/resume playback rendering issues.
* **AppId validation** — Improved validation error message with guidance for obtaining credentials.
* **Audio stability** — Fixed a crash when audio playback is interrupted by system events (e.g. phone call).


# JavaScript SDK
Source: https://docs.spatialreal.ai/sdk-reference/js-sdk/js-sdk

Server SDK for JavaScript/Node.js

Coming soon.


# Python SDK
Source: https://docs.spatialreal.ai/sdk-reference/python-sdk/python-sdk

Using the Python Server SDK

A Python SDK for connecting to avatar services via WebSocket, supporting audio streaming and receiving animation frames.

## Repository

The Python SDK is available on GitHub: [spatialwalk/avatar-sdk-python](https://github.com/spatialwalk/avatar-sdk-python) and [PyPI](https://pypi.org/project/avatarkit/).

```bash theme={null}
pip install avatarkit
```

## Requirements

* `api_key`
* `app_id`
* `avatar_id`
* console endpoint URL
* ingress endpoint URL

## Endpoints

Build your endpoint URLs using your selected region (see [Regions](/api-reference/regions)):

* Console endpoint URL: `https://console.<region>.spatialwalk.cloud/v1/console`
* Ingress endpoint URL: `wss://api.<region>.spatialwalk.cloud/v2/driveningress`

## Quick Start

```python theme={null}
import asyncio
from datetime import datetime, timedelta, timezone

from avatarkit import new_avatar_session


async def main():
    session = new_avatar_session(
        api_key="your-api-key",
        app_id="your-app-id",
        console_endpoint_url="https://console.us-west.spatialwalk.cloud/v1/console",
        ingress_endpoint_url="wss://api.us-west.spatialwalk.cloud/v2/driveningress",
        avatar_id="your-avatar-id",
        expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
        transport_frames=lambda frame, last: print(f"Received frame: {len(frame)} bytes"),
        on_error=lambda err: print(f"Error: {err}"),
        on_close=lambda: print("Session closed"),
    )

    await session.init()
    connection_id = await session.start()
    print(f"Connected: {connection_id}")

    audio_data = b"..."  # Your PCM audio data
    request_id = await session.send_audio(audio_data, end=True)
    print(f"Sent audio: {request_id}")

    await asyncio.sleep(10)
    await session.close()


if __name__ == "__main__":
    asyncio.run(main())
```

## Session Configuration

Use `new_avatar_session()` to configure and create a session:

```python theme={null}
from datetime import datetime, timedelta, timezone

from avatarkit import new_avatar_session


session = new_avatar_session(
    avatar_id="avatar-123",
    api_key="your-api-key",
    app_id="your-app-id",
    # Set use_query_auth=True for web-style auth that sends appId and sessionKey
    # in the WebSocket query string instead of request headers.
    use_query_auth=False,
    expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
    console_endpoint_url="https://console.us-west.spatialwalk.cloud/v1/console",
    ingress_endpoint_url="wss://api.us-west.spatialwalk.cloud/v2/driveningress",
    sample_rate=16000,
    transport_frames=on_frame_received,
    on_error=on_error,
    on_close=on_close,
)
```

## Session Lifecycle

```python theme={null}
# 1. Initialize and request a session token
await session.init()

# 2. Start the WebSocket connection
connection_id = await session.start()

# 3. Send audio data
request_id = await session.send_audio(audio_bytes, end=True)

# 4. Receive animation frames through the callback

# 5. Close the session
await session.close()
```

## Audio Format

The SDK currently supports **mono 16-bit PCM (`s16le`)** audio:

* Sample rate: `8000`, `16000`, `22050`, `24000`, `32000`, `44100`, `48000`
* Channels: `1` (mono)
* Bit depth: `16-bit`
* Format: raw PCM bytes

```python theme={null}
with open("audio.pcm", "rb") as f:
    audio_data = f.read()

await session.send_audio(audio_data, end=True)
```

## LiveKit Egress Mode

When configured with `livekit_egress`, audio and animation data are streamed to a LiveKit room instead of being returned through the WebSocket connection. For end-to-end integration details, see the [LiveKit Agent server side guide](/guide/rtc-livekit-server).

```python theme={null}
from datetime import datetime, timedelta, timezone

from avatarkit import LiveKitEgressConfig, new_avatar_session


session = new_avatar_session(
    avatar_id="avatar-123",
    api_key="your-api-key",
    app_id="your-app-id",
    console_endpoint_url="https://console.us-west.spatialwalk.cloud/v1/console",
    ingress_endpoint_url="wss://api.us-west.spatialwalk.cloud/v2/driveningress",
    expire_at=datetime.now(timezone.utc) + timedelta(minutes=5),
    livekit_egress=LiveKitEgressConfig(
        url="wss://livekit.example.com",
        api_key="livekit-api-key",
        api_secret="livekit-api-secret",
        room_name="my-room",
        publisher_id="avatar-publisher",
    ),
)
```

When LiveKit egress is enabled:

* the server streams output to the specified LiveKit room
* the `transport_frames` callback is not invoked
* audio and animation data are published under the configured publisher ID

### Interrupt

The `interrupt()` method sends an interrupt signal to stop current audio processing. This is only available when using LiveKit egress mode.

```python theme={null}
request_id = await session.send_audio(audio_data, end=True)

# Later, interrupt the most recent in-flight request.
interrupted_id = await session.interrupt()
print(f"Interrupted request: {interrupted_id}")
```

The interrupt uses the most recent request ID, even after `end=True` has been sent.

## Callbacks

### `transport_frames`

Receives animation frames from the server:

```python theme={null}
def on_frame_received(frame_data: bytes, is_last: bool):
    print(f"Received frame: {len(frame_data)} bytes")
    if is_last:
        print("This is the last frame")
```

### `on_error`

Handles structured SDK errors from the session:

```python theme={null}
from avatarkit import AvatarSDKError


def on_error(error: Exception):
    print(f"Session error: {error}")

    if isinstance(error, AvatarSDKError):
        print("  code:", error.code.value)
        print("  phase:", error.phase)
        print("  http_status:", error.http_status)
        print("  server_code:", error.server_code)
        print("  server_detail:", error.server_detail)
```

The SDK reports structured `AvatarSDKError` instances for token creation failures, WebSocket upgrade rejections, handshake failures, runtime `ServerError` messages, and unexpected connection drops.

### `on_close`

Called when the session closes:

```python theme={null}
def on_close():
    print("Session has been closed")
```

## Error Handling

Use `SessionTokenError` for token creation failures and `AvatarSDKError` for all other structured SDK errors:

```python theme={null}
from avatarkit import AvatarSDKError, SessionTokenError


try:
    await session.init()
    await session.start()
except SessionTokenError as error:
    print("token failed", error.code.value, error.server_detail)
except AvatarSDKError as error:
    print("sdk error", error.code.value, error.phase, error.server_detail)
```

`AvatarSDKError` and `SessionTokenError` expose these fields:

* `code` - stable SDK error code
* `message` - human-readable message
* `phase` - failure phase such as `session_token`, `websocket_connect`, `websocket_handshake`, `websocket_runtime`, or `websocket_send`
* `http_status` - HTTP status for token or WebSocket upgrade failures
* `server_code` - server-provided error code, including runtime protobuf `ServerError.code`
* `server_title` / `server_detail` - parsed server error details when available
* `connection_id` / `req_id` - server correlation identifiers when available
* `raw_body` - raw HTTP rejection body for token or WebSocket upgrade failures
* `close_code` / `close_reason` - WebSocket close details for unexpected disconnects

### Common `AvatarSDKErrorCode` values

* `sessionTokenExpired` - session token expired or unauthorized
* `sessionTokenInvalid` - invalid or empty session token
* `appIDUnrecognized` - App ID is not recognized by the server
* `appIDMismatch` - session token belongs to a different app
* `avatarNotFound` - avatar does not exist
* `billingRequired` - session denied by billing checks
* `creditsExhausted` - runtime or connect-time credits exhausted
* `sessionDurationExceeded` - billing-enforced session timeout reached
* `unsupportedSampleRate` - handshake rejected unsupported audio sample rate
* `invalidEgressConfig` - LiveKit or Agora egress config is invalid
* `egressUnavailable` - egress service is unavailable or not configured
* `idleTimeout` - server closed the session after input inactivity
* `upstreamError` - internal upstream service failed
* `protocolError` - invalid protobuf or unexpected message sequence
* `connectionFailed` - transport-level connection failure
* `connectionClosed` - unexpected WebSocket close
* `serverError` - server-side failure that did not match a more specific mapping
* `invalidRequest` - other client-side request validation errors
* `unknown` - fallback when the SDK cannot classify the failure

For a cross-SDK overview, see [Error Codes](/resources/error-codes).

## API Reference

### `AvatarSession`

Main class for managing avatar sessions.

#### Methods

* `async init()` - initialize the session and obtain a token
* `async start() -> str` - start the WebSocket connection and return a `connection_id`
* `async send_audio(audio: bytes, end: bool = False) -> str` - send audio data and return a request ID
* `async interrupt() -> str` - interrupt current audio processing in LiveKit egress mode
* `async close()` - close the session and clean up resources
* `config -> SessionConfig` - current session configuration

### `SessionConfig`

Configuration dataclass for avatar sessions.

#### Fields

* `avatar_id: str` - avatar identifier
* `api_key: str` - API key for authentication
* `app_id: str` - application identifier
* `use_query_auth: bool` - send WebSocket auth in query params instead of headers
* `expire_at: datetime` - session expiration time
* `sample_rate: int` - audio sample rate, default `16000`
* `bitrate: int` - audio bitrate, default `0`
* `transport_frames: Callable[[bytes, bool], None]` - frame callback
* `on_error: Callable[[Exception], None]` - error callback
* `on_close: Callable[[], None]` - close callback
* `console_endpoint_url: str` - Console API URL
* `ingress_endpoint_url: str` - ingress WebSocket URL
* `livekit_egress: Optional[LiveKitEgressConfig]` - LiveKit egress configuration

### `LiveKitEgressConfig`

Configuration for streaming to a LiveKit room.

#### Fields

* `url: str` - LiveKit server URL, for example `wss://livekit.example.com`
* `api_key: str` - LiveKit API key
* `api_secret: str` - LiveKit API secret
* `room_name: str` - room name to join
* `publisher_id: str` - publisher identity in the room
* `extra_attributes: dict[str, str]` - extra participant attributes
* `idle_timeout: int` - idle timeout in seconds, `0` uses server defaults

### Utility Functions

* `generate_log_id() -> str` - generate a unique log ID in the format `YYYYMMDDHHMMSS_<nanoid>`

### Exceptions

* `AvatarSDKError` - structured SDK error with stable code and context fields
* `SessionTokenError` - subclass of `AvatarSDKError` raised when session token creation fails

## Changelog

[avatar-sdk-python releases](https://github.com/spatialwalk/avatar-sdk-python/releases)


# API Reference
Source: https://docs.spatialreal.ai/sdk-reference/web-sdk/api-reference

Complete API documentation for AvatarKit Web SDK

## Quick Reference

| Class              | Purpose            | Key Methods                                   |
| ------------------ | ------------------ | --------------------------------------------- |
| `AvatarSDK`        | SDK initialization | `initialize()`, `setSessionToken()`           |
| `AvatarManager`    | Avatar loading     | `load()`                                      |
| `AvatarView`       | 3D rendering       | `onFirstRendering`, `dispose()`               |
| `AvatarController` | Communication      | `start()`, `send()`, `pause()`, `interrupt()` |

<Warning>
  **Important:** App ID and Session Token are paired and must be used together. Each App ID corresponds to a specific list of avatars that can be driven. You must obtain both App ID and Session Token from the [SpatialReal Studio](https://app.spatialreal.ai).
</Warning>

```typescript theme={null}
export class AvatarSDK
```

***

## AvatarSDK

Main entry point for SDK initialization and global configuration.

```typescript theme={null}
import { AvatarSDK, Environment } from '@spatialwalk/avatarkit'
```

### Static Properties

| Property        | Type             | Description                |
| --------------- | ---------------- | -------------------------- |
| `isInitialized` | `boolean`        | Whether SDK is initialized |
| `appId`         | `string \| null` | Current App ID             |
| `configuration` | `Configuration`  | Current SDK configuration  |
| `sessionToken`  | `string \| null` | Current session token      |
| `userId`        | `string \| null` | Current user ID            |
| `version`       | `string`         | SDK version                |

### Static Methods

#### initialize(appId, configuration)

Initialize the SDK. Must be called before any other operations.

```typescript theme={null}
await AvatarSDK.initialize('your-app-id', {
  environment: Environment.cn,  // or Environment.intl
  drivingServiceMode: DrivingServiceMode.sdk,  // Optional, default: sdk
  logLevel: LogLevel.warning,  // Optional, default: off
  audioFormat: {  // Optional
    channelCount: 1,
    sampleRate: 16000,
  },
})
```

#### setSessionToken(token)

Set session token for authentication. **Only required for SDK Mode** (server-driven animation).

```typescript theme={null}
AvatarSDK.setSessionToken('your-session-token')
```

**Important:**

* **Only needed for SDK Mode** - not required for Host Mode or RTC Mode
* Token must be obtained from your backend server
* Token has max 24 hours validity
* Token must be paired with App ID

#### setUserId(userId)

Set user ID for logging and analytics.

```typescript theme={null}
AvatarSDK.setUserId('user-123')
```

#### cleanup()

Release all SDK resources. Call when SDK is no longer needed.

```typescript theme={null}
AvatarSDK.cleanup()
```

***

## AvatarManager

Handles avatar loading and caching.

```typescript theme={null}
import { AvatarManager } from '@spatialwalk/avatarkit'
```

### Static Properties

| Property | Type            | Description        |
| -------- | --------------- | ------------------ |
| `shared` | `AvatarManager` | Singleton instance |

### Instance Methods

#### load(id, onProgress?)

Load an avatar by ID.

```typescript theme={null}
const avatar = await AvatarManager.shared.load('avatar-id', (progress) => {
  switch (progress.type) {
    case 'downloading':
      console.log(`Loading: ${progress.progress}%`)
      break
    case 'completed':
      console.log('Load complete')
      break
    case 'failed':
      console.error('Load failed:', progress.error)
      break
  }
})
```

**Parameters:**

| Parameter    | Type                                   | Description       |
| ------------ | -------------------------------------- | ----------------- |
| `id`         | `string`                               | Avatar ID         |
| `onProgress` | `(progress: LoadProgressInfo) => void` | Progress callback |

**Returns:** `Promise<Avatar | null>`

#### clearAll()

Clear all cached avatar resources.

```typescript theme={null}
AvatarManager.shared.clearAll()
```

***

## AvatarView

3D rendering view that displays the avatar.

```typescript theme={null}
import { AvatarView } from '@spatialwalk/avatarkit'
```

### Constructor

```typescript theme={null}
const avatarView = new AvatarView(avatar, container)
```

| Parameter   | Type          | Description                                     |
| ----------- | ------------- | ----------------------------------------------- |
| `avatar`    | `Avatar`      | Loaded avatar object                            |
| `container` | `HTMLElement` | Container element (canvas auto-fills container) |

<Warning>
  **⚠️ Important:** The container element MUST have non-zero width and height. The canvas will automatically fill the container size.
</Warning>

### Instance Properties

| Property           | Type               | Description                       |
| ------------------ | ------------------ | --------------------------------- |
| `controller`       | `AvatarController` | Communication controller          |
| `onFirstRendering` | `() => void`       | Callback when first frame renders |
| `avatarTransform`  | `{ x, y, scale }`  | Avatar position and scale         |

### Instance Methods

#### dispose()

Clean up view resources. Call when view is no longer needed.

```typescript theme={null}
avatarView.dispose()
```

***

## AvatarController

Handles real-time communication with the avatar service.

```typescript theme={null}
// Accessed via AvatarView
const controller = avatarView.controller
```

### Instance Properties

| Property            | Type                | Description                |
| ------------------- | ------------------- | -------------------------- |
| `connectionState`   | `ConnectionState`   | Current connection state   |
| `conversationState` | `ConversationState` | Current conversation state |

### Event Callbacks

```typescript theme={null}
// Connection state changes
controller.onConnectionState = (state: ConnectionState) => {
  // 'disconnected' | 'connecting' | 'connected' | 'failed'
}

// Conversation state changes
controller.onConversationState = (state: ConversationState) => {
  // 'idle' | 'playing' | 'pausing'
}

// Error handling
controller.onError = (error: AvatarError) => {
  console.error('Error:', error.code, error.message)
}
```

### Instance Methods

#### initializeAudioContext()

Initialize audio context. **MUST be called in user gesture context.**

```typescript theme={null}
button.addEventListener('click', async () => {
  await controller.initializeAudioContext()
})
```

#### start()

Connect to the avatar service. SDK mode only.

```typescript theme={null}
await controller.start()
```

#### send(audioData, end)

Send audio data to server. SDK mode only.

```typescript theme={null}
// Send audio data (continue)
controller.send(audioData, false)

// Send final audio data (end conversation)
controller.send(audioData, true)
```

| Parameter   | Type          | Description                                                                       |
| ----------- | ------------- | --------------------------------------------------------------------------------- |
| `audioData` | `ArrayBuffer` | PCM16 (S16LE) mono audio data. Byte length **must** be even (2 bytes per sample). |
| `end`       | `boolean`     | Whether this is the final chunk                                                   |

<Warning>
  Audio data byte length must be even (2 bytes per sample). Odd-length data will cause a server-side validation error and WebSocket disconnect.
</Warning>

**Returns:** `string` - Conversation ID

#### close()

Close service connection. SDK mode only.

```typescript theme={null}
controller.close()
```

### Host Mode Methods

For Host Mode (`DrivingServiceMode.host`) only:

#### yieldAudioData(audioData, end)

Provide audio data in Host Mode.

```typescript theme={null}
const conversationId = controller.yieldAudioData(audioData, false)
```

#### yieldFramesData(frames, conversationId)

Provide animation frames in Host Mode.

```typescript theme={null}
controller.yieldFramesData(animationDataArray, conversationId)
```

### Common Methods (Both Modes)

#### pause()

Pause current playback.

```typescript theme={null}
controller.pause()
```

#### resume()

Resume paused playback.

```typescript theme={null}
await controller.resume()
```

#### interrupt()

Interrupt current playback and clear data.

```typescript theme={null}
controller.interrupt()
```

#### clear()

Clear all data and resources.

```typescript theme={null}
controller.clear()
```

#### getCurrentConversationId()

Get the current active conversation ID.

```typescript theme={null}
const conversationId = controller.getCurrentConversationId()
// Returns: string | null
```

#### volume

Playback volume property (avatar audio only, not system volume). Range: 0.0 to 1.0.

```typescript theme={null}
controller.volume = 0.5              // Set to 50%
const currentVolume = controller.volume  // Get current volume
```

***

## Types and Enums

### Configuration

```typescript theme={null}
interface Configuration {
  environment: Environment
  drivingServiceMode?: DrivingServiceMode  // Default: 'sdk'
  logLevel?: LogLevel                       // Default: 'off'
  audioFormat?: AudioFormat                 // Default: { channelCount: 1, sampleRate: 16000 }
}
```

### Environment

```typescript theme={null}
enum Environment {
  cn = 'cn',      // China region
  intl = 'intl'   // International region
}
```

### DrivingServiceMode

```typescript theme={null}
enum DrivingServiceMode {
  sdk = 'sdk',    // SDK handles WebSocket communication
  host = 'host'   // Host provides audio and animation data
}
```

### LogLevel

```typescript theme={null}
enum LogLevel {
  off = 'off',
  error = 'error',
  warning = 'warning',
  all = 'all'
}
```

### ConnectionState

```typescript theme={null}
enum ConnectionState {
  disconnected = 'disconnected',
  connecting = 'connecting',
  connected = 'connected',
  failed = 'failed'
}
```

### ConversationState

```typescript theme={null}
enum ConversationState {
  idle = 'idle',        // Breathing animation
  playing = 'playing',  // Active conversation
  paused = 'paused'     // Paused during playback
}
```

### LoadProgressInfo

```typescript theme={null}
type LoadProgressInfo = {
  type: 'downloading' | 'completed' | 'failed'
  progress?: number  // 0-100
  error?: Error
}
```

### AudioFormat

```typescript theme={null}
interface AudioFormat {
  channelCount: 1  // Fixed to mono
  sampleRate: number  // 8000 | 16000 | 22050 | 24000 | 32000 | 44100 | 48000
}
```

***

## Error Codes

| Code                           | Description                                 | Recovery                                                         |
| ------------------------------ | ------------------------------------------- | ---------------------------------------------------------------- |
| `appIDUnrecognized`            | App ID not recognized.                      | Verify your app ID in the dashboard.                             |
| `sessionTokenInvalid`          | Session token is invalid or appId mismatch. | Verify token generation on your server.                          |
| `sessionTokenExpired`          | Session token has expired.                  | Request a new token from your server.                            |
| `insufficientBalance`          | Account balance is insufficient.            | Top up your account in the dashboard.                            |
| `sessionTimeout`               | Session timed out.                          | Reconnect by calling `start()` again.                            |
| `concurrentLimitExceeded`      | Concurrent connection limit exceeded.       | Close other active sessions.                                     |
| `avatarIDUnrecognized`         | Avatar ID not found.                        | Check the avatar ID exists and is published.                     |
| `failedToFetchAvatarMetadata`  | Failed to fetch avatar metadata.            | Check network connectivity and retry.                            |
| `failedToDownloadAvatarAssets` | Failed to download avatar assets.           | Check network connectivity and retry.                            |
| `websocketError`               | WebSocket connection error.                 | Check network and retry.                                         |
| `websocketClosedAbnormally`    | WebSocket closed abnormally.                | Retry connection.                                                |
| `serverError`                  | Server-side error.                          | Check the error message for details.                             |
| `audioContextNotInitialized`   | Audio context not initialized.              | Call `initializeAudioContext()` in a user gesture handler first. |

***

## Complete Usage Example

```typescript theme={null}
import {
  AvatarSDK,
  AvatarManager,
  AvatarView,
  Environment,
  ConnectionState,
  ConversationState,
} from '@spatialwalk/avatarkit'

class AvatarApp {
  private avatarView: AvatarView | null = null

  async init(appId: string, sessionToken: string, avatarId: string, container: HTMLElement) {
    // Initialize SDK
    await AvatarSDK.initialize(appId, { environment: Environment.cn })
    AvatarSDK.setSessionToken(sessionToken)

    // Load avatar
    const avatar = await AvatarManager.shared.load(avatarId)
    if (!avatar) throw new Error('Failed to load avatar')

    // Create view
    this.avatarView = new AvatarView(avatar, container)
    this.avatarView.onFirstRendering = () => {
      console.log('First frame rendered')
    }

    // Set up handlers
    this.avatarView.controller.onConnectionState = (state) => {
      console.log('Connection:', state)
    }
    this.avatarView.controller.onConversationState = (state) => {
      console.log('Conversation:', state)
    }
    this.avatarView.controller.onError = (error) => {
      console.error('Error:', error)
    }
  }

  // Must be called in user gesture context
  async start() {
    await this.avatarView?.controller.initializeAudioContext()
    await this.avatarView?.controller.start()
  }

  send(audioData: ArrayBuffer, isEnd: boolean) {
    this.avatarView?.controller.send(audioData, isEnd)
  }

  interrupt() {
    this.avatarView?.controller.interrupt()
  }

  dispose() {
    this.avatarView?.controller.close()
    this.avatarView?.dispose()
    this.avatarView = null
  }
}
```


# AvatarKit UI
Source: https://docs.spatialreal.ai/sdk-reference/web-sdk/avatarkit-ui

SpatialReal avatar components for React.

AvatarKit UI is a shadcn-style React component package for embedding a SpatialReal avatar player into your app. It wraps AvatarKit RTC setup, LiveKit session wiring, loading and error states, and a small set of presentational primitives so you can ship an avatar surface with minimal glue code.

## Usage

Use `SpatialRealAvatarProvider` to initialize AvatarKit and LiveKit, then compose the avatar surface with `SpatialRealAvatarFrame`, `SpatialRealAvatarCanvas`, `SpatialRealAvatarLoading`, `SpatialRealAvatarError`, and `SpatialRealAvatarStatus`.

```tsx theme={null}
import {
  SpatialRealAvatarCanvas,
  SpatialRealAvatarError,
  SpatialRealAvatarFrame,
  SpatialRealAvatarLoading,
  SpatialRealAvatarProvider,
  SpatialRealAvatarStatus,
} from '@/components/spatialreal-avatar'

export function AgentAvatar({ token, url, roomName }: {
  token: string
  url: string
  roomName: string
}) {
  return (
    <SpatialRealAvatarProvider
      appId={import.meta.env.VITE_SPATIALREAL_APP_ID}
      avatarId={import.meta.env.VITE_SPATIALREAL_AVATAR_ID}
      connection={{ token, url, roomName }}
    >
      <SpatialRealAvatarFrame className="overflow-hidden">
        <SpatialRealAvatarCanvas className="min-h-[520px]" />
        <SpatialRealAvatarLoading />
        <SpatialRealAvatarError />

        <div className="absolute inset-x-0 top-0 flex justify-end p-4">
          <SpatialRealAvatarStatus />
        </div>
      </SpatialRealAvatarFrame>
    </SpatialRealAvatarProvider>
  )
}
```

This is the same composition pattern used in the working Vite demo: an avatar frame for rendering, built-in overlays for loading and failure states, and context-driven controls for microphone and session UI.

### Adding microphone controls with context

Use `useSpatialRealAvatarContext()` inside the provider tree when you want to drive UI from connection state or publish the local microphone into the avatar session.

```tsx theme={null}
import { useState } from 'react'

import {
  SpatialRealAvatarCanvas,
  SpatialRealAvatarError,
  SpatialRealAvatarFrame,
  SpatialRealAvatarLoading,
  SpatialRealAvatarProvider,
  SpatialRealAvatarStatus,
  useSpatialRealAvatarContext,
} from '@/components/spatialreal-avatar'

function AvatarPanel() {
  const avatar = useSpatialRealAvatarContext()
  const [pending, setPending] = useState(false)

  const toggleMicrophone = async () => {
    if (pending) return

    setPending(true)

    try {
      if (avatar.isPublishingMicrophone) {
        await avatar.stopPublishingMicrophone()
      } else {
        await avatar.startPublishingMicrophone()
      }
    } finally {
      setPending(false)
    }
  }

  return (
    <div className="grid gap-6 lg:grid-cols-[minmax(0,420px)_minmax(0,1fr)]">
      <section className="space-y-4">
        <SpatialRealAvatarFrame>
          <SpatialRealAvatarCanvas />
          <SpatialRealAvatarLoading />
          <SpatialRealAvatarError />
        </SpatialRealAvatarFrame>

        <div className="flex items-center justify-between rounded-2xl border p-3">
          <button
            disabled={!avatar.isConnected || pending}
            onClick={() => void toggleMicrophone()}
            type="button"
          >
            {avatar.isPublishingMicrophone ? 'Mute mic' : 'Enable mic'}
          </button>

          <SpatialRealAvatarStatus />
        </div>
      </section>

      <section className="rounded-3xl border p-4">
        {avatar.room
          ? 'Render your LiveKit transcript or agent UI here.'
          : 'Waiting for the LiveKit session to become available...'}
      </section>
    </div>
  )
}

export function VoiceAgentAvatar(props: {
  token: string
  url: string
  roomName: string
}) {
  return (
    <SpatialRealAvatarProvider
      appId={import.meta.env.VITE_SPATIALREAL_APP_ID}
      avatarId={import.meta.env.VITE_SPATIALREAL_AVATAR_ID}
      connection={props}
    >
      <AvatarPanel />
    </SpatialRealAvatarProvider>
  )
}
```

<Note>
  `SpatialRealAvatarCanvas` must render inside a container with non-zero width and height. The provider waits for the canvas container to be measurable before it starts loading the avatar.
</Note>

## Features

* Render a SpatialReal avatar surface with a small set of composable shadcn-style primitives.
* Connect AvatarKit and LiveKit through `SpatialRealAvatarProvider` without writing your own player wiring.
* Show built-in loading, error, and connection status UI out of the box.
* Access the active LiveKit room and avatar session state through `useSpatialRealAvatarContext()`.
* Start, stop, reconnect, disconnect, and publish microphone audio with hook-based controls.

## Installation

`AvatarKit UI` is built on:

* **shadcn/ui** conventions (local component files + Tailwind styles)
* **[LiveKit Agent UI](https://livekit.com/products/agents-ui) ecosystem** (`@livekit/components-react`, `@livekit/components-styles`)

Before adding this AvatarKit UI component to your project, you can use this [guide](https://ui.shadcn.com/docs/installation/manual) to setup shadcn

```bash theme={null}
npx shadcn@latest init
npx shadcn@latest add https://ui.spatialreal.ai/r/spatialreal-avatar.json
```

## API Reference

### Exports

```tsx theme={null}
export {
  SpatialRealAvatarCanvas,
  SpatialRealAvatarError,
  SpatialRealAvatarFrame,
  SpatialRealAvatarLoading,
  SpatialRealAvatarProvider,
  SpatialRealAvatarStatus,
  useSpatialRealAvatar,
  useSpatialRealAvatarContext,
}
```

### `SpatialRealAvatarProvider`

Initializes AvatarKit, connects the LiveKit-backed avatar stream, provides avatar state through React context, and renders remote room audio.

| Prop                  | Type                                                  | Description                                                                                             |
| --------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `children`            | `ReactNode`                                           | Provider contents. All AvatarKit UI primitives that read context must be rendered inside this provider. |
| `appId`               | `string`                                              | Your AvatarKit app ID. Required.                                                                        |
| `avatarId`            | `string`                                              | The avatar identifier to load. Required.                                                                |
| `connection`          | `{ roomName: string; token: string; url: string }`    | LiveKit connection details for the avatar stream. Required.                                             |
| `enabled`             | `boolean`                                             | Defaults to `true`. When `false`, the provider does not initialize or connect the avatar.               |
| `publishMicrophone`   | `boolean`                                             | Defaults to `false`. When `true`, the provider starts publishing the local microphone after connection. |
| `muted`               | `boolean`                                             | Mutes audio rendered by `RoomAudioRenderer`.                                                            |
| `volume`              | `number`                                              | Playback volume passed to `RoomAudioRenderer`.                                                          |
| `characterApiBaseUrl` | `string`                                              | Optional override for the AvatarKit character API base URL.                                             |
| `drivingServiceMode`  | `DrivingServiceMode`                                  | AvatarKit driving mode. Defaults to `DrivingServiceMode.host`.                                          |
| `environment`         | `Environment`                                         | AvatarKit environment. Defaults to `Environment.intl`.                                                  |
| `sdkLogLevel`         | `LogLevel`                                            | AvatarKit SDK log level.                                                                                |
| `sessionToken`        | `string`                                              | Optional AvatarKit session token to inject into `AvatarSDK`.                                            |
| `userId`              | `string`                                              | Optional end-user identifier passed to `AvatarSDK.setUserId()`.                                         |
| `playerOptions`       | `AvatarPlayerOptions`                                 | Options forwarded to `AvatarPlayer`.                                                                    |
| `onConnected`         | `(room: Room \| null) => void`                        | Called once the avatar player connects and the LiveKit room is available.                               |
| `onDisconnected`      | `() => void`                                          | Called when the avatar player disconnects unexpectedly or returns to idle.                              |
| `onAvatarError`       | `(error: Error) => void`                              | Called whenever avatar initialization, connection, reconnect, or microphone publishing fails.           |
| `onLoadProgress`      | `(progress: LoadProgressInfo) => void`                | Reports avatar asset download progress from `AvatarManager.shared.load()`.                              |
| `onStateChange`       | `(status: SpatialRealAvatarConnectionStatus) => void` | Called whenever connection status changes.                                                              |
| `onDisconnect`        | `() => void`                                          | Called after `session.end()` or `avatar.disconnect()` completes through the provider session wrapper.   |

### `SpatialRealAvatarFrame`

Visual wrapper for the avatar canvas and overlays.

| Prop        | Type                              | Description                                                         |
| ----------- | --------------------------------- | ------------------------------------------------------------------- |
| `tone`      | `'default' \| 'muted' \| 'ghost'` | Visual treatment for the frame background. Defaults to `'default'`. |
| `className` | `string`                          | Additional classes for layout and styling.                          |
| `...props`  | `HTMLAttributes<HTMLDivElement>`  | Standard div attributes such as `id`, `style`, and event handlers.  |

### `SpatialRealAvatarCanvas`

Mount target for `AvatarView`. The rendered avatar appears inside this div.

| Prop        | Type                             | Description                                                |
| ----------- | -------------------------------- | ---------------------------------------------------------- |
| `minHeight` | `CSSProperties['minHeight']`     | Minimum canvas height. Defaults to `420`.                  |
| `className` | `string`                         | Additional classes for sizing and styling.                 |
| `...props`  | `HTMLAttributes<HTMLDivElement>` | Standard div attributes forwarded to the canvas container. |

### `SpatialRealAvatarLoading`

Loading overlay shown while the avatar is initializing or connecting.

| Prop        | Type                             | Description                                                                                                  |
| ----------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `children`  | `ReactNode`                      | Optional custom loading UI. When omitted, the default card shows current status and asset download progress. |
| `className` | `string`                         | Additional classes for the overlay container.                                                                |
| `...props`  | `HTMLAttributes<HTMLDivElement>` | Standard div attributes forwarded to the overlay container.                                                  |

### `SpatialRealAvatarError`

Error overlay shown when avatar setup or streaming fails.

| Prop        | Type                             | Description                                                                        |
| ----------- | -------------------------------- | ---------------------------------------------------------------------------------- |
| `children`  | `ReactNode`                      | Optional custom error UI. When omitted, the default card displays `error.message`. |
| `className` | `string`                         | Additional classes for the overlay container.                                      |
| `...props`  | `HTMLAttributes<HTMLDivElement>` | Standard div attributes forwarded to the overlay container.                        |

### `SpatialRealAvatarStatus`

Small status badge for the current avatar connection state.

| Prop        | Type                             | Description                                             |
| ----------- | -------------------------------- | ------------------------------------------------------- |
| `className` | `string`                         | Additional classes for the badge.                       |
| `...props`  | `HTMLAttributes<HTMLDivElement>` | Standard div attributes forwarded to the badge wrapper. |

Supported status labels:

* `idle`
* `initializing`
* `connecting`
* `connected`
* `disconnecting`
* `error`

### `useSpatialRealAvatar(options)`

Low-level hook that powers the provider. Use it when you want to wire AvatarKit into your own context or compose a custom provider.

#### Options

`useSpatialRealAvatar()` accepts the same connection and SDK props as `SpatialRealAvatarProvider`, except for `children`, `muted`, `volume`, and `onDisconnect`.

```tsx theme={null}
const avatar = useSpatialRealAvatar({
  appId,
  avatarId,
  connection: {
    roomName,
    token,
    url,
  },
  publishMicrophone: false,
})
```

#### Return value

| Field                         | Type                                     | Description                                                                             |
| ----------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- |
| `status`                      | `SpatialRealAvatarConnectionStatus`      | Current lifecycle state.                                                                |
| `isLoading`                   | `boolean`                                | `true` while status is `initializing` or `connecting`.                                  |
| `isConnected`                 | `boolean`                                | `true` when the avatar player is connected.                                             |
| `error`                       | `Error \| null`                          | The most recent connection or runtime error.                                            |
| `downloadProgress`            | `number \| null`                         | Asset download progress from `0` to `1`, or `null` when unavailable.                    |
| `isPublishingMicrophone`      | `boolean`                                | Indicates whether the local microphone track is currently published.                    |
| `micTrack`                    | `Track \| undefined`                     | The published local microphone track, when available.                                   |
| `room`                        | `Room \| null`                           | LiveKit room instance backing the avatar session.                                       |
| `connection`                  | `SpatialRealAvatarConnection`            | The current connection config passed into the hook.                                     |
| `containerRef`                | `(node: HTMLDivElement \| null) => void` | Ref callback that must be attached to the avatar canvas element.                        |
| `session`                     | `UseSessionReturn`                       | Session object from `@livekit/components-react`, with `end()` mapped to `disconnect()`. |
| `disconnect()`                | `() => Promise<void>`                    | Disconnects the avatar stream and resets state to idle.                                 |
| `reconnect()`                 | `() => Promise<void>`                    | Attempts to reconnect the existing avatar player.                                       |
| `startPublishingMicrophone()` | `() => Promise<void>`                    | Publishes the local microphone into the avatar session.                                 |
| `stopPublishingMicrophone()`  | `() => Promise<void>`                    | Stops microphone publishing.                                                            |

### `useSpatialRealAvatarContext()`

Returns the current avatar state from `SpatialRealAvatarProvider`.

```tsx theme={null}
const avatar = useSpatialRealAvatarContext()
```

Use this hook inside provider children to:

* show or hide UI based on `avatar.status`
* read the connected `avatar.room` for LiveKit chat and transcript components
* call `avatar.startPublishingMicrophone()` and `avatar.stopPublishingMicrophone()`
* invoke `avatar.disconnect()` or `avatar.reconnect()` from your own controls

<Warning>
  `useSpatialRealAvatarContext()` throws an error if it is used outside `SpatialRealAvatarProvider`.
</Warning>


# Changelog
Source: https://docs.spatialreal.ai/sdk-reference/web-sdk/changelog

AvatarKit Web SDK version update history

## 1.0.0-beta.97

### ⚡ Breaking Changes

* **`transform` renamed to `avatarTransform`** — `avatarView.transform` is now `avatarView.avatarTransform`. Aligned naming across Web, Android, and iOS SDKs.

### ✨ Features

* **Avatar transform from character settings** — Avatar automatically applies position and scale from character settings. User-set `avatarTransform` stacks on top of the base values.

### 🐛 Bugfixes

* **Rendering quality improvement** — Improved color accuracy for avatar rendering.

## 1.0.0-beta.96

### 🐛 Bugfixes

* **WASM loading compatibility** — Fixed WASM loading failure when consumed via Vite dev server.

## 1.0.0-beta.95

### 🐛 Bugfixes

* **Rendering quality fix** — Fixed blurry splat edges on arms/hands in WebGPU renderer.

### ⚡ Performance

* **WASM download reliability** — Added integrity validation and automatic retry (up to 3 attempts) for WASM module loading. Prevents loading failures from truncated or corrupted downloads.

## 1.0.0-beta.94

### 🐛 Bugfixes

* **Tab switch rendering fix** — Fixed avatar rendering corruption when switching back to a tab (WebGPU).

### ✨ Features

* **WebSocket close code handling** — Added handling for server custom close codes: auth failure, insufficient balance, session timeout, concurrent limit exceeded. Non-recoverable errors no longer trigger reconnection attempts.
* **Error transparency** — All non-normal close events are now propagated to `onError` callback.
* **New ErrorCode values** — Added `insufficientBalance`, `sessionTimeout`, `concurrentLimitExceeded`.

## 1.0.0-beta.92

### ✨ Features

* **Host mode animation end signal** — `yieldFramesData()` now returns a `boolean` indicating whether the server has sent all animation data for the current conversation.

## 1.0.0-beta.91

### 🐛 Bugfixes

* **Vite consumer build fix** — Fixed build failure when consuming the SDK in Vite projects.

## 1.0.0-beta.90

### 🐛 Bugfixes

* **Compressed model support fix** — Fixed `useCompressedModel` failing to load compressed point cloud models.

## 1.0.0-beta.89

### ⚡ Breaking Changes

* **Enum rename** — `ConversationState.pausing` renamed to `ConversationState.paused` for cross-SDK consistency.
* **FrameRateInfo** — Unified as pre-computed fields for cross-SDK consistency.

### ✨ Features

* **Frame rate monitor** — Unified public API for frame rate monitoring with enable/disable switch.
* **Device benchmark** — `deviceScore` and `isDeviceSupported` APIs for runtime device capability assessment.
* **Compressed model (Beta)** — `load()` accepts optional `useCompressedModel` parameter for loading compressed models (\~30% smaller). Please verify rendering quality before using in production.

### ⚡ Performance

* **Playback scheduling improvement** — Improved frame pacing accuracy during speaking state.

## 1.0.0-beta.88

### 🐛 Bugfixes

* **Build packaging fix** — Fixed WASM binary missing from npm package in beta.87.

## 1.0.0-beta.87

### ⚡ Performance

* **Playback smoothness** — Significantly reduced frame jank during avatar speaking. Jank rate reduced \~95% compared to previous version.

### 🐛 Bugfixes

* **Frame pacing precision** — Fixed occasional frame skipping at frame boundaries during playback.
* **Duplicate initialize guard** — Prevented duplicate `initialize()` calls from causing unexpected state.

## 1.0.0-beta.86

### 🐛 Bugfixes

* **Transition smoothness** — Fixed first-frame jump when transitioning between idle and speaking states.

## 1.0.0-beta.82

### 🔧 Improvements

* **Protocol update** — Updated to latest communication protocol version.

## 1.0.0-beta.80

* **Next.js Plugin** — Added `withAvatarkit` wrapper for Next.js projects to automate WASM file configuration.

## 1.0.0-beta.75

* **Local Log Upload** — Automatic local log upload for debugging and troubleshooting.

## 1.0.0-beta.71

* **API Refinement** — Improved API structure by hiding internal implementation details from public API.

## 1.0.0-beta.64

* **Vite Plugin** — Added `avatarkitVitePlugin` to automate WASM file configuration.

## 1.0.0-beta.63

* **Audio Context API** — Added `initializeAudioContext()` method that must be called in user gesture context.

## 1.0.0-beta.55

* **Eye Tracking Control** — Added `eyeTrackingEnabled` parameter to `PostProcessingConfig`.
* **Point Count API** — Added `getPointCount()` method to `AvatarController`.

## 1.0.0-beta.51

* **Post-Processing Enhancement** — Added `translation` and `eyePose` fields to `PostProcessingConfig`.
* **Bugfix** — Fixed avatar settings update issue when using cached avatars.

## 1.0.0-beta.45

* **Immediate Rendering** — Post-processing and camera updates apply immediately when playback is paused.
* **API Change** — Replaced `ready` Promise with `onFirstRendering` callback.

## 1.0.0-beta.36

* **Audio Configuration** — Added configurable audio sample rate support (8000-48000 Hz, default: 16000).

## 1.0.0-beta.35

* **API Change** — Changed `setTransform()` method to `transform` getter/setter property.

## 1.0.0-beta.29

* **Breaking Changes** — Removed `Environment.test`, changed default log level to `off`.
* **New Features** — Added avatar caching methods (`retrieve`, `clear`, `clearAll`).
* **Background Image** — Added background image control in `AvatarView`.

## 1.0.0-beta.28

* **Breaking Change** — Renamed `AvatarKit` to `AvatarSDK`. Update imports accordingly.

## 1.0.0-beta.22

* **API Changes** — Renamed `onAvatarState` to `onConversationState`, `Environment.us` to `Environment.intl`.
* **Volume Control** — Added `setVolume()` and `getVolume()` methods.

## 1.0.0-beta.17

* **Fallback Mechanism** — Added automatic audio-only fallback when animation data is unavailable.

## 1.0.0-beta.3

* **Architecture** — Initial release with SDK mode and Host mode support.


# API Keys
Source: https://docs.spatialreal.ai/studio/aki-key

Create and manage API credentials

API Keys are the foundation of your SpatialReal integration. They allow you to authenticate your applications and securely access SpatialReal services. In this guide, you'll learn how to create API Keys in [SpatialReal Studio](https://app.spatialreal.ai) and best practices for managing them.

## Creating an API Key

<Steps>
  <Step title="Open API Keys page">
    Go to [https://app.spatialreal.ai/apps](https://app.spatialreal.ai/apps).
  </Step>

  <Step title="Click Create API Key">
    On the API key page, click **Create API Key**.

    <Frame>
      <img alt="API Keys page" />
    </Frame>
  </Step>

  <Step title="Enter application name">
    Type your **Application Name** and click **Create**.

    <Frame>
      <img alt="Click Create API Key" />
    </Frame>
  </Step>

  <Step title="Copy App ID and API Key">
    In the popup, copy both **App ID** and **API Key**.

    <Frame>
      <img alt="Create app and view popup" />
    </Frame>
  </Step>
</Steps>

***

## Manage your API Key

Your API Key is used to authenticate server-side requests to SpatialReal services.

<Warning>
  **Important:** Keep your API Key secure and never expose it in client-side code or commit it to version control. Treat it like a password to prevent unauthorized access to your SpatialReal account.
</Warning>

**Best Practices:**

* Store your API Key in environment variables or a secrets manager
* Never expose your API Key in client-side code
* Never commit your API Key to version control

***

## Temporary Session Token

For quick testing and development, you can generate a **Temporary Session Token** directly in [SpatialReal Studio](https://app.spatialreal.ai).

<Note>
  Temporary Session Tokens are valid for **24 hours** only. They are intended for testing purposes and should not be used in production.
</Note>

<Steps>
  <Step title="Open Your Application">
    Navigate to [API keys page in SpatialReal Studio](https://app.spatialreal.ai/apps), then click the details button of your application.

    <Frame>
      <img alt="Token Step 1" />
    </Frame>
  </Step>

  <Step title="Generate Token">
    Click **Generate Temporary Token** to create a new temporary session token.

    <Frame>
      <img alt="Token Step 2" />
    </Frame>
  </Step>

  <Step title="Use for Testing">
    Use this token in your client SDK to test avatar rendering without setting up server-side authentication.
  </Step>
</Steps>

***

## Production Authentication

In production, you should generate session tokens from your backend server using the API Key. See the [Authentication Guide](/server/auth) for implementation details.

<Card title="Authentication Guide" icon="lock" href="/server/auth">
  Learn how to implement secure server-side authentication
</Card>


# Public Avatar
Source: https://docs.spatialreal.ai/studio/public-avatar

Browse and use pre-built avatars from the SpatialReal library

## What are Public Avatars?

Public Avatars are a collection of **ready-to-use, high-quality stock avatars** provided by SpatialReal. These avatars are professionally designed and optimized for immediate integration into your applications — no creation or customization needed.

<Frame>
  <img alt="avatars" />
</Frame>

## Accessing Public Avatars

Navigate to the **Public Avatar** page in [SpatialReal Studio](https://app.spatialreal.ai/avatars/library) to browse the complete library of available avatars.

## Using a Public Avatar

<Steps>
  <Step title="Browse the Gallery">
    Explore all available stock avatars displayed as avatar cards in the gallery.
  </Step>

  <Step title="Copy Avatar ID">
    Hover over an avatar card and click the **copy icon** in the **top-right corner** to copy the avatar's unique ID to your clipboard.
  </Step>

  <Step title="Integrate into Your App">
    Use the copied avatar ID in your application code to render the avatar. You can use [guide](/guide/sdk-mode) to see how to do this.
  </Step>
</Steps>

## Next Steps

<CardGroup>
  <Card title="SDK Integration" icon="code" href="/guide/introduction">
    Learn how to integrate avatars into your app
  </Card>
</CardGroup>