External Events & Multi-Stream (WebSocket)

Amigo WebSockets support multiple information streams on a single connection so you can build rich, interactive apps. In addition to user text or audio, you can send external events (e.g., device telemetry, UI actions, page/navigation changes) that the agent incorporates in its next response.

This page focuses on how to publish external events alongside user input, and how they are associated with agent interactions.

Why Multi‑Stream?

• Device information streams: battery/network/orientation updates for mobile assistants

• UI interaction streams: button clicks, navigation, selection changes

• Context updates: screen state, cart changes, presence/location signals

• Real‑time apps: send events while a user is speaking (VAD or manual audio streaming)

Streams on the Connection

On a single WebSocket you can interleave these client → server messages:

• client.new-text-message with message_type: 'user-message' (user text)

• client.new-audio-message (user audio streaming) and client.new-audio-message with audio: null (end of audio)

• client.new-text-message with message_type: 'external-event' (external events)

• Control messages: client.switch-vad-mode, client.extend-timeout, client.finish-conversation, client.close-connection

And you will receive these server → client streams:

• server.new-message (text chunks or base64 audio chunks)

• server.interaction-complete (marks the end of a turn)

• server.current-agent-action (optional; controlled by query filter)

• VAD events: server.vad-speech-started, server.vad-speech-ended, server.vad-speech-reset-zero, server.vad-mode-switched

Association & Ordering

• External events are timestamped and attached to the next interaction.

• Any external events sent before or during a user’s input (text or audio) become context for that interaction.

• After an interaction completes, the external‑event buffer is cleared.

• You can also start an interaction using only an external event (no user text/audio).

Sequence Diagram

VAD Mode Sequence

Minimal VAD Mode Example (messages sent)

// 1) Enable VAD mode
ws.send(JSON.stringify({
  type: 'client.switch-vad-mode',
  vad_mode_on: true
}));

// 2) Continuously stream PCM audio chunks (16kHz mono)
function onPcmFrame(pcmBase64) {
  ws.send(JSON.stringify({
    type: 'client.new-audio-message',
    audio: pcmBase64
  }));
}

// 3) Send external events during speech; they attach to the same interaction
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'ui.navigate', page: '/checkout' })
}));

ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'cart.add', sku: 'SKU-123' })
}));

// 4) Receive server.new-message chunks and server.interaction-complete as usual
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'server.new-message') {
    // handle text/audio chunk
  } else if (msg.type === 'server.interaction-complete') {
    // end of turn; external events above are associated with this interaction
  }
};

Send External Events

Send external context as structured text alongside the conversation. We recommend JSON‑string payloads so your agent can parse event types and data.

// Example: send a device telemetry update
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'device.update',
    battery: 72,
    network: 'wifi',
    orientation: 'landscape'
  })
}));

// Example: send a UI action
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'ui.click',
    target: 'checkout_button',
    page: '/cart'
  })
}));

Notes:

• text is a string; using JSON.stringify(...) keeps a consistent, parseable structure.

• The server records these as external-event messages with timestamps for the interaction.

Use With Audio Streaming

You can interleave external events while streaming audio. Events sent between the start and end of a user’s audio are attached to that same interaction.

// 1) Start audio stream (first chunk includes config)
ws.send(JSON.stringify({
  type: 'client.new-audio-message',
  audio: firstBase64PcmChunk,
  audio_config: { format: 'pcm', sample_rate: 16000, sample_width: 2, n_channels: 1, frame_rate: 16000 }
}));

// 2) Interleave external events while the user is speaking
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'ui.navigate', page: '/checkout' })
}));

// 3) Continue sending audio chunks
ws.send(JSON.stringify({ type: 'client.new-audio-message', audio: nextBase64PcmChunk }));

// 4) Signal end of audio
ws.send(JSON.stringify({ type: 'client.new-audio-message', audio: null }));

VAD mode is also supported. When client.switch-vad-mode enables VAD, you continuously stream PCM audio; the server detects speech boundaries, and any external events sent during speech are still attached to that interaction.

Start With Only an External Event

Kick off an interaction without user text/audio by sending an external event as the first message:

ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: 'ALERT: Payment gateway timeout rate spiking above threshold',
  start_interaction: true  // Optional: explicitly start a new interaction
}));

Interrupting Interactions in VAD Mode

When in VAD mode, external events with start_interaction: true can interrupt ongoing conversations:

// VAD mode enabled, agent is responding...

// Send urgent external event
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'system.alert',
    severity: 'critical',
    message: 'Database connection lost'
  }),
  start_interaction: true  // Interrupts current interaction
}));

// Behavior depends on user speech state:
// - If agent hasn't detected user speaking: Interrupts any existing interaction, starts new interaction immediately
// - If user is speaking (agent has detected): Agent waits for speech to end, then triggers new interaction

Important Notes:

• Without start_interaction: true, external events are buffered and attached to the next natural interaction

• With start_interaction: true in VAD mode:

  • Interrupts existing agent responses when user is not speaking

  • Respects user speech - waits until user finishes before triggering the new interaction

• This allows critical events to take precedence while maintaining natural conversation flow

Receive Agent Actions (Optional)

For interactive UIs, you can subscribe to agent action telemetry and filter what you receive. Use the current_agent_action_type query parameter when connecting.

// Only receive tool-related actions
const filter = encodeURIComponent('^tool\\..*');
const url = `wss://api.amigo.ai/v1/${org}/conversation/converse_realtime?response_format=text&current_agent_action_type=${filter}`;
const ws = new WebSocket(url, [`bearer.authorization.amigo.ai.${token}`]);

ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'server.current-agent-action') {
    // e.g., show progress, update UI state
    renderAgentAction(msg.current_agent_action);
  }
};

Limits & Reliability

• Messages/minute: 60 across all message types

• Connection inactivity timeout: 30s (send client.extend-timeout every ~15s when idle)

• Max message size: 1 MB (e.g., for audio chunks)

• One active connection per user/service

• VAD requires PCM audio (MP3 not supported in VAD mode)

Best Practices

• Structure external events as JSON strings with an event name and minimal payload

• Debounce high‑frequency updates (e.g., device telemetry every 3–5s or on meaningful change)

• Send events before or during the user’s turn so they’re applied to the next response

• Use regional endpoints and PCM for lowest latency in voice flows

Troubleshooting

Common close codes:

Code	Meaning	Typical fix
3000	Unauthorized	Refresh credentials / token format (bearer.authorization.amigo.ai.{JWT})
3003	Forbidden	Check user permissions
3008	Timeout	Send client.extend-timeout periodically when idle
4000	Bad Request	Validate message structure and types
4004	Not Found	Verify organization/service/conversation IDs
4009	Conflict	Only 1 connection per user/service
4015	Unsupported Media	Use PCM for VAD, check audio config
4029	Rate Limited	Backoff and reduce message frequency

Related

• Real‑time Voice (WebSocket)

• Conversation Events (HTTP)