External Events & Multi-Stream (WebSocket)

Send external events alongside user input on WebSocket connections for multi-stream apps.

Amigo WebSockets support multiple information streams on a single connection, so you can build rich, interactive apps. In addition to user text or audio, you can send external events (for example, device telemetry, UI actions, page or navigation changes) that the agent incorporates into its next response.

This page focuses on how to publish external events alongside user input, and how they are associated with agent interactions.

Why Multi-Stream?

• Device information streams: battery, network, and orientation updates for mobile assistants.
• UI interaction streams: button clicks, navigation, selection changes.
• Context updates: screen state, cart changes, presence or location signals.
• Real-time apps: send events while a user is speaking (VAD or manual audio streaming).

Streams on the Connection

On a single WebSocket you can interleave these client → server messages:

• client.new-text-message with message_type: 'user-message' (user text)
• client.new-audio-message (user audio streaming) and client.new-audio-message with audio: null (end of audio)
• client.new-text-message with message_type: 'external-event' (external events)
• Control messages: client.switch-vad-mode, client.extend-timeout, client.finish-conversation, client.close-connection

And you will receive these server → client streams:

• server.new-message (text chunks or base64 audio chunks)
• server.interaction-complete (marks the end of a turn)
• server.current-agent-action (optional; controlled by query filter)
• VAD events: server.vad-speech-started, server.vad-speech-ended, server.vad-speech-reset-zero, server.vad-mode-switched

Association and Ordering

• External events are timestamped and attached to the next interaction.
• Any external events sent before or during a user's input (text or audio) become context for that interaction.
• After an interaction completes, the external-event buffer is cleared.
• You can also start an interaction using only an external event (no user text or audio).

Sequence Diagram

VAD Mode Sequence

Minimal VAD Mode Example (messages sent)

// 1) Enable VAD mode
ws.send(JSON.stringify({
  type: 'client.switch-vad-mode',
  vad_mode_on: true
}));

// 2) Continuously stream PCM audio chunks (16kHz mono)
function onPcmFrame(pcmBase64) {
  ws.send(JSON.stringify({
    type: 'client.new-audio-message',
    audio: pcmBase64
  }));
}

// 3) Send external events during speech; they attach to the same interaction
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'ui.navigate', page: '/checkout' }),
  start_interaction: false
}));

ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'cart.add', sku: 'SKU-123' }),
  start_interaction: false
}));

// 4) Receive server.new-message chunks and server.interaction-complete as usual
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'server.new-message') {
    // handle text/audio chunk
  } else if (msg.type === 'server.interaction-complete') {
    // end of turn; external events above are associated with this interaction
  }
};

Send External Events

Send external context as structured text alongside the conversation. We recommend JSON-string payloads so your agent can parse event types and data.

// Example: send a device telemetry update
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'device.update',
    battery: 72,
    network: 'wifi',
    orientation: 'landscape'
  }),
  start_interaction: false
}));

// Example: send a UI action
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'ui.click',
    target: 'checkout_button',
    page: '/cart'
  }),
  start_interaction: false
}));

Notes:

• text is a string. Using JSON.stringify(...) keeps a consistent, parseable structure.
• The server records these as external-event messages with timestamps for the interaction.

Use With Audio Streaming

You can interleave external events while streaming audio. Events sent between the start and end of a user's audio are attached to that same interaction.

// 1) Start audio stream (first chunk includes config)
ws.send(JSON.stringify({
  type: 'client.new-audio-message',
  audio: firstBase64PcmChunk,
  audio_config: { type: 'pcm', frame_rate: 16000, sample_width: 2, n_channels: 1 }
}));

// 2) Interleave external events while the user is speaking
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({ event: 'ui.navigate', page: '/checkout' }),
  start_interaction: false
}));

// 3) Continue sending audio chunks
ws.send(JSON.stringify({ type: 'client.new-audio-message', audio: nextBase64PcmChunk }));

// 4) Signal end of audio
ws.send(JSON.stringify({ type: 'client.new-audio-message', audio: null }));

VAD mode is also supported. When client.switch-vad-mode enables VAD, you continuously stream PCM audio. The server detects speech boundaries, and any external events sent during speech are still attached to that interaction.

Start With Only an External Event

Kick off an interaction without user text or audio by sending an external event as the first message:

ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: 'ALERT: Payment gateway timeout rate spiking above threshold',
  start_interaction: true  // Required on external events: true starts (or interrupts) an interaction, false buffers the event for the next interaction
}));

Interrupting Interactions in VAD Mode

When in VAD mode, external events with start_interaction: true can interrupt ongoing conversations:

// VAD mode enabled, agent is responding...

// Send urgent external event
ws.send(JSON.stringify({
  type: 'client.new-text-message',
  message_type: 'external-event',
  text: JSON.stringify({
    event: 'system.alert',
    severity: 'critical',
    message: 'Database connection lost'
  }),
  start_interaction: true  // Interrupts current interaction
}));

// Behavior depends on user speech state:
// - If agent hasn't detected user speaking: Interrupts any existing interaction, starts new interaction immediately
// - If user is speaking (agent has detected): Agent waits for speech to end, then triggers new interaction

Important Notes:

• start_interaction is required on every external-event message; messages that omit it are rejected as invalid and the connection is closed (code 1008).
• With start_interaction: false, external events are buffered and attached to the next natural interaction.
• With start_interaction: true in VAD mode:

  • Existing agent responses are interrupted when the user is not speaking.
  • The agent respects user speech, waiting until the user finishes before triggering the new interaction.
• This lets critical events take precedence while keeping natural conversation flow.

Receive Agent Actions (Optional)

For interactive UIs, you can subscribe to agent action telemetry and filter what you receive. Use the current_agent_action_type query parameter when connecting:

// Only receive tool-related actions
const filter = encodeURIComponent('^tool\\..*');
const url = `wss://api.amigo.ai/v1/${org}/conversation/converse_realtime?response_format=text&current_agent_action_type=${filter}`;
const ws = new WebSocket(url, [`bearer.authorization.amigo.ai.${token}`]);

ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === 'server.current-agent-action') {
    // e.g., show progress, update UI state
    renderAgentAction(msg.current_agent_action);
  }
};

Limits and Reliability

External-event messages count toward the same rate limits, timeouts, and message-size caps as every other message on the connection, and the same WebSocket close codes apply. See Performance & Limits and Error Handling on the Real-time Voice reference page:

Real-time Voice (WebSocket)

Best Practices

• Structure external events as JSON strings with an event name and minimal payload.
• Debounce high-frequency updates (for example, device telemetry every 3-5s or on meaningful change).
• Send events before or during the user's turn so they are applied to the next response.
• Use regional endpoints and PCM for the lowest latency in voice flows.

Related

• Real-time Voice (WebSocket)
• Conversation Events (HTTP)