Voice

Voice note exchange over HTTP and real-time WebSocket voice with audio filler management.

Amigo supports two voice modes. Choose the one that matches your UX and latency needs:

Mode	Transport	Best for	Latency	Notes
Voice Notes (HTTP)	HTTP + NDJSON	Asynchronous push-to-talk, in-app voice replies	Low-to-medium	Upload a short clip; receive streamed TTS back
Real-time Voice (WebSocket)	WebSocket	Natural, full-duplex conversations	Very low	Bidirectional audio with VAD and interruption

See real-time details in Real-time Voice (WebSocket).

Phone-based voice: these endpoints are for text-channel voice (push-to-talk notes and WebSocket streaming). For enterprise phone-based inbound and outbound calls with emotion detection and EHR integration, see Platform API: Voice Agent.

Voice Mode Comparison

Voice Notes (HTTP)

Treat each /interact call as an asynchronous voice-note exchange, not a full-duplex call.

Request Essentials

1. Encode microphone audio as MP3, or capture raw PCM (mono, 16-bit or 32-bit samples, 4000-44100 Hz frame rate). WAV containers and FLAC are not accepted and return 415 Unsupported Media Type.
2. POST the clip as recorded_message with request_format=voice, and set the request_audio_config query parameter to match the encoding: {"type":"mp3"} for MP3, or {"type":"pcm","frame_rate":16000,"sample_width":2,"n_channels":1} for raw PCM.
3. Set response_format=voice.
4. Read the NDJSON stream. new-message events contain base64 PCM16 audio chunks. There is no response codec selection; the Accept header does not change the audio format.

TypeScript SDK note: voice over HTTP is not yet supported in the TS SDK. Use direct API calls.

Sequence Diagram: Voice Note Exchange

API Reference

Interact with a conversation

post

/v1/{organization}/conversation/{conversation_id}/interact

Send a new user message to the conversation. The endpoint will perform analysis and generate an agent message in response.

A UserMessageAvailableEvent will be the first event in the response, which includes the user message if it's sent as text, or the transcribed message if it's sent as voice. A series of CurrentAgentActionEvents will follow, which indicates steps in the agent's thinking process. Then the agent message is generated sequentially in pieces, with each piece being sent as a NewMessageEvent in the response. After all the pieces are sent, an InteractionCompleteEvent is sent. Depending on the conversation_completed property in this event, the conversation will be awaiting a new message from the user, or it might automatically end (for instance, because the user message indicates the user wants to end the session), while the conversation is marked as finished and the post-conversation analysis asynchronously initiated. The connection will then terminate.

Any further action on the conversation is only allowed after the connection is terminated.

A 200 status code doesn't indicate the successful completion of this endpoint, because the status code is transmitted before the stream starts. At any point during the stream, an ErrorEvent might be sent, which indicates that an error has occurred. The connection will be immediately closed after.

This endpoint can only be called on a conversation that has started but not finished.

Permissions

This endpoint requires the following permissions:

• User:UpdateUserInfo on the user who started the conversation.
• Conversation:InteractWithConversation on the conversation.

This endpoint may be impacted by the following permissions:

• CurrentAgentActionEvents are only emitted if the authenticated user has the Conversation:GetInteractionInsights permission.

Authorizations

AuthorizationstringRequired

The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.

AuthorizationstringRequired

Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.

X-ORG-IDstringRequired

An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.

Path parameters

conversation_idstringRequired

The identifier of the conversation to send a message to.

Pattern: ^[a-f0-9]{24}$

organizationstringRequired

Query parameters

request_formatstring · enumRequired

The format in which the user message is delivered to the server.

Possible values:

response_formatstring · enumRequired

The format of the response that will be sent to the user.

Possible values:

current_agent_action_typestringOptional

A regex for filtering the type of the current agent action to return. By default, all are returned. If you don't want to receive any events, set this to a regex that matches nothing, for instance ^$.

Default: ^.*$

request_audio_configone of · nullableOptional

Configuration for the user message audio. This is only required if request_format is set to voice.

or

Header parameters

content-typestringRequired

The content type of the request body, which must be multipart/form-data followed by a boundary.

Pattern: ^multipart\/form-data; boundary=.+$

x-mongo-cluster-namestring · nullableOptional

The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.

Sec-WebSocket-Protocolstring[]OptionalDefault: []

Body

or

or

Responses

200

Succeeded. The response will be a stream of events in JSON format separated by newlines. The server will transmit an event as soon as one is available, so the client should respond to the events as soon as one arrives, and keep listening until the server closes the connection.

application/x-ndjson

or

or

or

or

400

This may occur for the following reasons:

• The user message is empty.
• The preferred language does not support voice transcription or response.
• The response_audio_format field is not set when voice output is requested.
• The timestamps for external event messages are not in the past.
• The timestamps for external event messages are inconsistent with the conversation.
• The agent does not have voice config specified.

401

Invalid authorization credentials.

403

Missing required permissions.

404

Specified organization or conversation is not found.

408

The request body stream timed out.

409

The specified conversation is already finished, or a related operation is in process.

415

The format of the supplied audio file is not supported.

422

Invalid request path parameter or request body failed validation.

429

The user has exceeded the rate limit of 15 requests per minute for this endpoint.

503

The service is going through temporary maintenance.

post/v1/{organization}/conversation/{conversation_id}/interact

POST /v1/{organization}/conversation/{conversation_id}/interact?request_format=text&response_format=text HTTP/1.1
Host: api.amigo.ai
Authorization: Bearer YOUR_SECRET_TOKEN
X-ORG-ID: YOUR_API_KEY
Content-Type: multipart/form-data
Accept: */*
Content-Length: 163

{
  "initial_message_type": "text",
  "recorded_message": "text",
  "external_event_message_content": [
    "text"
  ],
  "external_event_message_timestamp": [
    "2026-01-01T00:00:00.000Z"
  ]
}

{
  "type": "interaction-complete",
  "message_id": "text",
  "interaction_id": "text",
  "full_message": "text",
  "conversation_completed": true
}

Minimal Client Handling (browser-friendly)

if (evt.type === "new-message" && typeof evt.message === "string" && evt.message) {
  const bytes = Uint8Array.from(atob(evt.message), (c) => c.charCodeAt(0));
  playAudio(bytes.buffer); // your audio player implementation
}

Tips

• Keep uploads short (a few seconds) for responsive turn-taking.
• Accumulate audio chunks from new-message into a single buffer for smooth playback.
• Use interaction-complete as the boundary between turns.

Managing Perceived Latency

During voice interactions, the agent plays pre-generated audio fillers (for example, "Let me look that up...") when an operation exceeds its configured threshold, so users hear natural feedback instead of silence. Filler phrases and thresholds are configured per Context Graph state, and the threshold should be kept as close to zero as the schema allows. See the Events page for the full event structure and configuration reference:

Events

On the client, audio fillers arrive as current-agent-action events with type action-too-long:

if (evt.type === "current-agent-action" && evt.action.type === "action-too-long") {
  // Audio filler contains base64 PCM audio (or text fallback)
  const audioFiller = evt.action.filler;
  playAudio(base64ToBytes(audioFiller));
}

Real-time Voice (WebSocket)

For low-latency, natural conversation with VAD and barge-in, use Real-time Voice (WebSocket). It supports continuous upstream audio, interruption handling, and streaming TTS.

Real-time Voice (WebSocket)