Feedback Actions
Update From Voice Feedback Action
Explains the Voice feedback action that runs during and after a live AI voice call. Useful for handling requests from the voice assistant, keeping the conversation history, wrapping up when the call ends, and getting the call recording.
Overview
A live voice call is started with the Start Voice Chat task. While the call runs, and once when it ends, the _update_from_voice feedback action is triggered on the stub.
stubpost.data._incoming_voice_data.event tells you why the action ran:
Each update only carries what was said since the previous update. A sentence that was still being said when an update was sent continues in the next one.
The transcript since the previous update is set to the message of the stubpost, i.e. stubpost.message, as lines like:
For session_ended, the message starts with a Voice session ended: <reason> line, and for the transfer events with a Voice session transferred to <stubref> or from <stubref> line. For recording_ready, the message is Voice call recording ready.
All the details are available in the stubpost.data._incoming_voice_data data path.
A delegation doesn't say what the assistant needs. Work it out from the messages, eg by running a GPT Chat Task with them.
Handling the events
delegation
- Run a GPT Chat Task with the
messages. - When its answer arrives, send it to the call with a
commentaryvoice notification and thedelegation_id:
While it waits, the assistant tells the caller it is checking.
interval
Store the messages in the chat history, so the agent has the whole conversation the next time it runs.
session_ended
Wrap up, eg save a summary of the call or send a follow-up. session_ended.reason tells you how the call ended.
The charge for the length of the call comes with this update, and is added to the stubpost's transaction history like the charges of tasks.
After a transfer, the stub that started the call always gets session_ended, with the charges for the whole call. The stub that had the call when it ended gets session_ended too, first, with the last of the transcript and without the charges. session_ended.ended_on_stubref says which stub had the call.
session_transferred_out and session_transferred_in
Another stub took the call over with Voice Session Transfer. The caller stays on the line.
The stub that had the call gets session_transferred_out, with the last of what was said while it had the call. Wrap up its part, eg note the handover. From now on its voice notifications and Voice Commands fail with voice_session_transferred.
The stub that took the call over gets session_transferred_in. messages is empty, the whole conversation so far is in conversation, ready to keep as the chat history. delegation_id is the latest delegation, usually the request that caused the transfer, so answer it with a commentary voice notification.
recording_ready
Every call is recorded. Shortly after the call ends, the recording is uploaded and arrives with one more _update_from_voice, attached to the stubpost: it's in stubpost.attachments, like the attachments of an email or a WhatsApp message.
The recording is a stereo WAV file: the caller on the left channel, the assistant on the right. Phone calls are recorded at 8 kHz telephone quality, webchat and WhatsApp calls at 24 kHz.
Use it like any other attachment, eg keep it with the stub, or pass recording.fileuuid to a task that transcribes or analyses audio:
recording.truncated is true when the recording stopped early, eg the voice service ran low on disk space or restarted during the call. The recording then has the call up to that point.
When a call couldn't be recorded at all, no recording_ready follows. Instead, a Voice call recording failed event is added to the stub, under the stubpost that started the call.
Data Structure
Feedback Action Data
Example data that is passed to the _update_from_voice feedback action for a delegation:
When the call has ended:
When the recording is ready:
The same file is in stubpost.attachments:
When another stub took the call over, on that new stub:
Properties
event string
Why the action ran: delegation, interval, session_ended, recording_ready, session_transferred_out or session_transferred_in.
delegated boolean
true when the assistant needs the stub's help, i.e. when event is delegation.
voice_session_uuid string
The id of the voice call, from Start Voice Chat.
chat_name string
The chat_name the call was started with.
channel string
The channel of the call: webchat, whatsapp or dial (phone calls).
messages array
What was said since the previous update, as chat messages. The caller is user and the assistant is assistant. This is the same format as the GPT Chat Task messages. Not set for recording_ready.
Empty when nothing new was said, eg for a session_ended straight after a delegation.
transcript array
The timed fragments behind the messages. Each fragment has a speaker (caller or assistant), the delta text, and start_ms and end_ms, the time in milliseconds from the start of the call. Not set for recording_ready.
delegation_id string
delegation and session_transferred_in. Pass it back with the answer, so the assistant knows which request it answers. On session_transferred_in it's the latest delegation on the call, usually the one that asked for the transfer.
offset_ms number
delegation only. When the assistant delegated, in milliseconds from the start of the call.
session_ended.status string
session_ended only. closed, or abandoned when the caller's channel never joined the call.
session_ended.reason string
session_ended only. How the call ended:
session_ended.usage_seconds number
session_ended only. The billed length of the call in seconds. Time before the caller joined isn't billed.
session_ended.usage_confirmed boolean
session_ended only. true when the AI provider confirmed the final usage_seconds. When false, usage_seconds is the last known value.
session_ended.ended_on_stubref string
session_ended only. The stub that had the call when it ended. Differs from the stub getting the update when the call was transferred away from it.
conversation array
session_transferred_in only. The whole conversation so far, as chat messages, the same format as messages.
session_transferred.to_stubref string
session_transferred_out only. The stub that took the call over.
session_transferred.from_stubref string
session_transferred_in only. The stub that had the call.
session_transferred.initiating_stubref string
session_transferred_in only. The stub that started the call. It's charged for the call and gets session_ended with the charges.
session_transferred.transferred_at string
The transfer events. When the call was transferred, as an ISO 8601 date-time.
recording.fileuuid string
recording_ready only. The recording's file, the same as in stubpost.attachments.
recording.duration_seconds number
recording_ready only. The length of the recording in seconds.
recording.format object
recording_ready only. The audio format of the recording, the same as the call's: audio/pcm at 24000 for webchat and WhatsApp calls, audio/pcmu or audio/pcma at 8000 for phone calls.
recording.channels object
recording_ready only. Which side of the call is on which stereo channel: the caller on the left, the assistant on the right.
recording.truncated boolean
recording_ready only. true when the recording stopped before the call ended.
Call state
_update_from_voice also carries the state of the stub's voice call in stubpost.data._com_platforms.voice. For the first delegation of a call:
A voice call happens inside the stub's conversation on WhatsApp, webchat or a phone call, so _update_from_voice only sets _com_platforms.voice. The last_incoming_platform and last_from that channel set stay as they are, so replies with auto_platform still go to that channel.
Stub data is merged, so each field is only sent when it changes, and stub.data._com_platforms.voice always has the latest of each.
Use it from any action, eg to answer a WhatsApp message in the call while it's running, and as text otherwise:
voice.active boolean
true while the stub has a running call. false once the call was told to hang up, with the hangup voice notification or the Voice Command task, on session_ended, and on session_transferred_out, where the call moved to another stub. On recording_ready, which comes after the call ended, it says whether the stub has started another call since.
voice.delegation_id string
Sent on delegation, and on session_transferred_in with the latest delegation on the call.
voice.voice_session_uuid string
The call's id, sent on the first update a stub gets for a call: when a new call starts, and on session_transferred_in. Use it from any action on the stub, eg to steer the call with Voice Command or hand it over with Voice Session Transfer.
voice.channel string
The call's channel, sent with voice_session_uuid: webchat, whatsapp or dial.
JSONSchema
The JSONSchema of the stubpost.data._incoming_voice_data is: