Live Transcription WebSocket Integration

Feature Description

The Live Transcription feature is a configuration that can be enabled on a channel like extension, queue or ERG. When a call is made that has the configuration set up, the recording is streamed to the configured AI service and the result, which is the transcribed data, is streamed to the WebSocket publisher alongside the metadata configured with the Live Transcription. This is done in real time.

Websocket server setup

The Live Transcription requires the address of a websocket server to which the service will connect when the call with the Live Transcription happens. These connections will persist until the call ends, which signals the end of the transcription for that call.

A minimal WebSocket server that prints any message it receives.

Node.js
Python
Go
Rust

Install the ws package: npm install ws

  import { WebSocketServer } from 'ws';

  const wss = new WebSocketServer({ port: 8080 });

  wss.on('connection', (ws) => {
    console.log('Client connected');

    ws.on('message', (data) => {
      console.log('Received:', data.toString());
    });

    ws.on('close', () => console.log('Client disconnected'));
  });

  console.log('WebSocket server running on ws://localhost:8080');

Install: pip install websockets

  import asyncio
  from websockets.asyncio.server import serve

  async def handler(websocket):
      print("Client connected")
      try:
          async for message in websocket:
              print(f"Received: {message}")
      finally:
          print("Client disconnected")

  async def main():
      async with serve(handler, "localhost", 8080):
          print("WebSocket server running on ws://localhost:8080")
          await asyncio.Future()  # run forever

  asyncio.run(main())

Install: go get github.com/coder/websocket

  package main

  import (
      "log"
      "net/http"

      "github.com/coder/websocket"
  )

  func handler(w http.ResponseWriter, r *http.Request) {
      c, err := websocket.Accept(w, r, nil)
      if err != nil {
          log.Println("accept error:", err)
          return
      }
      defer c.CloseNow()

      log.Println("Client connected")
      ctx := r.Context()

      for {
          _, msg, err := c.Read(ctx)
          if err != nil {
              log.Println("Client disconnected:", err)
              return
          }
          log.Printf("Received: %s", msg)
      }
  }

  func main() {
      http.HandleFunc("/", handler)
      log.Println("WebSocket server running on ws://localhost:8080")
      log.Fatal(http.ListenAndServe(":8080", nil))
  }

Add to Cargo.toml:

  [dependencies]
  tokio = { version = "1", features = ["full"] }
  tokio-tungstenite = "0.24"
  futures-util = "0.3"

  use futures_util::StreamExt;
  use tokio::net::{TcpListener, TcpStream};
  use tokio_tungstenite::tungstenite::Message;

  async fn handle_connection(stream: TcpStream) {
      let ws_stream = tokio_tungstenite::accept_async(stream)
          .await
          .expect("WebSocket handshake failed");
      println!("Client connected");

      let (_write, mut read) = ws_stream.split();

      while let Some(Ok(msg)) = read.next().await {
          if let Message::Text(text) = msg {
              println!("Received: {}", text);
          }
      }
      println!("Client disconnected");
  }

  #[tokio::main]
  async fn main() {
      let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap();
      println!("WebSocket server running on ws://localhost:8080");

      while let Ok((stream, _)) = listener.accept().await {
          tokio::spawn(handle_connection(stream));
      }
  }

Running the Server

Node.js
Python
Go
Rust

Save as server.mjs (or use .js with "type": "module" in your package.json), then run:

  node server.mjs

Save as server.py, then run:

  python server.py

Save as main.go in a Go module (run go mod init example.com/wsserver if you haven't yet), then:

  go run main.go

Inside your Cargo project, run:

  cargo run

For an optimized build use cargo run --release.

Live Transcription Data Output

Data is delivered in a JSON protocol with the following structure:

{
  "transcribed_data": {...},
  "metadata" : {...}
}

transcribed_data - The data the AI service returns
metadata - The configured metadata with replaced PLACEHOLDERS

metadata is always the same with every transcription update. And the structure is equal to the configured metadata in Live Transcription Configuration

transcribed_data depends on the AI service used in the transcription.

OpenAI Realtime Transcription

For OpenAI, you can check the current data structure from their official documentation. One addition is made to the OpenAI transcribed_data, and that is the speaker key, which can hold one of two values (caller, callee). This differentiates the transcription without the need for diarization.

This is the delta event. It pushes data quicker and word for word.

{
  "type": "conversation.item.input_audio_transcription.delta",
  "item_id": "item_003",
  "content_index": 0,
  "delta": "Hello,",
  "speaker": "callee"
}

This is the completed event. This event fires after some time has passed and gives a complete sentence or part of a sentence.

{
  "type": "conversation.item.input_audio_transcription.completed",
  "item_id": "item_003",
  "content_index": 0,
  "transcript": "Hello, how are you?",
  "speaker": "caller"
}

Both of these showcase the added key speaker. This is NOT standard OpenAI JSON protocol.

Deepgram

For Deepgram, transcribed_data is more detailed, and instead of adding a speaker key like in OpenAI, Deepgram differentiates caller from callee using channel_index.

{
  ...
  "channel_index" : [0, 2]
  ...
  // OR 
  ...
  "channel_index" : [1, 2]
  ...

}

channel_index - The channel number 0 or 1 differentiates caller from callee respectively.

0 - caller
1 - callee

Example of Deepgram transcribed_data:

{
  "channel": {
    "alternatives": [
      {
        "confidence": 0.9970703,
        "transcript": "this is the caller",
        "words": [
          {
            "confidence": 0.92333984,
            "end": 5.19,
            "start": 4.87,
            "word": "this"
          },
          {
            "confidence": 0.99902344,
            "end": 5.27,
            "start": 5.19,
            "word": "is"
          },
          {
            "confidence": 0.9970703,
            "end": 5.43,
            "start": 5.27,
            "word": "the"
          },
          {
            "confidence": 0.8366699,
            "end": 5.83,
            "start": 5.43,
            "word": "caller"
          }
        ]
      }
    ]
  },
  "channel_index": [
    0,
    2
  ],
  "duration": 1.19,
  "is_final": true,
  "metadata": {
    "model_info": {
      "arch": "nova-3",
      "name": "general-nova-3",
      "version": "2025-04-17.21547"
    },
    "model_uuid": "40bd3654-e622-47c4-a111-63a61b23bfe8",
    "request_id": "019e26a1-f1db-7062-9346-9ebc487e1ecf"
  },
  "speech_final": true,
  "start": 4.87,
  "type": "Results"
}

AWS Transcribe

Example of AWS Transcribe response:

{
  "Alternatives": [
    {
      "Entities": null,
      "Items": [
        {
          "Confidence": 0.9428,
          "Content": "I'm",
          "EndTime": 18.017,
          "Speaker": null,
          "Stable": null,
          "StartTime": 17.897,
          "Type": "pronunciation",
          "VocabularyFilterMatch": false
        },
        {
          "Confidence": 0.9962,
          "Content": "gonna",
          "EndTime": 18.347,
          "Speaker": null,
          "Stable": null,
          "StartTime": 18.017,
          "Type": "pronunciation",
          "VocabularyFilterMatch": false
        },
        {
          "Confidence": 0.9421,
          "Content": "set",
          "EndTime": 18.537,
          "Speaker": null,
          "Stable": null,
          "StartTime": 18.527,
          "Type": "pronunciation",
          "VocabularyFilterMatch": false
        },
        {
          "Confidence": 0.9967,
          "Content": "up",
          "EndTime": 18.957,
          "Speaker": null,
          "Stable": null,
          "StartTime": 18.537,
          "Type": "pronunciation",
          "VocabularyFilterMatch": false
        },
        {
          "Confidence": 0.9982,
          "Content": "now",
          "EndTime": 19.637,
          "Speaker": null,
          "Stable": null,
          "StartTime": 18.957,
          "Type": "pronunciation",
          "VocabularyFilterMatch": false
        }
      ],
      "Transcript": "I'm gonna set up now"
    }
  ],
  "ChannelId": "ch_1",
  "EndTime": 19.707,
  "IsPartial": false,
  "LanguageCode": "",
  "LanguageIdentification": null,
  "ResultId": "e4ffde22-092c-4a60-8b71-a7b7c20e77ba",
  "StartTime": 17.887
}

AWS Transcribe also separates speakers by channel, so ChannelId can have values ch_0 and ch_1. Same as with the rest:

ch_0 - caller
ch_1 - callee

If you enable Sentiment Analysis on AWS Transcribe, then the responses have a different protocol:

{
  "BeginOffsetMillis": 687,
  "ChannelId": "ch_1",
  "EndOffsetMillis": 8107,
  "Entities": null,
  "IsPartial": false,
  "IssuesDetected": [],
  "Items": [
    {
      "BeginOffsetMillis": 767,
      "Confidence": 0.9998,
      "Content": "Hello",
      "EndOffsetMillis": 1047,
      "Stable": null,
      "Type": "pronunciation",
      "VocabularyFilterMatch": false
    },
    {
      "BeginOffsetMillis": 1047,
      "Confidence": null,
      "Content": ",",
      "EndOffsetMillis": 1047,
      "Stable": null,
      "Type": "punctuation",
      "VocabularyFilterMatch": false
    },
    {
      "BeginOffsetMillis": 1167,
      "Confidence": 0.964,
      "Content": "how",
      "EndOffsetMillis": 1527,
      "Stable": null,
      "Type": "pronunciation",
      "VocabularyFilterMatch": false
    },
    {
      "BeginOffsetMillis": 1527,
      "Confidence": 0.9998,
      "Content": "are",
      "EndOffsetMillis": 2117,
      "Stable": null,
      "Type": "pronunciation",
      "VocabularyFilterMatch": false
    },
    {
      "BeginOffsetMillis": 2367,
      "Confidence": 0.9965,
      "Content": "you",
      "EndOffsetMillis": 3037,
      "Stable": null,
      "Type": "pronunciation",
      "VocabularyFilterMatch": false
    },
    {
      "BeginOffsetMillis": 3527,
      "Confidence": 0.8285,
      "Content": ".",
      "EndOffsetMillis": 3937,
      "Stable": null,
      "Type": "punctuation",
      "VocabularyFilterMatch": false
    }
  ],
  "LanguageCode": "",
  "LanguageIdentification": null,
  "Sentiment": "NEUTRAL",
  "Transcript": "Hello, how are you.",
  "UtteranceId": "c604162f-680f-41ad-922e-70babe4034a2"
}

Sentiment can have one of these values:

NEGATIVE
NEUTRAL
POSITIVE

Google Speech-to-Text

Data received when using Google Speech-to-Text:

{
  "alternatives": [
    {
      "transcript": "Hello. How are you.",
      "confidence": 0.90245813
    }
  ],
  "is_final": true,
  "result_end_time": {
    "seconds": 15,
    "nanos": 860000000
  },
  "channel_tag": 2,
  "language_code": "en-us"
}