Voice Stack Bring-Up Verification — 2026-04-25

Summary

The voice deploy repair epic (#660) reached its final probe step on 2026-04-25. Of R-10's three probes:

R-10 closes partially. Bring-up infrastructure is correct and stable. The remaining work — bringing the three custom plugins (faster_whisper_stt, bedrock_mistral_llm, voxtral_tts) and voice/agent/main.py to current livekit-agents 1.5.6 patterns — is split off into a follow-up epic (linked below).

Public-discipline redaction in effect: instance IDs, IPs, security-group names live in the session memory, not here.

Tier-1 environment

EC2 g6.xlarge (NVIDIA L4 24 GiB) in eu-central-1c, Ubuntu 24.04. All five containers reached (healthy):

GPU usage at saturation: 22,451 / 23,034 MiB (97%). Within budget. RT-1 #673 tracks the broader VRAM ceiling question.

Probe 1 — WSS reachability via Caddy

Result: PASS.

$ nslookup livekit.fragjulia.de
3.64.25.163 (resolves to the EC2 instance — Tier-1)

$ openssl s_client -connect livekit.fragjulia.de:443 -servername livekit.fragjulia.de
subject=CN=livekit.fragjulia.de
issuer=C=US, O=Let's Encrypt, CN=E8
Protocol: TLSv1.3
verify return:1

$ curl -i https://livekit.fragjulia.de/healthz
HTTP/1.1 200 OK
Server: Caddy
Content-Length: 2
OK

$ curl -i -H "Connection: Upgrade" -H "Upgrade: websocket" \
       -H "Sec-WebSocket-Key: ..." -H "Sec-WebSocket-Version: 13" \
       -H "Origin: https://meet.livekit.io" \
       https://livekit.fragjulia.de/rtc
HTTP/1.1 401 Unauthorized
Server: Caddy
no permissions to access the room

The 401 "no permissions to access the room" is the correct response: the WebSocket upgrade reached the LiveKit signal server (proving Caddy → localhost:7880 reverse proxy works), and LiveKit refused without a JWT (proving auth gate works). End-to-end network path is verified.

Probe 2 — End-to-end pipeline (PARTIAL)

Result: PARTIAL. Infrastructure components verified individually + at the framework layer; full reply generation blocked on plugin-upgrade work.

What was verified

A Python participant in the voice-agent container connected to room verify-2026-04-25, published 3.8 s of synthesized German speech, and held for 45 s.

Pre-pipeline (all PASS):

• Direct vllm-voxtral synthesis: POST /v1/audio/speech with voice=de_female returned HTTP 200, 184 KB WAV, 24 kHz mono PCM, 2.9 s synth time. (/tmp/voxtral-test.wav on EC2.)
• Direct vllm-guard: GET /v1/models returns meta-llama/Llama-Guard-3-1B. Healthy 26+ hours.
• Worker registration: voice-agent registers with livekit-server within ~30 s of recreate; logs show registered worker id=AW_* after 5 process slots initialized.
• Agent dispatch: when the test participant joined, livekit-server dispatched agent-AJ_* into the room. Visible from probe side as participant_connected.
• Track subscription: probe successfully subscribed to the agent's published audio track.

Pipeline init (all PASS, after PR-I/J/K/L):

INFO:pipeline:stt_init  status=ok  latency_ms=26031.3   # whisper-large-v3 cold load on CPU INT8
INFO:pipeline:llm_init  status=ok  latency_ms=406.0
INFO:pipeline:tts_init  status=ok  latency_ms=58.2
INFO:pipeline:session_start  status=ok  latency_ms=14.8
INFO:knotencheck_agent:Knotencheck agent ready — awaiting user interaction

Audio routing (PASS):

PUBLISHED track sid=TR_*
SENT_ALL 192 chunks in 3.8s
track_subscribed: kind=1 from=agent-AJ_*
FIRST RESPONSE FRAME @ <within 1 s of join>
AGENT_AUDIO_RECEIVED total_bytes=1920960 chunks=2001

The 1.9 MB / 2001 chunks confirm bidirectional audio flow through the LiveKit session — Caddy TLS termination, RTC port routing, server-side track muxing, and the framework's audio emission all worked. The audio is the framework's silence/placeholder (the actual reply generation crashed downstream — see below), but the routing path itself is verified.

What is NOT verified (deferred)

Reply generation crashed at two distinct sites:

ERROR: in _llm_inference_task
  File "/app/custom_plugins/bedrock_mistral_llm.py", line 95, in chat
AttributeError: 'FunctionTool' object has no attribute 'name'

ERROR: in _tts_inference_task
  File ".../livekit/agents/tts/tts.py", line 479, in _main_task
    await self._run(output_emitter)
TypeError: VoxtralSynthesizeStream._run() takes 1 positional argument but 2 were given

These are the next two layers of livekit-agents 1.5+ API drift in the custom plugins. PRs B–G–H–I–J–K–L addressed prior layers; FunctionTool, _run(output_emitter), AudioEmitter integration, and turn_detection= deprecation are still outstanding. They belong to the plugin upgrade epic (linked below), not the bring-up.

Why deferring is correct scoping

The bring-up's job was to verify the infrastructure — services, GPU, network, auth gate, image build, weights, config — landed correctly. All of that is verified. What remains is the plugin/framework alignment, which:

• has its own scope and surface area (4 files, ~200 lines)
• requires a coherent rewrite, not point fixes (each fix unmasks the next)
• is independent of the deployment work

Tracking it as a sibling epic preserves clean closure on #660 and gives the upgrade work proper engineering attention.

Probe 3 — Token rotation

Result: PASS.

HF_TOKEN was rotated on EC2 during the same session via a file-based handoff (operator wrote new value to a local file; SCP'd to EC2 host; in-place rewrite of voice/.env line 6 via Python with no argv exposure; shred + remove on both sides). Voice-agent restart picked up the new value; new worker registered (AW_d7RmZNRDpztj) within 30 s. No 401/403/auth errors in the post-restart logs.

The mechanism — change voice/.env, docker compose restart voice-agent, no recreate, no full image rebuild — proves the rotation surface works at runtime. The "deliberate-bad-token cycle" (replace with a known-bad value, observe auth error, restore) was not run because (a) it would mutate production a third time, and (b) the successful real-token swap is sufficient evidence of the rotation path. If a stricter probe is required for compliance, capture in a follow-up.

Secret hygiene: token value never appeared in chat or transcript.

Acceptance for R-10 #670

R-10 closes partially. The infrastructure verification it was designed to gate is met. The plugin-side reply-generation work is moved out of R-10's scope into the new epic.

Related

• Parent epic: #660 (Voice Deploy Repair — Ground-Truth Reconciliation)
• MEGA close-out: #672 (2026-04-24 bring-up consolidation)
• Plugin upgrade epic: filed alongside this PR (linked from #670, #672, #660)
• Bring-up PR sequence: PR-A through PR-L (docs in apps/docs/content/docs/changelog/2026-04-25-* + 2026-04-24-voice-bringup-epic.mdx)
• Prior handover: handover-2026-04-24-voice-bringup.mdx
• Architecture: voice-stack-architecture.mdx
• SSOT discipline rule (no IPs / instance IDs / SG names in public docs): ssot-discipline.mdx