Everything you need to drive the demo and understand what's underneath it.
The live app lives at vaanimesh.ajaykongala.online/app (it forwards to the current demo session). Access is gated by a PIN:
2468 — demo PIN for this beta. It gates access; it is not a security boundary. Sessions run on demo credit.
Speak a full sentence and pause. Your utterance appears in a pending tray with a small player — listen if you like, then press Send ➤ (or Discard). A few seconds later every other participant hears your sentence in their own language and sees both texts.
The Automatic send toggle in ⚙ settings stays locked until the relay declares the chain healthy — five clean deliveries with latency in budget. Once unlocked, utterances send themselves when you pause. You can always drop back to manual.
| Setting | Effect |
|---|---|
| My language | Switch your language mid-session — the very next utterance follows the new speak-your-language rule, everyone's roster updates, and the confirmation offers a one-tap ↺ Retranslate recent messages that re-renders the last few messages from others in your new language (text only) |
| Voice preference | The synthesized voice others hear for your translations |
| VAD sensitivity | How eagerly the voice detector treats a pause as end-of-sentence |
| Show mesh headers | Expands each received message's relay receipt (latencies, sequence, mode) |
Open /dashboard on the app origin (same PIN). It is read-only:
speaker cards with mic status, live loudness, audio format and round-trip
time; the pipeline's rolling latencies; and a live log of every relayed
message with its full header.
Meera (Hindi) runs procurement; Ravi (Telugu) supplies produce. Neither is comfortable in the other's language.
Three wired nodes. Browser speaker consoles do capture (voice-activity detection in an audio worklet, 16 kHz WAV encoding), playback, settings, and telemetry. The relay — an independent FastAPI server — runs the entire AI chain as a Pipecat frame pipeline: speech recognition → language identification → per-listener translation → speech synthesis → header stamping → routing. The dashboard is a read-only observer fed by the same server.
| Stage | Engine (current) | Notes |
|---|---|---|
| Recognition | gpt-4o-transcribe via OpenRouter | Chosen over Whisper after a live bake-off: Whisper translated Indic speech to English instead of transcribing it |
| Language ID | Local script analysis | The five languages span four scripts; Devanagari (Hindi/Marathi) is split by function-word markers. Zero-cost, deterministic |
| Translation | Grok 4.3 via OpenRouter | Interpreter-prompted: meaning over phonetics, colloquial register; reasoning disabled for a ~5× latency win |
| Voice | Gemini Flash TTS via OpenRouter | PCM output wrapped to WAV server-side |
Every stage sits behind a provider interface; Sarvam, Bhashini, and Deepgram adapters exist, and a benchmark harness scores any configured provider on licensed reference recordings (word-error rate, translation overlap, synthesis round-trip intelligibility).
One device captures the whole room. Because each participant owns a distinct language, identifying an utterance's language identifies its speaker; a voice-fingerprint (speaker-embedding) tie-break covers low-confidence cases, seeded by a one-sentence enrollment per participant. Verified with two real recorded speakers through a single microphone.
WebSocket JSON, one connection per console:
wss://…/ws/speaker. The dashboard feed is
wss://…/ws/dashboard?pin=…. Health: GET /healthz.
{"type":"join","name":"Meera","language":"hi-IN","pin":"2468","settings":{…}}
{"type":"utterance","utterance_id":"…","audio_b64":"<16kHz PCM16 WAV>",
"sample_rate":16000,"duration_ms":3200,"mode":"manual"}
{"type":"settings","tts_voice":"Puck","vad_sensitivity":"normal"}
{"type":"telemetry","rms":0.04,"mic_on":true,"rtt_ms":23}
{"type":"mode_request","mode":"auto"} // granted only when eligible
{"type":"ping","t":123456.7}{"type":"joined","speaker_id":"…","mode":"manual","auto_allowed":false,"roster":[…]}
{"type":"own_transcript","utterance_id":"…","text":"…","detected_language":"hi-IN"}
{"type":"translated",
"mesh":{"msg_id":"…","seq":7,"hop":"relay@:8030","mode":"manual",
"src":{"speaker":"Ravi","lang":"te-IN"},"tgt_lang":"hi-IN",
"received_at":"…","forwarded_at":"…",
"latency_ms":{"asr":800,"mt":950,"tts":3100,"relay_total":4900},
"audio":{"sample_rate":16000,"bits":16,"kbps":256.0}},
"original_text":"…","translated_text":"…","audio_b64":"…",
"tts_failed":false,"mt_failed":false}
{"type":"mode_state","mode":"manual","auto_allowed":true,"reason":"chain healthy …"}
{"type":"error","stage":"join","message":"wrong access PIN"}The mesh block is the relay's stamp — attach-only, present on
every forwarded message, rendered in the consoles and the dashboard log.