AI, ML & Computer Vision Meetup · October 15, 2026
The two-loop architecture
for voice AI
Foreground & background interactions in spoken dialog systems
Abhinav Tushar · Co-founder, Panini
About
Abhinav Tushar
Co-founder, Panini
An AI product technology lab from Bengaluru
Eight years putting voice AI in front of real callers.
- Production spoken dialog systems since 2018, from finite state machines to LLMs
- Voicebots at enterprise scale, in 10+ Indian languages
- Speech products for noisy, low-resource, error-prone settings
Skit.ai · 2018–2024
Head of AI, founding ML engineer
lepisma.xyz/wiki/about · paninihq.com · hello@paninihq.com
The problem
Latency vs intelligence
600–1500 ms
of dead time before a reply starts to feel slow
Panini · The two-loop architecture for voice AI
The problem
Which model faces the user?
Small model
Fast enough to be the frontend
…but not always correct
Large model
Smart enough to get it right
…but too slow to be the frontend
We need the speed of one and the brains of the other.
Panini · The two-loop architecture for voice AI
01 · Today
A contemporary spoken dialog system
Cascade
speech in
→
ASR
→
LLM
→
TTS
→
speech out
End-to-end
speech in
→
Speech-to-speech model
→
speech out
Either way, one loop listens, thinks and talks.
Panini · The two-loop architecture for voice AI
02 · Patterns
Background ⇄ foreground
02 · Patterns
Tools
The most common paradigm today
Foreground
calls →
Background tool
→ result back
The foreground calls the background and uses what comes back.
Simple to integrate. The foreground stays in charge.
Panini · The two-loop architecture for voice AI
02 · Patterns
The catch: the foreground orchestrates
When to call
Missed or needless tool calls
What to ask
Underspecified queries
How to use it
Results misread or dropped
The smallest model in the system is making the biggest decisions.
Panini · The two-loop architecture for voice AI
03 · The idea
Two loops
Foreground · fast
Talks in real time, never blocks
Background · slow
Reasons asynchronously, often a Larger LLM
The hard part is how they talk to each other.
Panini · The two-loop architecture for voice AI
03 · The idea
Timing is the design space
Panini · The two-loop architecture for voice AI
04 · The flip
Let the background orchestrate, asynchronously.
More powerful patterns
4.1 Enrichment & correction
4.2 Utterance hot swapping
4.3 Re-engagement, speculation & prefetch
4.1 · Patterns
Enrichment & correction
The background steers by editing the foreground's context.
- Notes via an enricher role
- Corrections
- Memory
Panini · The two-loop architecture for voice AI
4.1 · Demo
Enrichment & correction
4.2 · Patterns
Utterance hot swapping
Swap the words going to TTS, mid-speech.
“…actually, …”
KAME-like behaviour, no new model to train.
KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI · Sakana AI, ICASSP 2026
Panini · The two-loop architecture for voice AI
4.2 · Demo
Utterance hot swapping
4.3 · Patterns
Re-engagement, speculation & prefetch
When to surface a result
Now, if urgent
Next turn
When asked
Prefetch what's likely next: scores, weather.
Panini · The two-loop architecture for voice AI
Takeaway
Much of what end-to-end models promise can be engineered and put to work before ML maturity
Engineered first, end-to-end later
Speech translation
ASR → MT → TTS cascades, years before direct translation
Barge-in
VAD and echo cancellation, before full-duplex models
Turn-taking
Endpointing rules and classifiers, before native turn-taking
Expressive speech
SSML prosody tags and fillers, before expressive generation
Thanks
Questions?
Post
lepisma.xyz · Foreground & background interactions
Code
Newsletter
Subscribe to Computational Conversations
Panini
paninihq.com · hello@paninihq.com
These slides were generated with an AI tool. Core content is from my original post.