Building Switchboard: A Local-First AI Calling-Agent Platform

A case study on designing a modular, voice-capable AI agent system

Switchboard is a multi-tenant platform for building AI calling agents, configurable AI personalities that hold real, goal-directed phone-style conversations. A user writes natural-language instructions, picks a conversation type (sales, survey, customer support, appointment booking, lead qualification, order-taking), grants a set of approved tools, and gets an agent that can talk to callers, answer from a private knowledge base, take actions in real systems, and hand off to a human when it hits its limits.

The entire MVP had to run on a laptop via Docker, no paid telephony, no mandatory cloud, while being architected so every external dependency (the LLM, speech engines, vector store, storage, telephony) could be swapped without rewriting application logic.

That single principle shaped almost every decision that follows, and it paid for itself dramatically when we swapped speech vendors mid-project.

This mentions how it was built, what we chose and why, the architecture in detail, and the bugs and trade-offs that don't usually make it into the polished version.

1. Stack Decision

Postgres + pgvector is exactly what Supabase runs, so the schema is portable to hosted or self-hosted Supabase unchanged.

2. System architecture

At the highest level, the browser talks only to the FastAPI backend and to LiveKit. The backend owns all business logic, tenancy, and authorization; every external capability sits behind an interface resolved by a provider registry.

The browser never talks to the database directly, and never sees a vendor. One auditable authorization model lives in FastAPI, which matters for a paying-customer, multi-tenant product. The voice worker is a separate process that treats the backend as its "brain", the same orchestrator powers both text and voice.

3. The abstraction layer: designing for swappability

Every external dependency is defined by an abstract interface, and a single provider registry chooses the concrete implementation from configuration.

4. The orchestrator: separating deterministic workflow from AI reasoning

The orchestrator is the heart of the system. It assembles the system prompt from the agent's configuration, loads conversation history, runs a bounded agentic loop (model -> tool calls -> model), enforces guardrails, detects escalation, persists every message, and tracks token usage for cost accounting.

The crucial design principle here: deterministic code wraps the AI, not the other way around. Claude can request actions, but whether they execute is decided in code. The loop is bounded so an agent can't spin forever.

Guardrails (escalation detection, output bounds, and the hooks for PII redaction and prompt-injection heuristics) are deterministic functions around the model, never instructions the model is merely trusted to follow.

5. The tool framework: the security boundary

If the orchestrator is the heart, the tool framework is the immune system. A calling agent's inputs, the caller's speech and the RAG documents it reads, are both untrusted, which makes prompt injection a real threat, not a hypothetical. The tool layer is the hard, deterministic boundary between AI reasoning and real-world actions.

The model can only invoke tools it was explicitly granted, destructive actions (sending email, placing orders) route through a human approval queue rather than firing automatically; tools execute server-side with per-tenant credentials the model never sees; and every attempt, permitted, rejected, or pending, lands in an audit table.

6. The data model

Multi-tenancy and compliance were baked in from the first migration, not retrofitted. Every business object carries a tenant_id; consent, do-not-call, recording-consent, and opt-out fields live on the contact from day one even though they aren't exercised until real telephony.

7. Retrieval-augmented generation (RAG)

Business knowledge (FAQs, policies, documents) is chunked, embedded, and stored in pgvector so the agent answers from real business facts rather than prompt guesswork. Retrieval is exposed to the model as the safe, read-only knowledge_lookup tool.

A deliberate distinction runs through the knowledge design: unstructured knowledge goes to RAG; exact facts go to structured tables. A hallucinated policy nuance is recoverable; a hallucinated price on a live order is not. So menu prices are a real table the agent queries exactly, and order totals are computed server-side from those rows, the model supplies only item names and quantities and can never invent a number.

8. The voice pipeline

Voice was the hardest part, and the architecture reflects hard-won lessons about latency and turn-taking. LiveKit owns the real-time audio session and turn detection; the voice worker chains VAD -> STT -> the orchestrator -> TTS. Critically, the orchestrator is unchanged, voice is just a different transport into the same brain.

Two design decisions defined the voice layer. First, text simulator before voice: the entire orchestration, RAG, and tool stack was proven in a text "Call Simulator" before any audio existed, de-risking 80% of the system with zero latency and zero speech dependencies. Second, the worker bridges LiveKit to our backend through a custom LLM class, not by overriding a lifecycle hook, a distinction that turned out to matter enormously.

9. Features delivered

• Agent builder - natural-language system prompt, conversation type, personality, structured success criteria, per-agent tool allow-list, model selection.

• Call Simulator - text and hands-free voice, with a live "line strip" (status lamp, timer), per-turn tool badges, token counts, and escalation handling.

• Prompt versioning - every revision saved and revertible.

• Knowledge base / RAG - ingest documents scoped tenant-wide or per-agent; retrieval via a safe tool.

• Secure tool framework - allow-listing, approval gates, full audit trail.

• Human approval queue - destructive actions wait for a human "yes," executed through the same audited path.

• Order-taking - structured menu, server-computed totals, a live orders board with status workflow (new → confirmed -> completed), linked back to the call.

• Structured answer capture - record_answer fills objectives during the call; results are viewable per-call and exportable as a per-agent CSV (one row per call, one column per question).

• Transcripts - full conversation history with a "captured answers" panel.

• Multi-language - Urdu voice end-to-end (Deepgram STT + Claude + Cartesia TTS) via configuration.

• Multi-tenancy, RBAC, audit logs, escalation, and compliance fields - built in from the start.

10. The design system: "Switchboard"

The UI was designed around the product's own world rather than a generic dashboard template. The metaphor is a vintage telephone operator's console.

• Ground: a warm-gray "handset plastic" background.

• Cards: white "ledger" panels with hard 1px ink rules.

• Type: a grotesque for body text, monospace carrying every status, timer, and ID.

• The switchboard lamp: a single status-dot language used everywhere, signal red only for live calls and escalations (it pulses, and respects reduced-motion), green for connected/ready, amber for anything held.

11. The build journey and the bugs that shaped it

Real projects are defined as much by their obstacles as their features. A few worth documenting:

The passlib/bcrypt crash. The first login attempt crashed: a modern bcrypt release was incompatible with the older passlib. The fix was to drop passlib entirely and call bcrypt directly, a cleaner dependency, and a reminder that transitive version drift is a real production risk.

The JSONB codec bug. asyncpg returns JSONB columns as raw strings unless a type codec is registered, which would have silently broken objective tracking on the first agent with success criteria. Caught and fixed by registering a json/jsonb codec at the connection level.

The post-interruption dormancy bug. The most instructive one. The first voice implementation hooked into LiveKit by overriding llm_node. Barge-in worked, the agent stopped when interrupted, but the next utterance never got a reply; the agent went dormant. The root cause was fighting the framework's turn accounting instead of working with it. The fix was to stop overriding the lifecycle hook and implement a proper custom LLM class (BackendLLM), the framework's real extension point, which reliably receives every finalized turn, including the one right after an interruption. Lesson: use the framework's intended seam, not the nearest hook that seems to work.

The TTS vendor odyssey. The voice started robotic (browser speech), so we moved to real neural TTS. Cartesia gave emotional English that Claude couldn't provide. Because everything sat behind the TTSProvider interface, each pivot was a small, contained change, the abstraction layer's thesis, proven under fire.

The ops friction. Not every obstacle was code. Docker Desktop virtualization, WSL2, and flaky router DNS each blocked builds at various points. Worth stating plainly: in a local-first project, the host environment is part of the system, and provisioning it reliably is real engineering work.

12. Deployment and cost

The platform is designed to deploy two ways.

Costs fall into two buckets: fixed infra (~$25–50/month self-hosted on one VPS; ~$100–200 managed) and variable per-call, Claude tokens plus STT and TTS per minute.

A typical order call runs roughly $0.15–0.35 all-in, dominated by the LLM and TTS. The biggest cost levers are: routing conversational turns to a cheaper/faster model (the per-agent model field already supports this), prompt caching to cut the repeated system-prompt cost, context summarization for long calls, and self-hosting the free/open components.

Every vendor in play offers free starting credits, so a pilot can run at near-zero beyond Claude tokens.

13. Limitations

• Latency: turns are ~1.5–3s end-to-end because the orchestrator returns a full completion before speaking; streaming to TTS is the next latency win.

• Turn-based, not full-duplex barge-in-everywhere, yet solid, but not perfectly interruptible mid-thought in every path.

• Capture reliability: structured answer capture depends on the agent calling, a post-call extraction backstop (re-reading the transcript to fill gaps) is the production-grade safety net still on the roadmap.

• Compliance fields exist but aren't enforced until real telephony, DNC scrubbing, recording consent, and AI disclosure must be live before any outbound call.

• Dev credentials (LiveKit dev keys, default secrets) must be replaced before facing the internet.

Naming these plainly is part of the case study: a system you can trust is one whose authors are clear about where its edges are.

15. What's next

• Evaluation harness, a second model plays the customer against agents and scores transcripts against success criteria, wired to prompt versioning for regression testing.

• Streaming responses, stream Claude's output sentence-by-sentence into TTS to cut perceived latency.

• Post-call extraction backstop, guarantee structured capture even when the agent forgets.

• SIP telephony + the compliance layer, real inbound/outbound phone numbers, the true endgame.

• Per-agent language so one deployment can mix English and Urdu agents.

15. Lessons learned

1. Abstraction earns its keep. Putting every vendor behind an interface felt like over-engineering until we swapped TTS providers three times without touching application code.

2. De-risk with the simplest transport first. A text simulator proved the entire brain before a single millisecond of audio, making the voice phase a transport problem rather than a system problem.

3. Make the AI/deterministic boundary load-bearing. Treating the tool layer as a hard security wall, allow-lists, approval gates, server-side price computation, full audit is what makes an autonomous agent safe to deploy.

4. Model exact facts as data, not prose. Prices in a table, answers in structured slots. RAG is for knowledge, not for numbers that must be right.

5. Use the framework's real seam. The interruption bug came from hooking in at the wrong place; the fix was the framework's intended extension point.

6. The environment is part of the system. In local-first work, DNS, virtualization, and Docker are as much a part of shipping as the code.

Switchboard began as a text box that talked to Claude and became a voice-capable, tool-using, multi-tenant agent platform, built to run on a laptop and designed so that every hard part could be replaced independently as it grew.