How to Build an AI Voice Agent with Retell AI for a Doctor's Office

Building an AI voice agent with Retell AI for a doctor's office means generating a knowledge-base prompt from the practice's own website, loading it into Retell AI with GPT-4.1 as the model, connecting a real calendar through Cal.com so it can book appointments, tuning voice and transcription settings for natural conversation, and configuring security settings so sensitive patient information is never collected or stored.

What the finished agent sounds like on a real call

Before getting into the build, it's worth hearing what the finished product actually does. A caller dials in wanting to set up a consultation. The agent asks what procedure or area they want to discuss, asks whether they'd prefer an in-person or virtual consultation, checks real availability on the calendar, and offers specific open times. When the caller changes their mind about the day, the agent adjusts and re-checks availability rather than breaking the conversation. Once a time is agreed, it books the appointment and confirms the details back to the caller — in this case, scheduling a tummy tuck consultation at the practice's office.

The agent can also go beyond scheduling: it answered follow-up questions about what other services the practice offers (facial procedures, breast procedures, body procedures, non-surgical treatments), explained what a "mommy makeover" is as a combination of procedures, and handled an awkward real-time request to change the phone number used for appointment reminders. It also correctly recognized that the call was happening outside business hours and explained that a transfer to a live coordinator or doctor wouldn't go through until the office opened, while still offering to help with scheduling in the meantime.

Start with the knowledge base: let AI write the prompt

The system breaks down into three general parts: the prompt (the knowledge base, or "the brain" of the agent), the settings in the middle, and the testing tools on the right. You don't have to write the whole prompt by hand. The fastest way to build it is to take the business's own website — one that you may have already built for the client — and give the URL to ChatGPT, asking it to generate a prompt you can use in Retell AI.

ChatGPT effectively scrapes the site for the important information: what services the business offers (in this example, procedures like tummy tucks and other plastic surgery services), and it even drafts example Q&A pairs covering the kinds of questions real customers ask, like setting up appointments. That scraped, structured output becomes your starting knowledge base.

It's worth stress-testing the result once it's live. Asking the agent random, off-script, even slightly absurd questions — whether it's single, what time the office closes, what number it's calling from — is a fast way to see how well it holds up under real conversational pressure rather than just reading from a script.

Pick the right model: GPT-4.1 for voice, for now

In the top left of the Retell AI interface, you'll see the model selector. For voice agent work specifically, GPT-4.1 is the recommended choice. Newer models exist, but they're still too fresh to be as efficient and accurate for real-time voice conversations as GPT-4.1 currently is. If you have an enterprise account with Retell AI, their team will also walk you through call settings tailored to your specific use case over a direct call.

Set up the functions: ending calls and transferring to a human

The middle section of the builder holds the agent's functions — logic instructions that tell Retell AI what to do in specific situations. An "end call" function, for example, tells the agent to end the call once the customer has successfully scheduled their appointment. A transfer-call function routes the caller to a human when that's the right move — though as the real call demonstrated, that transfer will correctly fail (and explain why) if it happens outside business hours.

Connect a real calendar through Cal.com

This is the step that turns the agent from "a nice conversation" into something that actually schedules appointments. Cal.com is a free calendar booking platform. You create an API key inside Cal.com — a small piece of code that lets Retell AI and Cal.com communicate — and paste it into the corresponding function in Retell AI. You also need the event type ID: inside Cal.com's event types, each kind of booking (a consultation, a 30-minute video call, a one-on-one onboarding slot) has its own ID in the URL, and that ID tells the agent which specific calendar event to check and book against.

Once that's wired up, the calendar connects automatically to a linked Google Calendar, so every appointment the agent books shows up there without any extra manual syncing. If a business runs multiple types of appointments, each needs its own matching API key and event type ID so the agent books the right kind of slot every time.

Tune the voice and conversation settings

The agent's voice comes from ElevenLabs, a popular AI audio platform, and Retell AI gives you a library of imported voices to choose from. A handful of settings meaningfully change how natural the conversation feels:

• Ambient noise: a background setting like a subtle coffee-shop ambience makes the call feel less sterile than dead silence.
• Response eagerness: how quickly the agent starts responding after the caller finishes talking — set high (around 0.95) so it doesn't leave an awkward pause.
• Interruption sensitivity: how the agent handles being talked over mid-sentence, tuned around 0.75 so it responds naturally to interruptions without getting confused.
• AI speaks first: whether the agent or the caller speaks first when the call connects — typically set so the agent greets the caller immediately.
• Transcription settings: real-time transcription can be set to optimize for speed or for accuracy; accuracy performs better for understanding the back-and-forth between caller and agent, and noise settings can be tuned to filter out background speech specifically, not just background noise.

Handle sensitive data and HIPAA compliance

Under security and fallback settings, you control how the agent handles personally identifiable information — social security numbers, dates of birth, and similar sensitive data. For HIPAA-conscious setups, the agent is configured not to ask for or retain that information, using a basic contact method like email instead of storing sensitive identifiers. Post-call data extraction can still capture a call summary (what was discussed, whether an appointment was booked) and even a sentiment read on how the caller felt about the interaction, without needing to touch protected health information to do it. That data can then be routed to other tools, like automation platforms, through webhooks or MCP connections.

Beyond a single prompt agent

Everything above describes a single-prompt agent, which is enough for most service businesses. For situations that need more complexity — specific information gathered before a transfer happens, or calls routed to different people depending on what the caller says — Retell AI supports deeper customization, and tools like Vapi AI exist for setups that need to go even further than that.

Key takeaways

• Generate the starting knowledge base by feeding the business's own website to ChatGPT and asking for a Retell AI prompt — don't write it from scratch.
• GPT-4.1 is currently the recommended model for real-time voice agent conversations.
• Cal.com handles real calendar availability and booking through an API key and an event type ID, synced automatically to Google Calendar.
• Voice and conversation settings — ambient noise, response eagerness, interruption sensitivity, transcription accuracy — are what separate a robotic-sounding agent from a natural one.
• HIPAA-sensitive setups should configure security settings to avoid collecting or storing personally identifiable information at all.
• This same build applies beyond doctors' offices — to dentists, plumbers, landscapers, cleaning companies and most other appointment-driven service businesses.

Want a voice agent built for your service business?

A Retell AI voice agent like this one can be adapted to handle inbound calls for almost any appointment-based business.