GenAI-agent powered voice assistant for Tesla

The voice assistant, with the Realtime API at its core:

• Triggers continuous audio streaming while also transcribing audio to text with a built-in mechanism. This ability reduces latency, establishes natural conversation flow, and increases accuracy.
• Captures and preserves distinctive phonetic features, such as intonation, pace, pitch, accent, and others.
• Demonstrates a solid understanding of various topics, contexts, idioms, indirect requests, and more.
• Supports function calling, enabling the voice assistant to interpret driver requests, identify underlying intent, and create appropriate responses tailored to drivers’ emotional conditions.
• Uses function calling to connect LLMs with third-party APIs, orchestrated by the agent-based OpenAI Assistant framework, to execute various commands—from ordering items and booking hotel rooms to finding charging stations and creating navigation routes.
• Ensures hands-free voice interactions, eliminating the need to tap the screen on the infotainment system, thereby increasing safety for drivers.

Order creation workflow

The ordering process through the assistant includes the following steps:

1. The Tesla driver says a command, for example, “Find a nearby café.”
2. The driver’s voice is sent directly to the Realtime API as audio streams.
3. The voice assistant captures the incoming audio streams to determine the appropriate function and extract the necessary arguments to fulfill the driver’s request. In this scenario, the function is “find_store,” while café names, coordinates, and other parameters serve as the arguments.
4. The assistant then sends a request to the backend to perform the required function using the given arguments.
5. The backend processes the request and invokes third-party services or APIs, such as the MealMe API and OpenAI, to retrieve a list of nearby restaurants and create an order.
6. The backend returns the result of the function execution in text format, which is then processed by the voice assistant and converted into audio streams, providing a list of nearby restaurants.
7. The user selects the café by vocalizing their choice, and the assistant recommends the next step—navigating to the café or ordering something. If the driver chooses to place an order, the software provides the menu and suggests product customizations, for example, specifying cream for coffee.
8. Once the user chooses the product and customizations, the voice assistant sends a request to the backend to obtain the final quote. Upon confirmation, the software requests the backend to place the order and complete the payment process.

Navigation to charging stations

Navigation to charging stations is another functionality our developers incorporated into Kilowatts. After receiving a command, the voice assistant calls the function “find_chargers” and relays arguments, such as charging speed (high- or low-power), location (current, along the route, or at the destination), and desired amenities (cafés, shops, hotels, dog-friendly restaurants, or restrooms) to the backend. The latter relies on external services, such as the Iternio API, to complete the request according to the driver’s preferences. Capitalizing on this feature allows drivers to easily identify the nearest charging station, keeping their Teslas on the go as long as needed.

Battery level control

The GenAI-driven voice assistant we built includes a feature for battery level control. By asking the driver in advance about the desired energy level at the destination, the software constructs the route to maintain battery capacity as requested. This feature helps optimize routes, reduce energy consumption, and minimize the risk of power loss in remote areas.