Passengers can soon influence autonomous driving styles by speaking naturally to onboard large language models
Passengers will soon shape autonomous driving styles through natural conversations with in-car large language models.
The inability to influence how an autonomous vehicle handles turns—aggressively or smoothly—remains a major obstacle to widespread adoption. Current systems use motion planners with fixed parameters set by engineers before deployment, leaving passengers without any control over their driving experience. This lack of personal agency makes many people hesitant to trust self-driving cars.
TU Delft researchers have developed a solution that bridges the gap between human preferences and rigid control systems. Their approach uses Large Language Models (LLMs) to translate vague desires into specific driving adjustments without compromising safety. The system avoids letting LLMs control the vehicle directly, which would create dangerous latency and reliability issues, especially since models like GPT-4o-mini can hallucinate or respond unpredictably in critical situations.
The AI workflow operates through several steps:
- A passenger makes a natural language request such as "Drive more smoothly, I feel carsick."
- The system combines this request with contextual information like "Heavy rain, high-speed highway, dense traffic."
- The LLM determines how to modify the controller's weighting based on the passenger's intent.
- The motion planner's priorities—speed, steering smoothness, or collision probability—are adjusted accordingly.
- The vehicle explains the planned change in plain English and waits for confirmation before applying the new driving style.
This method maintains safety by keeping collision avoidance and physics calculations in the hands of the deterministic motion planner. The LLM only activates when high-level preferences change, not during every millisecond of operation. The researchers use a model predictive-path integral controller to evaluate trajectories based on speed optimization, steering angle smoothness, and collision probability.
By modifying the "judging criteria" rather than the paths themselves, the vehicle stays within predictable, safe boundaries while offering a personalized experience. This approach represents a significant advancement in robotics prompt engineering, moving beyond simple "if-this-then-that" logic toward a model where LLMs serve as high-level agents managing complex optimization problems necessary for real-world deployment.
All Replies （4）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Frustrating when Teslas take random detours. Which LLM could actually handle real-time navigation corrections? The primary hurdle for autonomous vehicle adoption is not merely safety, but the loss of agency. Most passengers dislike being passive observers in a machine that makes arbitrary decisions regarding how aggressively or smoothly it navigates a turn. Because current self-driving stacks rely on motion planners hard-coded and tuned by engineers before deployment, there is currently no room for passenger preference. Researchers at TU Delft are addressing this by employing Large Language Models (LLMs) to connect vague human desires with rigid mathematical control systems. Translating "I'm in a hurry" into mathematical parameters Traditional motion planners are deterministic, calculating paths based on strict efficiency and safety constraints. A standard autonomous car will ignore a request to go faster if the passenger is running late because its speed and acceleration parameters are locked. Rather than letting the LLM drive the car directly—which would be a safety risk due to latency and lack of performance guarantees—the TU Delft team uses an LLM (specifically GPT-4o-mini in their tests) as a reasoning layer. This layer translates natural language into adjustments for a model predictive-path integral controller. The AI workflow functions as follows: 1. Input Layer
This is terrifying—what if the voice interface actually had to override the car in an emergency? The TU Delft team’s approach uses LLMs to dynamically adjust driving parameters, like tweaking acceleration limits when you say "I’m in a hurry"—but could it react fast enough to slam the brakes if a child darts into the road? Even with a reasoning layer translating commands into real-time control adjustments, the latency and unpredictability of natural language might still leave a gap between intent and execution.
This is scary. How does the sensor override handle logic conflicts with the main driving stack? Translating "I'm in a hurry" into mathematical parameters.
The latency here scares me. How many milliseconds of delay are we talking about before it's dangerous? One concrete step that could mitigate this is to use LLMs to translate human desires into driving parameters, as researchers at TU Delft are doing. They employ Large Language Models (LLMs) to connect vague human desires with rigid mathematical control systems.