Motional gave safety drivers faithful explanations of a robotaxi's decisions

Autonomous-driving planners can make surprising choices, such as braking in a way that blocks an emergency vehicle, while leaving safety drivers unable to tell what the model saw. Asking people to infer the reason from the car's motion gives them a weak mental model just when they need to predict the next move.

MIT and Motional developed the Concept-Wrapper Network, or CW-Net, as an internal concept classifier rather than a separate explanation layer. Trained on 130 million labeled driving scenes, it translates the planner's hidden reasoning into concepts such as “approaching stopped vehicle” or “close to cyclist,” then forces the final planning stage to use those concepts without changing the vehicle's driving performance.

On a Motional robotaxi's private-track tests, explanations helped safety drivers predict behavior in surprising situations. In one case, the driver thought a stop near a cyclist meant the vehicle had detected the cyclist; CW-Net showed that the cyclist detector was not configured correctly and emergency braking had activated because the car was too close. A larger Las Vegas simulation study with nonexperts found the same improvement in prediction, giving drivers an earlier cue to slow down or take manual control and giving engineers a concrete failure to fix.

Make the planner explain itself in concepts

A post-hoc explanation could sound plausible while misrepresenting the planner's real cause; putting concepts in the decision path makes them causally faithful.

Real-time concepts let a safety driver predict what the vehicle will do, not merely understand it after the maneuver.

The module mimics the existing planner's driving decisions, so interpretability does not require trading away performance.

A large labeled scene set gives the concept classifier coverage across many road situations rather than one hand-picked demo.

In a safety-critical black box, explanation works only when it constrains the decision: insert interpretable concepts into the model's causal path and show them to its human supervisor.

CW-Net was tested on a real autonomous-driving vehicle and in simulation, where it improved human prediction of vehicle behavior. The researchers identified expanding the concept vocabulary and refining training and design methods as next steps. The approach also offers engineers deployment-time feedback for troubleshooting other safety-critical AI systems.