Next time you’re driving down the road or walking down the street, pause to consider how you read your surroundings. How you pay extra attention to the kid kicking a soccer ball around her front lawn and the slightly wobbly, nervous looking cyclist. How you deprioritize the woman striding toward the street, knowing she’s heading for the group of friends waving to her from the sidewalk.

You make these calls by drawing on a lifetime of social and cultural experience so ingrained you hardly need to think about it. But imagine you’re an autonomous car trying to do the same thing, without that accumulated knowledge or the shared humanity that lets you read others’ nuanced behavioral cues. Treating every pedestrian, cyclist, and vehicle as an obstacle to be avoided might keep you from hitting anything, but it could just as easily keep you from getting anywhere.

“We call it the freezing robot problem,” says Anca Dragan, who studies autonomy in UC Berkeley’s electric engineering and computer sciences department. “Anything the car could do is too risky, because there is some worst-case human action that would lead to a collision.”

Expect a thaw. Researchers like Dragan are tackling the challenges of interpreting—and predicting—human behavior to make self-driving cars safer and more efficient, but also more assertive. After all, if every machine screeches to a stop for every unpredictable human, we’ll have soon millions of terrified robots choking the streets.

To prevent the clog, those researchers are leaning on artificial intelligence and the ability to teach driving systems, through modeling and repetitive observation, what behaviors mean what, and how the system should react to them.

That begins with recognizing that people are not, in fact, obstacles. “Unlike, say, a tumbleweed moving along the street under the wind’s effect, people move because they make decisions,” Dragan says. “They want to do something, and they act to achieve it. We’re first looking into inferring what people want based on the actions they’ve been taking so far. So their actions are rational when seen from [that perspective], and would appear irrational when seen from the perspective of other intentions.”

Say a driver in the right lane of the freeway accelerates. The computer knows people should slow down as they approach exits, and can infer this person is likely to continue straight ahead instead of taking that upcoming off ramp. It’s a basic example that makes the point: Once computers can estimate what humans want and how they might achieve it, they can reasonably predict what they’ll do next, and react accordingly.

Machines en Scene

The key, even with machine learning, is to look beyond the individual elements of a scene. “It’s important to make strides there, but it’s only seeing part of what’s going on in a roadway setting,” says Melissa Cefkin, a design anthropologist at Nissan’s Silicon Valley R&D center. “We’re really good as human beings at recognizing certain kinds of behaviors that look one way to a machine, but in our social lens, it’s something else.”

Imagine you’re driving down a city block when you see a man walking toward the curb. The robot driver might calculate his speed and trajectory, determine he’s about to cross the street, and stop to avoid hitting him. But you see he’s holding car keys, and realize he’s stepping into the street to reach the driver’s side door of his parked car. You’ll slow down to be sure, but no need to stop traffic.

“The ways people move through the environment are already culturally and socially encoded,” Cefkin says. “It’s not always people-to-people interactions, but people interacting with things, too.”

Again, that’s a simple example. Cefkin points to what she calls the “multi-agent problem,” in which pedestrians and other drivers react to everyone around them. “If a pedestrian is going to cross in front of me, rather than looking at me they’re just as likely to look out into traffic for a gap,” Cefkin says. “So now I’m trying to figure out whether or not it’s safe to keep going based on what the rest of the traffic is going to do.”

Buying Time

If it seems the world is now headed for some sort of drivers-ed hellscape, don’t worry. Teaching AI-based autonomous systems to navigate the eternal weirdness of the human wilderness is tough, Cefkin says, but hardly impossible. In the Netherlands, where cities buzz with pedestrians and cyclists, researchers are doing the work. Dariu Gavrila, who researches intelligent vehicles at Delft University of Technology, training computers for the challenges like road debris, traffic police, and, things as unusual as someone pushing a cart down the middle of the street. The goal, he says, is to develop a more adaptive driving style for the machines—and thus enhancing social acceptance of the new hardware.

That work means factoring in the context around pedestrian traffic—proximity to curbs, the presence of driveways or public building entrances—and the norms of behavior in these environments. It extends all the way to individual movement, such as a person’s head looking one direction while their torso is pointing in another, and what that might mean. “Recognizing pedestrian intent can be a life saver,” Gavrila says. “We showed in real vehicle demonstration that an autonomous system can react up to one second faster than a human, without introducing false alarms.”

There are practical limits to what the computers can do, though. “This is no Minority Report,” Gavrila says—no one’s telling the future. “Uncertainty in future pedestrian or cyclist position rapidly increases with the prediction horizon, how many seconds in the future we’re trying to model. Basic behavior models already stop being useful after one second. More sophisticated behavior models might give us up to two seconds of predictability.”

Still, that second or two of warning might be all a computerized system needs, since it’s well within the scope of the human response times. But other autonomy experts think we might be setting our machines up to actually overthink every microsecond of driving.

“When you’re essentially trying to predict the future, that’s a massive computational task, and of course it just produces a probabilistic guess,” says Jack Weast, Intel’s chief systems architect for autonomous drive systems. “So rather than throw a supercomputer into every car, we just want to ensure that the car’s never going to hit any of those people anyway. It’s a much more economically scalable way of doing things.”

Getting Aggressive

There’s another wrinkle here. The ideal robocar won’t just comprehend its surroundings, it will understand how it itself changes the scene. Many robotic systems, Dragan says, come with a built-in flaw: Their makers assume the presence of an autonomous car won’t change how other actors move. “An autonomous car’s actions will influence human actions, whether we like it or not,” she says. “Cars need to start accounting for this influence.”

That’s why Dragan and her team have built a system that includes a model of human drivers’ responses to the car. “Our car is no longer ultra-defensive, because it knows it can trigger reactions from people, too,” she says. “Like other vehicles slowing down when our car merges in front of them. We’ve also looked at actively estimating human intentions, again by leveraging the autonomous car’s actions. In that case, our car might slow down gently to see if the person wants to be let in.”

That sort of assertiveness training will likely be key to traffic flow in the future. The key to a working robocar may be giving it not just human-like awareness, but a healthy dose of human-like entitlement.