Why we put too much trust in machines
In other words, we’re not only bad at weighing a machine’s reliability, we’re also easily seduced by mechanical objects when they start to behave a bit like a social partner who has our best interests at heart. .
So, unlike the person who has been stung by unresponsive computers and therefore distrusts everyone, the person who has learned to trust certain systems – such as aircraft autopilots – may find it difficult to understand how wrong she can be, even when they are.
De Petrillo notes that she has experienced a sense of trust in computers when interacting with voice-activated assistants like Apple’s Siri or Amazon’s Alexa.
“I assume they’re acting in my best interest, so I don’t need to question them,” she says. As long as they appear competent and reasonably warm, Kulms’ study suggests that this will continue to be the case, for de Petrillo and many others. Kulm points out that’s why it’s so important for tech designers to make sure their systems are ethical and their functionality is transparent.
The great irony of all this is that behind a seemingly trustworthy machine may be hiding a malevolent human with nefarious intentions. Undue reliance on a faulty machine is dangerous enough, let alone a machine designed to deceive.
“If we were taking the same kind of risk with a human, we would be much more alert to what the potential negative outcomes might be,” Proctor says. She and Haux agree that more work is needed to determine how much chimpanzees in their studies actually trust machines and how much that reveals truths about human behavior.
But there’s a hint here that our occasional, sometimes disastrous, over-reliance on technology was influenced by a simple fact: we evolved to be social animals in a world where machines didn’t exist. Now they do and we put our money, our personal data and even our lives under their surveillance all the time. It’s not necessarily the wrong thing to do – it’s just that we’re often pretty bad at judging when it’s right.
If you liked this story, sign up for the weekly bbc.com features newslettertitled ‘The Essential List’ – a handpicked selection of stories from the BBC Future, Culture, working life, To travel and Reel delivered to your inbox every Friday.