The big idea: Should we worry about artificial intelligence?

Could AI turn on us, or is natural stupidity a greater threat to humanity?

Ever since Garry Kasparov lost his second chess match against IBM’s Deep Blue in 1997, the writing has been on the wall for humanity. Or so some like to think. Advances in artificial intelligence will lead – by some estimates, in only a few decades – to the development of superintelligent, sentient machines. Movies from The Terminator to The Matrix have portrayed this prospect as rather undesirable. But is this anything more than yet another sci-fi “Project Fear”?

Some confusion is caused by two very different uses of the phrase artificial intelligence. The first sense is, essentially, a marketing one: anything computer software does that seems clever or usefully responsive – like Siri – is said to use “AI”. The second sense, from which the first borrows its glamour, points to a future that does not yet exist, of machines with superhuman intellects. That is sometimes called AGI, for artificial general intelligence.

How do we get there from here, assuming we want to? Modern AI employs machine learning (or deep learning): rather than programming rules into the machine directly we allow it to learn by itself. In this way, AlphaZero, the chess-playing entity created by the British firm Deepmind (now part of Google), played millions of training matches against itself and then trounced its top competitor. More recently, Deepmind’s AlphaFold 2 was greeted as an important milestone in the biological field of “protein-folding”, or predicting the exact shapes of molecular structures, which might help to design better drugs.

Machine learning works by training the machine on vast quantities of data – pictures for image-recognition systems, or terabytes of prose taken from the internet for bots that generate semi-plausible essays, such as GPT2. But datasets are not simply neutral repositories of information; they often encode human biases in unforeseen ways. Recently, Facebook’s news feed algorithm asked users who saw a news video featuring black men if they wanted to “keep seeing videos about primates”. So-called “AI” is already being used in several US states to predict whether candidates for parole will reoffend, with critics claiming that the data the algorithms are trained on reflects historical bias in policing.

Computerised systems (as in aircraft autopilots) can be a boon to humans, so the flaws of existing “AI” aren’t in themselves arguments against the principle of designing intelligent systems to help us in fields such as medical diagnosis. The more challenging sociological problem is that adoption of algorithm-driven judgments is a tempting means of passing the buck, so that no blame attaches to the humans in charge – be they judges, doctors or tech entrepreneurs. Will robots take all the jobs? That very framing passes the buck because the real question is whether managers will fire all the humans.

The existential problem, meanwhile, is this: if computers do eventually acquire some kind of god‑level self-aware intelligence – something that is explicitly in Deepmind’s mission statement, for one (“our long-term aim is to solve intelligence” and build an AGI) – will they still be as keen to be of service? If we build something so powerful, we had better be confident it will not turn on us. For the people seriously concerned about this, the argument goes that since this is a potentially extinction-level problem, we should devote resources now to combating it. The philosopher Nick Bostrom, who heads the Future of Humanity Institute at the University of Oxford, says that humans trying to build AI are “like children playing with a bomb”, and that the prospect of machine sentience is a greater threat to humanity than global heating. His 2014 book Superintelligence is seminal. A real AI, it suggests, might secretly manufacture nerve gas or nanobots to destroy its inferior, meat-based makers. Or it might just keep us in a planetary zoo while it gets on with whatever its real business is.

AI wouldn’t have to be actively malicious to cause catastrophe. This is illustrated by Bostrom’s famous “paperclip problem”. Suppose you tell the AI to make paperclips. What could be more boring? Unfortunately, you forgot to tell it when to stop making paperclips. So it turns all the matter on Earth into paperclips, having first disabled its off switch because allowing itself to be turned off would stop it pursuing its noble goal of making paperclips.

Related: The big idea: are we really so polarised? | Dominic Packer and Jay Van Bavel

That’s an example of the general “problem of control”, subject of AI pioneer Stuart Russell’s excellent Human Compatible: AI and the Problem of Control, which argues that it is impossible to fully specify any goal we might give a superintelligent machine so as to prevent such disastrous misunderstandings. In his Life 3.0: Being Human in the Age of Artificial Intelligence, meanwhile, the physicist Max Tegmark, co-founder of the Future of Life Institute (it’s cool to have a future-of-something institute these days), emphasises the problem of “value alignment” – how to ensure the machine’s values line up with ours. This too might be an insoluble problem, given that thousands of years of moral philosophy have not been sufficient for humanity to agree on what “our values” really are.

Other observers, though, remain phlegmatic. In Novacene, the maverick scientist and Gaia theorist James Lovelock argues that humans should simply be joyful if we can usher in intelligent machines as the logical next stage of evolution, and then bow out gracefully once we have rendered ourselves obsolete. In her recent 12 Bytes, Jeanette Winterson is refreshingly optimistic, supposing that any future AI will be at least “unmotivated by the greed and land-grab, the status-seeking and the violence that characterises Homo sapiens”. As the computer scientist Drew McDermott suggested in a paper as long ago as 1976, perhaps after all we have less to fear from artificial intelligence than from natural stupidity.

Further reading

Human Compatible: AI and the Problem of Control by Stuart Russell (Penguin, £10.99)

Life 3.0: Being Human in the Age of Artificial Intelligence by Max Tegmark (Penguin, £10.99)

12 Bytes: How We Got Here, Where We Might Go Next by Jeannette Winterson (Jonathan Cape, £16.99)