The Glass Cage: Automation and Us (18 page)

Even when aimed at questions amenable to probabilistic answers, computer analysis is not flawless. The speed and apparent exactitude of computer calculations can mask limitations and distortions in the underlying data, not to mention imperfections in the data-mining algorithms themselves. Any large data set holds an abundance of spurious correlations along with the reliable ones. It’s not hard to be misled by mere coincidence or to conjure a phantom association.
⁴⁷
Once a particular data set becomes the basis for important decisions, moreover, the data and its analysis become vulnerable to corruption. Seeking financial, political, or social advantage, people will try to game the system. As the social scientist Donald T. Campbell explained in a renowned 1976 paper, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
⁴⁸

Flaws in data and algorithms can leave professionals, and the rest of us, susceptible to an especially pernicious form of automation bias. “The threat is that we will let ourselves be mindlessly bound by the output of our analyses even when we have reasonable grounds for suspecting something is amiss,” warn Viktor Mayer-Schönberger and Kenneth Cukier in their 2013 book
Big Data
. “Or that we will attribute a degree of truth to the data which it does not deserve.”
⁴⁹
A particular risk with correlation-calculating algorithms stems from their reliance on data about the past to anticipate the future. In most cases, the future behaves as expected; it follows precedent. But on those peculiar occasions when conditions veer from established patterns, the algorithms can make wildly inaccurate predictions—a fact that has already spelled disaster for some highly computerized hedge funds and brokerage firms. For all their gifts, computers still display a frightening lack of common sense.

The more we embrace what Microsoft researcher Kate Crawford terms “data fundamentalism,”
⁵⁰
the more tempted we’ll be to devalue the many talents computers can’t mimic—to grant so much control to software that we restrict people’s ability to exercise the know-how that comes from real experience and that can often lead to creative, counterintuitive insights. As some of the unforeseen consequences of electronic medical records show, templates and formulas are necessarily reductive and can all too easily become straightjackets of the mind. The Vermont doctor and medical professor Lawrence Weed has, since the 1960s, been a forceful and eloquent advocate for using computers to help doctors make smart, informed decisions.
⁵¹
He’s been called the father of electronic medical records. But even he warns that the current “misguided use of statistical knowledge” in medicine “systematically excludes the individualized knowledge and data essential to patient care.”
⁵²

Gary Klein, a research psychologist who studies how people make decisions, has deeper worries. By forcing physicians to follow set rules, evidence-based medicine “can impede scientific progress,” he writes. Should hospitals and insurers “mandate EBM, backed up by the threat of lawsuits if adverse outcomes are accompanied by any departure from best practices, physicians will become reluctant to try alternative treatment strategies that have not yet been evaluated using randomized controlled trials. Scientific advancement can become stifled if front-line physicians, who blend medical expertise with respect for research, are prevented from exploration and are discouraged from making discoveries.”
⁵³

If we’re not careful, the automation of mental labor, by changing the nature and focus of intellectual endeavor, may end up eroding one of the foundations of culture itself: our desire to understand the world. Predictive algorithms may be supernaturally skilled at discovering correlations, but they’re indifferent to the underlying causes of traits and phenomena. Yet it’s the deciphering of causation—the meticulous untangling of how and why things work the way they do—that extends the reach of human understanding and ultimately gives meaning to our search for knowledge. If we come to see automated calculations of probability as sufficient for our professional and social purposes, we risk losing or at least weakening our desire and motivation to seek explanations, to venture down the circuitous paths that lead toward wisdom and wonder. Why bother, if a computer can spit out “the answer” in a millisecond or two?

In his 1947 essay “Rationalism in Politics,” the British philosopher Michael Oakeshott provided a vivid description of the modern rationalist: “His mind has no atmosphere, no changes of season and temperature; his intellectual processes, so far as possible, are insulated from all external influence and go on in the void.” The rationalist has no concern for culture or history; he neither cultivates nor displays a personal perspective. His thinking is notable only for “the rapidity with which he reduces the tangle and variety of experience” into “a formula.”
⁵⁴
Oakeshott’s words also provide us with a perfect description of computer intelligence: eminently practical and productive and entirely lacking in curiosity, imagination, and worldliness.

*
The use of terms like
neural network
and
neuromorphic processing
may give the impression that computers operate the way brains operate (or vice versa). But the terms shouldn’t be taken literally; they’re figures of speech. Since we don’t yet know how brains operate, how thought and consciousness arise from the interplay of neurons, we can’t build computers that work as brains do.

WORLD AND SCREEN

T
HE SMALL ISLAND OF
I
GLOOLIK
, lying off the coast of the Melville Peninsula in the Nunavut territory of the Canadian North, is a bewildering place in the winter. The average temperature hovers around twenty degrees below zero. Thick sheets of sea ice cover the surrounding waters. The sun is absent. Despite the brutal conditions, Inuit hunters have for some four thousand years ventured out from their homes on the island and traversed miles of ice and tundra in search of caribou and other game. The hunters’ ability to navigate vast stretches of barren Arctic terrain, where landmarks are few, snow formations are in constant flux, and trails disappear overnight, has amazed voyagers and scientists ever since 1822, when the English explorer William Edward Parry noted in his journal the “astonishing precision” of his Inuit guide’s geographic knowledge.
¹
The Inuit’s extraordinary wayfinding skills are born not of technological prowess—they’ve eschewed maps, compasses, and other instruments—but of a profound understanding of winds, snowdrift patterns, animal behavior, stars, tides, and currents. The Inuit are masters of perception.

Or at least they used to be. Something changed in Inuit culture at the turn of the millennium. In the year 2000, the U.S. government lifted many of the restrictions on the civilian use of the global positioning system. The accuracy of GPS devices improved even as their prices dropped. The Igloolik hunters, who had already swapped their dogsleds for snowmobiles, began to rely on computer-generated maps and directions to get around. Younger Inuit were particularly eager to use the new technology. In the past, a young hunter had to endure a long and arduous apprenticeship with his elders, developing his wayfinding talents over many years. By purchasing a cheap GPS receiver, he could skip the training and offload responsibility for navigation to the device. And he could travel out in some conditions, such as dense fog, that used to make hunting trips impossible. The ease, convenience, and precision of automated navigation made the Inuit’s traditional techniques seem antiquated and cumbersome by comparison.

But as GPS devices proliferated on Igloolik, reports began to spread of serious accidents during hunts, some resulting in injuries and even deaths. The cause was often traced to an overreliance on satellites. When a receiver breaks or its batteries freeze, a hunter who hasn’t developed strong wayfinding skills can easily become lost in the featureless waste and fall victim to exposure. Even when the devices operate properly, they present hazards. The routes so meticulously plotted on satellite maps can give hunters a form of tunnel vision. Trusting the GPS instructions, they’ll speed onto dangerously thin ice, over cliffs, or into other environmental perils that a skilled navigator would have had the sense and foresight to avoid. Some of these problems may eventually be mitigated by improvements in navigational devices or by better instruction in their use. What won’t be mitigated is the loss of what one tribal elder describes as “the wisdom and knowledge of the Inuit.”
²

The anthropologist Claudio Aporta, of Carleton University in Ottawa, has been studying Inuit hunters for years. He reports that while satellite navigation offers attractive advantages, its adoption has already brought a deterioration in wayfinding abilities and, more generally, a weakened feel for the land. As a hunter on a GPS-equipped snowmobile devotes his attention to the instructions coming from the computer, he loses sight of his surroundings. He travels “blindfolded,” as Aporta puts it.
³
A singular talent that has defined and distinguished a people for thousands of years may well evaporate over the course of a generation or two.

T
HE WORLD
is a strange, changeable, and dangerous place. Getting around in it demands of any animal a great deal of effort, mental and physical. For ages, human beings have been creating tools to reduce the strain of travel. History is, among other things, a record of the discovery of ingenious new ways to ease our passage through our environs, to make it possible to cross greater and more daunting distances without getting lost, roughed up, or eaten. Simple maps and trail markers came first, then star maps and nautical charts and terrestrial globes, then instruments like sounding weights, quadrants, astrolabes, compasses, octants and sextants, telescopes, hourglasses, and chronometers. Lighthouses were erected along shorelines, buoys set in coastal waters. Roads were paved, signs posted, highways linked and numbered. It has, for most of us, been a long time since we’ve had to rely on our wits to get around.

GPS receivers and other automated mapping and direction-plotting devices are the latest additions to our navigational toolkit. They also give the old story a new and worrisome twist. Earlier navigational aids, particularly those available and affordable to ordinary folks, were just that: aids. They were designed to give travelers a greater awareness of the world around them—to sharpen their sense of direction, provide them with advance warning of danger, highlight nearby landmarks and other points of orientation, and in general help them situate themselves in both familiar and alien settings. Satellite navigation systems can do all those things, and more, but they’re not designed to deepen our involvement with our surroundings. They’re designed to relieve us of the need for such involvement. By taking control of the mechanics of navigation and reducing our own role to following routine commands—turn left in five hundred yards, take the next exit, stay right, destination ahead—the systems, whether running through a dashboard, a smartphone, or a dedicated GPS receiver, end up isolating us from the environment. As a team of Cornell University researchers put it in a 2008 paper, “With the GPS you no longer need to know where you are and where your destination is, attend to physical landmarks along the way, or get assistance from other people in the car and outside of it.” The automation of wayfinding serves to “inhibit the process of experiencing the physical world by navigation through it.”
⁴

As is so often the case with gadgets and services that ease our way through life, we’ve celebrated the arrival of inexpensive GPS units. The
New York Times
writer David Brooks spoke for many when, in a 2007 op-ed titled “The Outsourced Brain,” he raved about the navigation system installed in his new car: “I quickly established a romantic attachment to my GPS. I found comfort in her tranquil and slightly Anglophilic voice. I felt warm and safe following her thin blue line.” His “GPS goddess” had “liberated” him from the age-old “drudgery” of navigation. And yet, he grudgingly confessed, the emancipation delivered by his in-dash muse came at a cost: “After a few weeks, it occurred to me that I could no longer get anywhere without her. Any trip slightly out of the ordinary had me typing the address into her system and then blissfully following her satellite-fed commands. I found that I was quickly shedding all vestiges of geographic knowledge.” The price of convenience was, Brooks wrote, a loss of “autonomy.”
⁵
The goddess was also a siren.

We want to see computer maps as interactive, high-tech versions of paper maps, but that’s a mistaken assumption. It’s yet another manifestation of the substitution myth. Traditional maps give us context. They provide us with an overview of an area and require us to figure out our current location and then plan or visualize the best route to our next stop. Yes, they require some work—good tools always do—but the mental effort aids our mind in creating its own cognitive map of an area. Map reading, research has shown, strengthens our sense of place and hones our navigational skills—in ways that can make it easier for us to get around even when we don’t have a map at hand. We seem, without knowing it, to call on our subconscious memories of paper maps in orienting ourselves in a city or town and determining which way to head to arrive at our destination. In one revealing experiment, researchers found that people’s navigational sense is actually sharpest when they’re facing north—the same way maps point.
⁶
Paper maps don’t just shepherd us from one place to the next; they teach us how to think about space.