Surprise!

Inaugural H.V. McKay Lecture:
The Future and Its Enemies:  The Growing Conflict over Creativity, Enterprise and Progress
14 November 2000

UNCERTAINTY IS AN ESSENTIAL INGREDIENT OF PROGRESS

FROM the late nineteenth century through the middle of the twentieth, futurists imagined electric lighting, but no electric guitars;  supersonic jets, but no hang gliders;  laser weapons, but no laser surgery or compact discs;  giant computer databases, but no Palm Pilots or video games;  nuclear power, but no nuclear medicine;  government surveillance cameras, but no baby monitors.

These stunted visions -- produced by social critics and science-fiction writers -- are neither random nor isolated.  Optimists and pessimists alike conceived of the future -- our present -- as a uniform society, a flattened, unnuanced world designed by a few smart men.  They didn't imagine the quirky products of creativity applied to small-scale, personal problems and passions.  They didn't factor in the power of vanity, self-expression, chance, novelty or fun.  Theirs was a future without surprise.

The infatuation with predictability has been deeply imprinted on modern times.  From the communist regimes to corporate giants, we came up in an age of central design, planning and control.  The leading futurists, the science-fiction writers, long depicted progress as the product of elites.  In his book Paris in the Twentieth Century -- written in 1863, but not published until 1996 -- science-fiction master Jules Verne wrote of "an age when everything was centralised, thought as well as mechanical power".

The unplanned outcomes that emerge from obsessive tinkering, competitive one-upmanship, incremental improvement and unarticulated longings have no place in a rigidly planned world.  As it turns out, however, they define the world in which we live -- and they will define our future.  For as we're now discovering, the future, in fact, is made of surprise.

Even the science-fiction writers nowadays recognise the inevitability of surprise.  They "see themselves more as conceptual gardeners, planting for fruitful growth, rather than engineers designing eternal, gray social machines," writes Gregory Benford, the author of such popular science fiction as Timescape and Cosm.  "Their views of that future are often playful, seeking to achieve an almost impressionistic effect, imagining small scattered details ... that imply more than they can say".

Small, scattered details aren't just writing techniques.  They're also fuel for social and economic propulsion.  Important things happen out of sight, often tapping occluded desires.  Cultural critics, on the Right and Left, still argue over where "the sixties" came from, as if someone designed them.  Cigar-and-martini bars, The Blair Witch Project and green-and-purple nail polish from Urban Decay, an upstart cosmetics company in California, all took the world by surprise -- unpredicted and unpredictable.  So, less fleetingly, did the Web.

Technology pundits searched in Silicon Valley for a challenge to Washington State's Microsoft Corporation, but never expected the alternative:  the free Linux computer operating system, created by hobbyists dispersed around the world who, for fun and hacker prestige, work incessantly to improve it.  Linux keeps getting better because thousands of Linux hackers think it's cool to look for bugs.  It's an "open source" system, whose code is available to anyone who wants to see it.  Linux welcomes ideas and improvements from people anywhere.

"Incessant search by many minds," wrote the late political scientist Aaron Wildavsky, "produces more [and more valuable] knowledge than the attempt to program the paths to discovery by a single one".  Professor Wildavsky, who taught at the University of California, Berkeley, could have been writing about Linux.  But the open systems Professor Wildavsky had in mind were social:  science, democracy, markets.  These competitive systems encourage scattered knowledge to emerge.  They allow for serendipity and thus for surprise.

Surprise drives progress because innovation depends on the sort of knowledge no one can gather in a central place.  The Austrian-born economist Friedrich A. Hayek, who won the Nobel Prize in 1974, applied this insight to market prices.  In his 1945 article, "The Use of Knowledge in Society," he argued that markets operate as a "system of telecommunications".  Prices relay scattered information about what people want, what producers have to offer, and how relative scarcities are changing.

But prices aren't the only way markets transmit information.  Markets also allow people with new ideas to test their hypotheses.  While other discount retailers concentrated on urban markets, Sam Walton, the founder of Wal-Mart, built stores in small towns, clustering them around central warehouses.  He guessed correctly that he had a good model for improving retailing.  But only by trying it out, in competition with other retailers, could he be sure.

This divining function of the market has been overlooked by futurist visions, some of which imagined new products arising from obvious, articulated consumer demands, not entrepreneurial inventions.  Edward Bellamy's 1888 novel Looking Backward, a monster bestseller in its day, hails a year 2000 in which nothing new is created unless someone explicitly asks the authorities to provide it.  "Suppose an article not before produced is demanded," explains a character to Mr Bellamy's nineteenth-century time traveller.  "If the administration doubts the reality of the demand, a popular petition guaranteeing a certain basis of consumption compels it to produce the desired article".

Not surprisingly, no one fills out a request for rock music, jacuzzis or Vidal Sassoon-style blunt haircuts.  Bellamy's 2000 is a year in which furniture and clothing have barely changed in a century.

Just as producers often give consumers things they want, but didn't think to ask for, consumers sometimes come up with surprising uses for new inventions.  When a new product appears, it can uncover dissatisfactions and desires no-one knew were there.  Minnesota Mining and Manufacturing Company (3M) first developed cellophane tape so a bakery could seal moisture-proof packages.  The bakery, like Bellamy's order-placing consumers, was indeed able to articulate its demand for something that didn't yet exist.  But a lot of other potential customers wanted things they hadn't expressed.  No sooner was Scotch tape on the market than people started finding new uses for it:  wrapping packages, repairing ripped curtains, making labels, even lining the ribs of dirigibles.

Similarly, Starbucks originally envisioned its shops as take-out stores in busy business districts.  It found instead that customers wanted neighbourhood hangouts -- stores in residential areas did much better than expected -- and adjusted its strategy accordingly.  No customer, and certainly no planner, "ordered" neighbourhood Starbucks.  The company itself was taken by surprise.

To discover what people really want, markets have to be left open to new ideas.  Picking winners in advance, as political planners often attempt, can shut down surprise and discovery.  Laws tend to get in the way of this innovative process and only rarely get changed.

When Starbucks first moved into San Francisco, it found that its neighbourhood hangouts were illegal:  new restaurants had been outlawed in residential areas.  Under these zoning restrictions, Starbucks could sell coffee, but it couldn't put in chairs.  By that time, Starbucks was a well-established brand, with enough clout to get the law changed.  The city created a new category, "beverage houses".  The change accommodated Starbucks, but it won't help the next upstart with a new idea.

Central control also must keep categories rigid.  The characters Mr Bellamy depicted in 1888 could listen to any music they liked at any time of the day or night -- performed live and delivered via the telephone system.  They could choose, for instance, between a waltz and organ music -- extraordinary choices for a nineteenth-century person who was lucky to sing around a parlour piano.  But the songs came from orchestras that played the same old genres, and the playlist would fill a single bin in a contemporary music store.

In the actual world, by contrast, dynamic new markets for radio and recordings gave rise to new genres, new instruments, new institutions.  In Japan, Sony Corporation took the kind of "serious" cutting-edge technologies that planners would have reserved for military and scientific purposes, and applied them to making music personal, first through transistor radios and later through an equally small cassette player called the Walkman.

Even the science underlying Sony's inventions was fundamentally unpredictable, a product of "incessant search by many minds".  As physicist Freeman Dyson, of the Institute for Advanced Studies in Princeton, N.J., has noted, "A 19th-century development program aimed at the mechanical reproduction of music might have produced a superbly engineered music box or Pianola [player piano], but it would never have imagined a transistor radio".  Neither, he notes, would such a programme have subsidised the electromagnetic research of James Clerk Maxwell, the nineteenth-century Scottish physicist whose work led to the development of the transistor radio.

The quiet, unpredictable way that desires and creativity match up in the marketplace disturbs and baffles many people.  It especially confounds people who excel in articulation, since those who are good at self-expression have an advantage in a world where everything is explained in words.  They would rather stick to plans made in public, with everything spelled out in advance and all new ideas firmly under control.  That planning, which intellectuals sometimes equate with "democracy", rewards the ability to explain and argue.  It discourages the restless pursuit and real-world testing of new ideas.

This wariness of uncontrolled creativity recalls the Bellamy model:  carefully articulated wants, approved in advance by a central administration.  In an ironic twist, our mainstream literary culture clings to the ideal of an engineered future, while the actual engineers and money-grubbers embrace creativity and surprise.  It's a strange world indeed.