Great innovators understand experimentation is the lifeblood of

innovation. Good experimentation helps them to better manage sources of

uncertainty (such as, does the product work as intended and does it
address actual customer needs?) when past experience can be limiting.
And it is only through such experimentation, which might include
structured cause-and-effect tests, informal trial-and-error experiments,
and rigorous randomized field trials, that companies can unlock their true
capacity for innovation.
In the real world, however, things are much more complex: Environments are
constantly changing, linkages between variables are complex and poorly
understood, and the variables are often uncertain or unknown. Innovators
must therefore not only move between observation, exploration, and
experimentation; they must also iterate between experiments.

Specifically, it is about how firms can learn so that they can better manage
various sources of uncertainty when past experience can be limiting. Such
sources of uncertainty include those with respect to the R&D process (does the
product work as intended?), production (can it be effectively manufactured?),
customer needs (does it address actual needs?), and the business itself (does
the opportunity justify the investment in resources?). Only by using
experimentation to manage those types of uncertainty can companies unlock
their capacity for innovation. Indeed, experimentation is inextricably connected
to innovation, and managers need to understand that fundamental link.
Simply put, there can be no innovation without experimentation. Or, in other
words, no product or service can be a product or service without first having
been an idea that was subsequently shaped through experimentation.
And that is where companies like 3M make fundamental mistakes when they
try to apply a technique like Six Sigma to innovation. In short, companies
cannot treat R&D like manufacturing because the two processes are inherently
different. In manufacturing, the tasks are mostly repetitive and the activities
are reasonably predictable. Not so in R&D, where the tasks are often unique
and the activities are difficult to anticipate because of changing project
requirements or discoveries along the way. Such fundamental differences have
profound implications. In repetitive processes like manufacturing and
transaction processing, the goal is to minimize any system slack in order to
achieve peak efficiency, and the relationship between added work and required
time is straightforward: Add 5% more work, and it may take 5% more time to
complete. Things are not so simple with processes that have high variability,
such as R&D. The amount of time that projects spend on hold, waiting to be
worked on, rises sharply as the system slack decreases.
6
Add 5% more work in
R&D, and it might take 100% longer to complete the project. And if R&D work
involves experiments, long delays will inhibit feedback and learning. The
bottom line is that when R&D employees are working near full tilt, project
speed, efficiency, and output quality (namely, innovation) will inevitably suffer.
In other words, no product or service can be a product or service without
first having been an idea that was subsequently shaped through
experimentation.
Learning to Embrace Failures
Another difficult lesson about experimentation (and innovation) is that, to be
successful, companies need to be willing to fail. Only through such failures can
they discard bad ideas, such as a confusing software interface, a packaging
concept that will lead to considerable waste, a drug with dangerous side effects,
and so on. And eliminating what does not work will then free people to pursue
other solutions. But many companies instead have a get it right the first time
mentality, which discourages people from pursuing breakthrough ideas. The
result: incremental product improvements rather than innovative offerings.
That said, companies have to be smart about how they fail. For one thing,
failing early on when an idea is far upstream so that it can be scrapped
without wasting considerable time and resources is far preferable to failing
late in the process. The lesson here is that early failures can lead to more
powerful successes faster. IDEO, an innovative product development firm, even
has a motto for that mindset: Fail often to succeed sooner.
Companies, however, are not typically set up to embrace failure. Our
experience has been that most big institutions have forgotten how to test and
learn. They seem to prefer analysis and debate to trying something out, and
they are paralyzed by fear of failure, however small, argued T. Peters and R.
Waterman in the bestseller In Search of Excellence.
7
Peters and Waterman
wrote those words about 30 years ago, but their keen observation could hardly
be truer today.
Of course, not every business has an acute fear of failure. Some, in fact, have a
healthy acceptance that failure is just part of the process. Google, for instance,
runs myriad experiments to continually improve its search algorithm. In 2010
alone, the company investigated more than 13,000 proposed changes, of which
around 8,200 were tested in side-by-side comparisons that were evaluated by
raters. Of those, 2,800 were further evaluated by a tiny fraction of the live
traffic in a sandbox area of the website. Analysts prepared an independent
report of those results, which were then evaluated by a committee. That
process led to 516 improvements that were made to the search algorithm from
the initial 13,000 proposals. In other words, Googles failure rate is higher than
95%.
8
This is a company where its absolutely okay to try something thats
very hard, have it not be successful, and take the learning from that, contends
Eric Schmidt, former CEO of Google.
9

The Value of Randomized Field Trials
When conducting experiments with customers like Google does, companies
have a powerful tool at their disposal: the randomized field trial. These tests
have been invaluable in medicine, helping researchers determine whether a
particular treatment is effective or not. The basic concept is simple. Take a
large population of individuals with the same affliction and randomly select two
groups. Administer the treatment to just one group and closely monitor
everyones health. If the treated (or test) group does statistically better than the
untreated (or control) group, then the therapy is deemed to be effective.
Similarly, randomized field trials can help companies determine whether
specific changes (such as a new layout for a chain of retail stores) will lead to
improved performance (a significant bump in sales). Consider, for instance,
Capital One, the financial services company. From its inception in 1988, the
company has based its very existence on the use of randomized field tests to
investigate potential innovations, no matter how seemingly trivial. Capital One
has used controlled experimentation to test just about everything, including
new product and service offerings, operational changes, marketing campaigns,
and so on. The company might, for instance, test the color of the envelopes
that product offers are mailed in by sending out two batches (one in the test
color and the other in white) to determine any differences in the responses.
10

Randomized field trials can help companies determine whether specific
changes will lead to improved performance.
Randomized field tests are indeed a powerful experimental tool, but they are
not without their challenges. For the results to be valid, the field trials must be
conducted in a statistically rigorous fashion. Specifically, people need to be
assigned to either the test or control group through a selection process that is
purely random to help ensure that the two groups dont differ in any pertinent
way other than with respect to the independent variable being studied.
Otherwise, other variables could easily skew the results. In the Capital One
envelope example, if a larger percentage of men had been assigned to the test
group, a lower response for that group might have nothing to do with envelope
color; it might simply mean that men are less apt to respond to such offers.
Moreover, both the test and control groups must be representative of the larger
customer base. If, for instance, both the test and control groups in the envelope
test had contained a much higher percentage of women, then Capital One
wouldnt know for sure whether the results were applicable to men. Thus
randomization plays an important role in experimentation: it helps to prevent
systematic bias, assigned consciously or unconsciously, and evenly spreads
any remaining (and possibly unknown) bias between test and control groups.
Such caveats notwithstanding, randomized field trials have become standard
practice in direct marketing, and they have even begun to spread to unlikely
areas like the gaming industry. Caesars Entertainment, which operates
Harrahs, Caesars, and other casino resorts, regularly uses controlled
experiments to develop and fine-tune its various marketing efforts. The
company might, for instance, test which perk a complementary meal versus a
free night of lodging would ultimately induce customers to spend more during
their stays. Gary Loveman, the CEO of Caesars Entertainment, has famously
stated that there are three ways to get fired from the hotel and casino
company: theft, sexual harassment, or running an experiment without a
control group.
11

Going Against Instincts
Given how valuable experimentation can be, the question must be asked: Why
dont companies experiment more? Certainly the drive toward increased
efficiency has been an issue, but another factor might also be at play. Consider
how senior management often has strong incentives to focus on the near term
and get rewarded for sticking to plans. But innovation activities can be highly
variable and difficult to plan and predict, especially over short timeframes. Dan
Ariely, the noted behavioral economist, contends that businesses often shy
away from experimentation because they are not good at tolerating short-term
losses in order to achieve long-term gains. Companies (and people) are
notoriously bad at making those trade-offs, he argues.
12
And, as mentioned
earlier, such business myopia becomes all the more acute in bad times, when
market conditions force many companies to tighten their belts. But not all
businesses have fallen into that trap.
Take, for example, ams, the Austrian-based manufacturer of analog
semiconductors. Employing around 1,200 people in more than 20 countries,
ams develops and manufactures sensors, wireless chips, and other high-
performance products for customers in the consumer, industrial, medical,
mobile communications, and automotive markets. Typical applications require
extreme precision, accuracy, dynamic range, sensitivity, and ultra-low power
consumption.
To maintain its technical edge, ams implemented a major initiative for business
experimentation in January 2007. Throughout the company, all employees
were encouraged to run experiments and submit them to a central coordinator.
These activities have distinct learning objectives, and they do not include
regular tasks such as feasibility studies and normal project work. Of the
proposed experiments, the company approves about two-thirds, but their costs
are not measured or accounted for in timesheets or work statements. The point
is that management does not oversee the experiments: employees come up with
the ideas, design the tests, and run them all in addition to their normal
responsibilities.
To document those activities, the company publishes annual proceedings of the
experiments. As of November 2012, ams had documented 369 completed tests,
of which more than 80% were technical in nature, about 10% were
organizational, and the remainder related to marketing and sales. Bonuses
have been awarded to the best experiments, with success measured by learning
objectives or outcomes. So far, those bonuses have reached a total of 124,000
euros. In addition, ams has also run company-wide experiments such as a
24h Day event during which employees dropped all their regular duties and
spent 24 hours nonstop to work on their own ideas.
The point is that management does not oversee the experiments:
employees come up with the ideas, design the tests, and run them all in
addition to their normal responsibilities.
It should be noted that many of those experiments have become the starting
points of new projects, product improvements, patents, and new product
proposals. As such, they help ensure that ams will have a healthy number of
offerings in the pipeline for when the economy recovers. In other words, while
other businesses were cutting back their innovation activities, ams not only
stayed the course; it upped the ante by launching a major initiative,
successfully managed the delicate balance between efficiency and building a
culture of experimentation, and empowered its employees to try out new
things. And, as a result, the company should be ready for any market upswing,
whereas other businesses could easily be caught off-guard.
According to various accounts, companies have been hoarding cash. In mid-
2012, for example, corporations in the S&P 500 had stockpiled about $900
billion.
13
Certainly, such financial liquidity has its advantages, and investing
recklessly on ill-advised initiatives should never be encouraged. But, when it
comes to innovation, being too frugal can also have its drawbacks, particularly
if the result is that a companys pipeline of new products and services begins to
dry up. And thats the danger facing extremely efficient businesses that value
standardization, optimization, and low variability: They leave themselves
vulnerable to underinvesting in experimentation and variation.
That lesson was something that 3M learned the hard way. After CEO McNerney
left the firm, the new CEO George Buckley began to undo some of his
predecessors actions. He increased the R&D budget substantially and freed
research scientists from the grips of Six Sigma. Invention is by its very nature
a disorderly process, explained Buckley. You cant put a Six Sigma process
into that area and say, well, Im getting behind on invention, so Im going to
schedule myself for three good ideas on Wednesday and two on Friday. Thats
not how creativity works.
14
Buckleys wise words capture, in a nutshell, why
innovation (and experimentation) will never be a process that is entirely
predictable nor highly efficient. Other executives and companies would do well
to remember that simple managerial truism.