In my work with you as an entrepreneur or small business owner, I find that most understand the need try out new business models and product innovations, but trust their intuition rather than running a disciplined business experiment first. The result is that you often take a big financial hit, and lose key customers, before you realize that the change didn’t work. There is a better way.
For example, several years ago Apple was so convinced that their new Apple III model would be a winner that they rolled it out broadly, rather than do a controlled release first with some key customers. The result was a huge negative quality hit for Apple in the marketplace that could have been avoided, and high repair costs that took years to recover.
I just finished a new book, “Experimentation Works,” by Stefan H. Thomke, which shows how to make experiments pay off, describes best practices, and provides some great examples of the process done right. He also debunks the most common myths I have heard about business experimentation, including the following:
Experimentation-driven innovation will kill intuition. Before you can experiment, you still need intuition and creativity to come up with innovations to test. Thus experiments are complementary to intuition, not mutually exclusive. Unfortunately, even expert intuition has been proven unreliable in predicting customer behavior, so we need both.
Experiments will lead to incremental, but no big changes. Incremental change has proven to be less risky, and multiple small change cycles done quickly often lead to a breakthrough. Even big changes, like a new business model, can have few elements and be validated by an experiment. Changing too many factors at once is never a smart risk.
We do so few changes, we don’t need formal testing tools. The pace of change to stay competitive in business today continues to increase. Every organization needs to look ahead to keep up with Amazon and Booking.com, who are rolling out hundreds or thousands of changes each year. Now is the time to update your processes and tools.
Brick-and-mortar companies don’t have enough transactions. While online retailers may define a sample as fifty thousand customers, a realistic test group for a physical store may be dozens or less. Yet a large sample does not necessarily lead to better data. In addition, even a small sample size shows statistically valid results for large changes.
A/B testing has been used often, with only modest results. This type of experiment has been around a long time, and many executives have an unreasonable expectation that the cumulative business impact should be at least the sum of the results. In reality, better techniques are now available, so don’t skip experimentation based on old news.
Big data will show results quickly, so experiments waste time. Analytics show results, while experiments isolate the cause. Thus these are actually complementary, and understanding causality is necessary for effective follow-on enhancements. Using intuition to deduce the cause is unreliable, and even dangerous in the case of medicine.
Experiments on customers without consent is unethical. Of course, every new offering to customers must be lawful and delivered ethically, and every business model and product is essentially an experiment in providing maximum customer value. A bigger ethical risk is not to experiment and forgo a change that’s critical to increasing that value.
Tricking customers or persuading them to do things that go against this value objective make no sense, and simply won’t work in the long run. Thus it makes sense to be as transparent as possible, and to do your testing with as much speed as technology allows.
To counter these myths, there is more and more evidence of the value of building a business experimentation culture in your company, compared to relying on intuition or flying blind. The tools are out there to collect data in real time, and provide the analysis that you could never do manually to show what innovations are working with customers, and which are not contributing.
Looking ahead, as customers get more demanding, and competition continues to expand worldwide, the ability to iterate more quickly and effectively will be key to your survival and growth. It’s time to take a hard look at how you and your organization make change decisions today, to make sure you are ready for the changes your customers expect from you tomorrow.
