Steps in Organizational Multi-Factor Experimentation

First, what organizational objective are you trying to achieve (or problem you’re trying to solve)? If it is an objective or problem where success is measurable/quantifiable, that is a good start. If such a measurement of success does not currently exist, an expert may be able to assist in setting one up.

Second, what type of process should you consider for experimentation? The best candidates are processes that 1) have fairly quick repetition, and 2) are expected to remain in place long enough for the implementation to have an adequate payoff after the experiment is conducted. The first of these is to assure the experiment can be carried out in a reasonable time frame. If the experimental design required 24 process runs and there was only one process run per month, the experiment would nominally take two years which is outside the planning timeframe of most organizations. In cases like this, an innovative experimenter may fine a different approach to get the experiment conducted in a timely way, but an experiment that takes more than 6-8 weeks is probably too long for many organizations to consider. The second requirement is so that positive results of the experiment can be in place long enough to make a substantial difference to the company. If your experiment ends in May and the results are implemented in July but the process sunsets in November, you’ll only get five months of value from the experiment, at most. It is better to choose a process that is expected to be in place much longer. In addition to these two requirements one should have measurement processes to measure the impact on the primary objectives with low measurement variability. The process should also have sufficient data so that the sensitivity of the experiment is able to detect meaningful changes.

Conducting an experiment to improve business results is best done when part of an overall effort to improve the business process, but could be used as a special initiative. Using a baseball analogy, most process improvement efforts involve hitting many singles with continual improvement as the objective with an occasional or rare home run along the way. A well-run experiment could be one of those home runs and some would even be considered grand slams.

Typical steps in a successful business experiment are

  • Clear business objective and an environment for success

Clear business objective

__ This should be a well-defined goal with high value to the organization.

__ Additional secondary or tertiary objectives are encouraged.

Environment for success

__ A process owner with sufficient clout should be champion of the improvement effort with a project manager (or equivalent) running the day-to-day activities. If the experiment is complex and crosses organizational boundaries a cross-functional team may be needed to coordinate the activities.

  • Valid measurements

You need the ability to measure process results directly related to the business objective, which could be a Key Performance Indicator (KPI) or other results metric. You may have more than one result metric of interest (e.g. profit, revenue and customer satisfaction). Every effort should be made to make sure the measurements are timely, consistent, and repeatable, with low variation. Measurement studies should always be done. Most measurement systems are found to be inadequate at the beginning of one of these experiments.

  • Some level of process stability

This is to ensure the effects of the ideas being tested can be seen above the noise of the process. If the process is not stable across time (as is the case with most business and online experiments) you may be able to gain stability by conducting all the experiment variations at the same time. (If that last sentence does not mean a lot to you, don’t worry. It just means there may be more than one way to gain the stability needed for your experiment, you just may need to get creative or call on an experimentation expert.)

  • Brainstorm ideas to test

Get ideas from many people, focusing on the experimental objective(s). All primary functions in the process should be represented, especially individuals who are closest to the day-to-day operation of the process.

  • Determine which ideas would be tested

Do not choose the ideas that you think will work the best. Our ability to predict which ideas will work and which will not has been shown to be almost the same as flipping a coin. Instead use criteria such as ideas that are easy to test, inexpensive to test and can be ready to be tested quickly.

  • Screening test to determine which ideas are best candidates for improvement

Since most experiments will have many ideas being tested, the first experiment is to determine which ideas are the most promising. Organizations that have conducted experiments consistently find that 25% of the ideas tested or less have the potential to improve results. So if you have 16 ideas in your screening experiment you may have 4 (more or less) ideas that make it to the next step. It is extremely important that sufficient control is maintained throughout the experiment so that the different treatment combinations are being carried out as planned.

  • Refining test to validate winners and optimize results

Many organizations consider this step optional, but best practice is to run a second experiment with the promising ideas from the screening experiment. The objectives of the refining experiment are to validate these ideas, get an improved estimate of the actual amount of improvement you can expect from them and to get the optimal combination. (Technically, a screening experiment does not give you any information about interactions between the ideas in that experiment so the optimal combination can only be determined from the results of the refining experiment.)

  • Implement and monitor improvement

Only the ideas from the screening phase that are validated should be implemented. You should be prepared to monitor a) how well the ideas are implemented and b) the impact the improvements have on the KPIs. A control chart is a good tool for the latter. In some cases, you may want to run a follow-up experiment to test ideas that couldn’t be tested in the first experiment or that were conceived during the first test. Also, some ideas may have additional settings that could be tested for further improvement. For example, if you found that offering the customer a one month extension on the annual license yielded 15% improvement in renewals, in your next experiment you may want to test whether offering a 2 or 3 month extension would have a sufficient ROI.


Follow My Blog

Get new content delivered directly to your inbox.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s