Elements that are Conducive to Good Experimental Results

It is not necessary to have all of these present for a good or valid organizational experiment, but the more that are satisfied, the more likely it is you will have a valid, impactful experimental results.

  1. Clear quantifiable outcome metrics that are directly related to business objectives.
  2. A sufficient number of potential/eligible “experimental units”. This depends on the test, but 12 is a bare minimum for almost any test and many tests may require 30 or more. (The outcomes for the experiment can be measured independently for the different experimental units and the treatments can also be applied independently to the units.) The number of experimental units needed is related to number 5 below. The larger the variation in the primary metric, the more experimental units are required.
  3. Relatively stable environment, or can be made so through appropriate controls, test design or transformation. For example,
    • In online tests, this is achieved by requiring all treatment groups be collecting data at the same time (concurrent) and the percentage in each group does not change throughout the test.
    • For retail experiments, this would involve choosing stores that are in a stable time period and using year-over-year ratios for outcomes.
  4. Controllable test variables/ideas in order to optimize outcomes.
  5.  Either a large amount of data or small variation in order to separate the signal from the noise. (With online tests the variation relative to the change we want to see is usually so large that we need a large number of users in these experiments.)
  6. Results of actions that can be seen in relatively short period of time (days or weeks, not months).
  7. Results have sufficient business merit to justify the expense of running the experiment.

Follow My Blog

Get new content delivered directly to your inbox.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s