Publications

A list of publications by members of Process Performance Management. There are three categories of publications: A/B Experimentation, Multi-Factor Experimentation, and Machine Learning and Statistics. A link to a pdf to some of the articles is provided with the initial description. Some also have external links to the publications.

A/B Online Experimentation

  1. Online Controlled Experiments. An article outlining the benefits and methodology for A/B online experiments. Citation: Ron Kohavi, Roger Longbotham. Online Controlled Experiments and A/B Tests. Encyclopedia of Machine Learning and Data Mining. April, 2017 (Link)
  2. Rules of Thumb for A/B Experimenters. Seven rules we developed to guide planning, running or interpretations for A/B online experiments. Citation: Ron Kohavi, Alex Deng, Roger Longbotham, Ya Xu. Seven Rules of Thumb for Web Site Experimenters. KDD 2014 Proceedings. (Link)
  3. Trustworthy Online Controlled Experiments. How to run A/B online experiments so that you can get results you can trust. Includes six examples of things that can go wrong to give incorrect or biased results. Citation: Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu. Trustworthy Online Controlled Experiments: Six Puzzling Outcomes Explained. KDD 2012 Proceedings. (Link)
  4. Website Monitoring and Improvement. Methodology for monitoring stability and improving websites including use of A/B experiments. Citation: Roger Longbotham, Ji Chen, Dave DeBarr, Shaojie Deng, Justin Wang. Website Monitoring and Improvement. JSM 2011 Proceedings. Section on Quality and Productivity. Alexandria, VA: American Statistical Association. pp. 3508-3517. (Link)
  5. Choice of the Randomization Unit in Online Controlled Experiments. The two most common randomization units used in A/B experiments are user and page view. We compare these for power and analysis considerations. Citation: Shaojie Deng, Roger Longbotham, Toby Walker, Ya Xu. Choice of the Randomization Unit in Online Controlled Experiments. JSM 2011 Proceedings
  6. Quality for Online Experimentation. Specific issues that must be addressed in order to have online A/B experiments you can trust. Citation: Ji Chen, Roger Longbotham, Dave DeBarr, Shaojie Deng, Justin Wang. Quality for Online Experimentation. JSM 2011 Proceedings. Section on Quality and Productivity. Alexandria, VA: American Statistical Association. pp. 4130-4138. (External Link)
  7. Online Experiments: Practical Lessons. Additional lessons learned in A/B experimentation at Microsoft. Citation: Ron Kohavi, Roger Longbotham and Toby Walker. Online Experiments: Practical Lessons. IEEE Computer. September, 2010. 43:82-85.
  8. Unexpected Results in Online Controlled Experiments. In this paper we give ten examples where subtle differences between Treatment and Control give significant changes to the user experience. Citation: Kohavi, Ron and Longbotham, Roger. Unexpected Results in Online Controlled Experiments. SIGKDD Explorations. 2010, Vol. 12, 31-35.
  9. Online Experimentation: Principles and Practice. Demonstrates some of the specific techniques and nuances in conducting online (A/B) experiments for an audience of statisticians. Citation: Roger Longbotham, Ji Chen, and Justin Wang. Online Experimentation: Principles and Practice. JSM 2010 Proceedings, Section on Statistics and Marketing. Alexandria, VA: American Statistical Association. 5371-5380.
  10. Online Experimentation at Microsoft. A workshop to KDD 2009 audience on how we were running A/B experiments at Microsoft. Citation: Ronny Kohavi, Thomas Crook, and Roger Longbotham, KDD 2009 Data Mining Case Studies Workshop, 3rd place winner: Online Experimentation at Microsoft. June 2009.
  11. Planning, Running, and Analyzing Controlled Experiments on the Web. A three hour tutorial presented at KDD 2009 on how to run A/B experiments. (Video link provided.) Citation: Ronny Kohavi and Roger Longbotham, KDD 2009 tutorial: Planning, Running, and Analyzing Controlled Experiments on the Web. June 2009. (Link to online video. Links to slides: Part 1, Part 2, Part 3.)
  12. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. This article explains seven common problems one may encounter when conducting A/B experiments and how to avoid them. Citation: Thomas Crook, Brian Frasca, Ronny Kohavi and Roger Longbotham. KDD 2009 paper: Seven Pitfalls to Avoid when Running Controlled Experiments on the Web April 2009. Proceedings.
  13. Controlled experiments on the web: survey and practical guide. An often-cited paper on how to conduct A/B online experiments along with guidance to make them trustworthy and successful. Citation: Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal M. Henne, Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. 2009. 18:140–181.
  14. Online Experimentation at Microsoft (Long version). Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, Tamir Melamed. Online Experimentation at Microsoft. Internal Microsoft ThinkWeek publication (sanitized for public view) 2009.
  15. Navigating the Depths of Multivariable Testing. An introduction to how Multi-Factor experiments should be run on internet retail sites with examples. Citation: Gordon H. Bell and Roger Longbotham, Navigating the Depths of Multivariable Testing, Internet Retailer, February 2007.
  16. Online Experiments: Lessons Learned. Some of the lessons we had learned in the early days of A/B experimentation at Microsoft. Citation: Ron Kohavi and Roger Longbotham, Online Experiments: Lessons Learned, IEEE Computer, Sept 2007.

Multi-Factor Experimentation

  1. Using Multi-Factor Experimentation for Business Success. An internal Microsoft publication outlining a number of scenarios where Multi-Factor experiments could improve results at Microsoft. Citation: Roger Longbotham. Harnessing the Scientific Method for Business Success. ThinkWeek 2011. (sanitized for public view)
  2. Navigating the Depths of Multivariable Testing. An introduction to how Multi-Factor experiments should be run on internet retail sites with examples. Citation: Gordon H. Bell and Roger Longbotham, Navigating the Depths of Multivariable Testing, Internet Retailer, February 2007.
  3. A Scientific Approach to Implementing Change. A case study of how Multi-Factor experimentation could be used to test ideas that may be effective in implementing a behavioral change in a large number of people in an organization. Citation: Gail J. Longbotham and C. Roger Longbotham, A Scientific Approach to Implementing Change, Journal of Practical Consulting, 2006, Vol. 1, Issue 1, 19-24.
  4. Characterizing Prediction Bias From a Designed Experiment. This article demonstrates how much bias one may expect when making predictions from a Multi-Factor experiment. Citation: C. Roger Longbotham, Characterizing Prediction Bias From a Designed Experiment. 25th International Symposium on Forecasting. 2005.
  5. A Problem Shared. A case study of how Multi-Factor experimentation was used to solve a difficult problem in the Process Manufacturing Industry. Citation: C. Roger Longbotham, A Problem Shared. Chemistry in Britain. 35 (5). May 1999, 35-37.
  6. The Role of Multi-Factor Experimentation in Process Improvement. Illustrates how and when Multi-Factor experiments should be used in process improvement. Citation: C. Roger Longbotham, The Role of Multivariable Testing Techniques in Process Improvement. Proceedings of the MD&M West 97 Conference. 202-41:202-44.

Machine Learning and Statistics

  1. Machine Learning Features for Determining Article Use in English. The goal is to devise a system that can analyze a sentence and determine whether a noun phrase should have an article (a, an, the) in front of it. The question of whether to use an article is not exactly easy, especially for non-native speakers and machine translation systems. The system I’ve developed uses lexical and grammatical features (like part of speech) to determine when an article is needed, and what kind. Citation: James Longbotham, Machine Learning Features for Determining Article Use in English. 2017. ISBN-13: 978-1973226666 Paperback (Kindle available). 66 pages.
  2. Seven Management and Planning Tools. An article introducing a collection of seven tools for management planning and decision-making. Citation: C. Roger Longbotham, Seven Management and Planning Tools. Proceedings of the Regional Forum of the Council for Continuous Improvement. 1994
  3. Transformations of the Chi-Square and F Distributions for Control Charting. An article showing how one could transform distributions related to the sample variance for improved process monitoring. Citation: C. Roger Longbotham, Transformations of the Chi-Square and F Distributions for Control Charting. Technical Report. Rockwell International. 1987
  4. Nonparametric Density Estimation. One of the earliest implementations of the recent theory of how to estimate a sample distribution and display it graphically using SAS software. Citation: C. Roger Longbotham, Nonparametric Density Estimation. SAS Conference Proceedings: SAS Users Group International 12 (SUGI 12)1987. 907-909
  5. Detection of Trends, Outliers, and Releases in Environmental Monitoring. Graphical and statistical modeling were used to describe and analyze geographic spread of a pollutant. Citation: C. Roger Longbotham and B. B. Lawton, Detection of Trends, Outliers, and Releases in Environmental Monitoring for Plutonium at Rocky Flats Plant. Technical Report. Rockwell International, 1986. Also presented at the 1987 JASA conference
  6. Measuring Inequality Between Income Distributions. In this article we gave a distribution-free method for measuring the difference in two distributions. Citation: Robert C. Hannum and C. Roger Longbotham, Measuring Inequality Between Income Distributions. Proceedings of the Second International Conference on Quantity and Quality in Economic Research. 1985. 55-72

Other Publications

  1. English for Non-Native Writers. This book is a reference with special focus on the needs of authors who are not native speakers. For someone who needs a reliable handbook that makes the most important rules and common errors accessible in a clear and concise manner. It also contains linguistic rules for everyday editing that integrate well into existing editing guides. Citation: Erdmann, Fleury, Nickl, Johnson Coenen, Link, Longbotham, Reuther, Schöffer, Siegel. Englisch für deutschsprachige Autoren (2nd Edition). 2017 Paperback. 100 pages.
  2. R-Pronouns and Preposition Stranding in the History of German. German has a series of words that consist of da, hier or wo + preposition. These so-called R-pronouns are often found in topicalized constructions at the beginning of a sentence, and in some dialects the two parts can be split. For example “davon weiß ich nichts” is sometimes “da weiß ich nichts von.” The split and non-split constructions can also be found in many Middle High German texts. On the basis of various Middle High German sources, this study investigates the syntactic, semantic, and stylistic factors behind the preference for a split or non-split construction. Citation: James Longbotham. R-Pronouns and Preposition Stranding in the History of German. 2010. ASIN: B00ERZ1FZI. Paperback. 119 pages.
  3. Seven Management and Planning Tools. An article introducing a collection of seven tools for management planning and decision-making. Citation: C. Roger Longbotham, Seven Management and Planning Tools. Proceedings of the Regional Forum of the Council for Continuous Improvement. 1994

Follow My Blog

Get new content delivered directly to your inbox.