Another Use for Excluded Test Groups

In my last post, I touched on how you can use excluded groups in your A/B tests to gradually ramp up a test while maintaining the integrity of your statistics. There are a few other uses for this concept of excluded groups.

At a basic level, an excluded group lets you separate the two concerns that underlie a basic A/B test: * what a user sees * what you are measuring

Many organizations treat the two issues as one, which can be very restrictive if you want to maintain accurate measurement. (Or more likely, your results will be invalid because you weren’t aware of the restrictions.)

But by using an excluded group you can play fast and loose with what a user sees without ever impacting your measurement plans.

 Continue Reading    

A Common Issue with A/B Tests and How to Fix It

Dr. Jason Davis has a new post up with eight common ways to screw up an A/B test. He uses the polite term “misconfigured” but the end result is the same – your test results will be invalid. It’s a solid list that I recommend reading, but I wanted to expand on a simple solution for one of the issues he describes.

 Continue Reading    

Three Easy Tools for Data Exploration

At a really simplified level, an analyst is responsible for taking a pile of data and come back with a handful of insights to report. Data in, insights out. If you’re not familiar with the process, it may be easy to assume they knew from the beginning those few key numbers that they wanted to report, and simply ran some calculations to get them.

But it’s not always that easy. Honestly, it never is. When you’re dealing with real world data sets, you don’t know what’s interesting until you’re knee deep in the data. To get there, you have to just start exploring.

 Continue Reading    

The Science of Empathy

I’m a big believer that understanding pyschology, especially behavioral psychology and social psychology, is a crucial aspect of product strategy. I gave this lightning talk a few years back at Ignite DC about the science of empathy and what it means for product development.

 Continue Reading