In 1835, at a tavern in Bavaria, a group of 120 people once met to drink from a randomized assortment of glass vials.
Before shuffling them, they divided the vials into two sets. One contained distilled water from a recent snowfall and the other a solution made by collecting 100 drops of that water and dropping into the pool a grain of salt, and then diluting a drop of the result into another 100 drops, again and again, 30 times in all.
They did this to test out a new idea in medicine called homeopathy, but it was the way they did it that changed things forever. By testing options A and B at the same time, but without telling sick people which option they would be getting, they not only debunked a questionable medical practice, they invented modern science and medicine.
About 200 years later a company in California tried something similar. A group of 700,000 people gathered inside a virtual tavern to share news and photos and stories both happy and sad. The company then used some trickery so that some people randomly encountered more happy things and others more sad things.
They did this to test out a new idea in networking called emotional contagion, but it was the way they did it that changed how many people felt about gathering online. By testing options A and B at the same time, but without telling people which option they would be getting, they not only learned if a computer program could make its users more happy or more sad, they created a backlash that resulted in a large-scale, world-wide panic.
Though we always learn something new when we perform an A/B test, we don’t always support the pursuit of that knowledge, which is strange, because without A/B testing we have to live with whatever option the world delivers to us, be it through chance or design. Should we use cancer drug A or B? Should we try gun control policy A or B? Should we try education technique A or B? It seems like our reaction to these questions would be to support testing A on half the people, B on the other, and then to look at which one works best and go with that moving forward, but as you will learn in this episode of the You Are Not So Smart Podcast, new research shows that a significant portion of the public does not feel this way, enough to cause doctors and lawmakers and educators to avoid A/B testing altogether.