Platform
Resources
A/B testing is essential for data-driven teams, but its real value isn’t just in the metrics—it’s in the ultimately qualitative decisions that those metrics enable.
Understanding the bigger picture
Real-world decision-making doesn’t happen in a statistical vacuum. People with diverse perspectives have to debate options in a room, weighing pros and cons between multiple (sometimes conflicting) objectives, many of which can't be captured in a single metric.
For data scientists, this means you have to look beyond p-values or effect sizes. Yes, it's true, you do need to ensure statistically sound results—but you also need to ask yourself, “What else is going on here?” Whether if it’s brand considerations, user trust, or alignment with broader business goals, qualitative dimensions are just as important to consider alongside your A/B testing results.
The limitations of data and the role of expert judgment
Tom Cunningham famously outlined two challenges in data science: inference and extrapolation. For inference, it's straightforward to see whether a test variant outperformed the control under certain conditions. However, extrapolating these insights to broader questions (i.e., different contexts or population shifts) often involves some degree of qualitative reasoning. Even the best quantitative models can’t account for every nuance. No test measures everything.
It’s also important to remember that organizations innovated long before modern experimental methods were a thing. As Tom stated, “People built cathedrals, designed airplanes, and launched globally successful apps with little more than domain expertise, instinct, and common sense”. This isn't to say that experiments and rigor are not important; rather, it means that qualitative judgment—nurtured through hands-on experience—is still vital.
A balanced approach: Numbers and narrative
So, what’s the best approach to take? As data scientists, we often do our most compelling work when we blend quantitative rigor with qualitative wisdom:
Start with a purpose: Don’t run experiments without a well-considered hypothesis. Don't just think about what metrics will move, think about why they might move.
Provide clear data, but emphasize context: Present your results plainly first in terms of quantitative measures like effect sizes, confidence intervals, and statistical significance. Then, offer qualitative nuance—the why, potential pitfalls, and whether the effect aligns with broader business goals.
Invite broader feedback: Listen to stakeholders who might have non-quantitative insights. They may spot factors that algorithms or dashboards might miss.
Frame decisions as trade-offs: A/B tests often reveal gains in one metric at the expense of another. Bring in qualitative judgments about which trade-offs matter most.
The end goal
At the end of the day, experimentation isn’t an academic exercise. It’s about informing decisions that drive real outcomes. In a complex world, data alone can’t reveal every detail. Ultimately, the strongest decisions weave together robust data with thoughtful, experience-driven interpretation.
Data scientists who learn to speak both languages—quantitative and qualitative—become invaluable partners in any experiment-driven culture. It’s not just about what you can measure, but how those measurements fit into the bigger picture of organizational decision-making.
Talk to the pros, become a pro
