It's not every day you get to sit down with a seasoned experimentation leader.

Last week, I hosted Kevin Anderson (Product Manager, Experimentation at Vista) for a candid fireside chat filled with interesting anecdotes and tips for building an experimentation-driven product culture. You can view the full recording below:

Having been in the domain for nearly two decades and currently working as a Product Manager, liaising with internal and external stakeholders, and running a weekly newsletter (Experimental Mind), Kevin has a deep understanding of how the space is evolving. Below is a teaser of five interesting topics that were discussed in our chat:

1. Latest trends in experimentation

Kevin noted how experimentation has evolved from focusing solely on conversion rate optimization on websites to influencing the entire product development culture. This shift has been facilitated as vendors have entered the space, aiming to democratize big-tech tooling, which has made it easier to adopt experimentation.

2. Reducing friction in the experimentation process

As a PM, Kevin focuses on identifying and eliminating points of friction to facilitate more high-quality experiments across the organization.

He shared various tactical approaches, such as maintaining a public Slack channel where anyone can post questions about experimentation and consistently speaking with stakeholders to gather insights on their hesitations towards experimenting.

What stood out to me was Vista’s commitment to transparency in the experimentation process — everyone has access to the Statsig project and can view data — so everyone benefits from the learnings.

3. Measuring the success and progress of the experimentation program

Kevin highlighted the importance of measuring the success of experimentation initiatives—a priority he set when he joined the company. He shared both qualitative and quantitative ways to measure progress but emphasized focusing on quantity initially—i.e., running more experiments. He argued that you can't improve quality unless you're already running numerous experiments.

4. The build vs buy debate

Kevin noted that companies often underestimate the effort required to build feature flagging and experimentation in-house. While it may be feasible to develop a V1 to handle specific use cases, advancing to the next steps requires significant investment and resources—often limiting progress.

These days, the baseline feature set available from third-party vendors is very high. Mature companies with a robust in-house platform, like Microsoft or Booking, started their development 10-20 years ago to reach their current status. Kevin's advice was to focus on acquiring the best solutions available in the market to accelerate the adoption of experimentation—a strategy he embraced at Vista when deciding to partner with Statsig.

5. Getting C-suite buy-in for experimentation

Kevin also shared some hot takes, such as pushing back against the common belief that you can't do experimentation without C-suite support. While it is easier when it comes from the top, it is certainly not a requirement.

Kevin's approach was to identify champions across cross-functional teams, even without an explicit mandate from the top, and achieved significant results in previous roles. Don't wait for a CXO to enforce the culture; just find the right allies (even from other teams in the company) to start with, achieve small wins, and kick off the experimentation flywheel!

These were my favorite highlights, but I'd highly recommend listening to the full conversation.

Get a free account

Get a free Statsig account today, and ping us if you have questions. No credit card required, of course.

Create my account