A cord cutter’s best friend, before it was cool

Surprise, suspense, and excitement—the “I can’t wait to tell my friend about this” feeling: If you’re reading this, you might feel these emotions reviewing the results of an A/B test. And more commonly, following the discovery of a new TV show or the shock of a surprise twist in a movie.

Founded in 2008, Plex’s original vision remains unchanged: connecting real humans through a simple, one-stop shop digital entertainment platform for millions of users across the globe. Before the trend of canceling traditional cable television and streaming services like Netflix really took off, Plex customers were enjoying the ability to remotely stream their personal movie, TV show and music collections with Plex’s Personal Media Server offering.

With the proliferation of streaming services over the past decade, Plex customers continue to enjoy the same beautifully-organized selection of entertainment regardless of how many subscriptions they have thanks to deep integrations and Plex’s continued investment in top-tier tech talent.

“Customers are our priority. Our goal is to serve our customers well. And to achieve that, we really need a solid foundation of technology. We’re very much a user-centric and technology-first company,” says Evelina Achilli, or EV, who is Plex’s product manager for growth.

This investment in technology means Plex customers today enjoy their own personal media library plus hundreds of live channels, tens of thousands of free movies on demand, lossless audio for audiophiles, and the recently launched Discover feature to search for any movie or show across basically every streaming service on the market through a single function. To continue delivering new offerings like Discover, Plex leans even harder into a culture of experimentation. According to EV, “As we focus on building a user-friendly experience, our goal is to continue iterating and innovating.”

“We plan to use Statsig for any new major feature releases. The beauty is that the tool allows us to both run experiments, but also track the impact of feature releases.”

Evelina Achilli

Product Growth Manager

Progress via personalization

When EV joined Plex in March 2022, she wasted no time as a true change agent for growth and experimentation. “We had an in-house experimentation tool that worked but was resource intensive and hard to turn into an efficient process. We had run a few experiments, and the process was limited to a few engineers,” she says.

While Plex was able to measure the success of an experiment’s primary key performance indicator (KPI) with the homegrown solution, it was often challenging to see the impact of additional KPIs and behaviors, falling short of a modern experiment platform like Statsig. “Statsig allows tracking multiple metrics. It also uses the latest statistical techniques to improve the precision of measurement.”

The differentiation and flexibility offered by Experiments and Metrics proved crucial to fuel growth. “We also appreciated the management UI’s ease of use and framework flexibility with lots of options for platform integration and APIs,” she says about the Statsig Console and integrations.

An award-winning supporting role

EV is the self-described “face of Statsig” internally at Plex to align her team’s primary business goal around growth efforts with experimentation at the core of those efforts. Her approach includes evangelizing experimentation to make clear what the capabilities are, aligning on the philosophy of how Plex wants to approach testing, and ultimately helping define the Plex product roadmap.

Her hard work continues to pay off as a growing culture of experimentation is “definitely gaining momentum…. the leadership team is very much behind experimentation.”

“As you start experimentation and changing processes within a company, there are lots of questions. Statsig has been very responsive and able to provide confidence in their method,” she says about the partnership between Plex and Statsig.

Similar to Statsig, Plex is a company of many integrations with the likes of Roku, Apple, Amazon Fire, Chromecast, Xbox, PlayStation, and Samsung, bolstering flexibility to reach more users with a plethora of APIs. Plex initially focused on completing integrations with a team of cross-functional partners and then shifted to running several smaller experiments in the first few months to prove out the value of experimentation with Statsig across the business.

While completing initial experiments, EV took the initiative to build a healthy network of collaborators. “There has been a strong appetite for making data-driven decisions at Plex, but given pressure on resources, it is essential to have partners across the organization. None of this would have happened with just me, it was a true cross-functional collaboration,” she says. “For instance an engineering representative and a PM representative who are as devoted to the process as you are.” At Plex, these devoted collaborators were Engineering Manager, Dai Williams and Product Manager, Jason Williams.

“I actually go to all of those team meetings with everyone who’s involved with experimentation and try to either identify potential testing opportunities or get resources to prioritize a new test,” EV says.

From (almost) zero to a hundred

By understanding the business needs, Plex experimentation with Statsig had an immediate impact. “The UIs are super easy to use, and the framework is really flexible. That helps in our world because we want to be able to test on everything that we’re on, not just in one platform,” EV explained when describing Statsig API integrations.

EV put her “test everything” mentality into practice with 21 tests completed in six months with approximately 40 total experiments in flight well before the end of the year. Core KPIs tested include common business signals like total minutes spent in the Plex app, new user registrations, how often users save media to their watchlist as well as the number of playbacks.

A near 10x increase in experimentation compared to the in-house solution doesn’t mean quality is sacrificed for quantity. EV explains it as, “We can only grow if the results are accurate. Statsig helps us be as accurate as possible”

With this accuracy came early, quantifiable success “We had a very good first test,” EV says. “An 8.2% increase in registrations with our very first test.”

EV attributes this early success to Statsig ensuring experimentation methodology is sound. This confidence frees up Plex to focus on the strategic aspects of building experiments without worrying about complexity. “The fact that we don’t have to think about it and reinvent the wheel is a huge confidence builder.”

“As we searched for a partner to help us with our growth efforts, we needed someone who could provide fast results and insights for us to be able to quickly iterate, which is why we decided to go with Statsig.”

Evelina Achilli

Product Growth Manager

A strategic business partnership on its way to stardom

“I know we’re a customer, but I see [Statsig] as my partners in this. And that’s been so, so great,” EV says when describing the support provided by Statsig and connection with other Statsig customers in the Slack community. “If I have a question, I go and ask and I feel like I always get an answer. The responsiveness has been amazing…it’s just the level of experience that Statsig brings.”

Looking down the roadmap, big initiatives for Plex include experiment democratization as part of the goal to “test as many feature releases as possible and enable engineers across the company to set up and run their own experiments.” In addition, Plex sees the partnership with Statsig as crucial to answer questions like:

• What can we learn from past experiments?
• How do we identify and prioritize the most impactful experiments?
• How do we improve velocity through empowering teams and individuals to test more, setting up tests that do not require engineering support?

To find answers, EV describes Statsig as fellow “experts in experimentation” who also “live and breathe” experimentation. “Statsig feels like part of my team,” she says. Every experiment gets me excited. Experimentation is the unknown and the more you know, the more you can iterate and grow.”

Building on another achievement

As Plex builds on the success of the recently launched Discover feature, A/B testing and experimentation with Statsig is a key part of the roadmap. “Plex has some unique features centered around additional content discovery and community where we will be trailblazing without many reference implementations in the wild. So, we expect a lot of experimentation guiding us to provide the best experience possible to our users,” EV says. “We certainly plan to use Statsig for any new major feature releases. The beauty is that the tool allows us to both run experiments, but also track the impact of planned feature releases.”

As Plex grows, EV views Statsig as evolving alongside with new quality of life improvements and the ability to inform product updates. “The Statsig product feels mature enough. It definitely feels like an evolving product because Statsig is leading by example and keeps improving and changing the product based on needs. The ‘learning mentality’ is engrained in the Plex culture, and I’m looking forward to expanding that even more.”

When asked how she describes Statsig to her colleagues and peers, EV replied: “They are the experts in experimentation and our partners as we build the muscle that allows us to learn more about our users, improving our product as we grow.”