With over 375 million tracks from 40 million artists from 193 countries, SoundCloud empowers artists and fans to connect and share through music.

Transitioning from feature flagging to experimentation

Don Browning, SVP, Data & Platform Engineering, described the need for a shift in both tooling and culture.

“When I joined SoundCloud in 2023, we were using a homegrown assignment service to implement feature flagging. We hadn’t yet embraced the idea of treating product features as experiments tested on our users. Our previous approach was more along the lines of: This is a significant change, so let’s roll it out to just 10% of our users and progressively roll out from there.”

Reflecting on the shift towards viewing features as experiments, Don remarked, “It was a culture shift as much as it was a technology change.”

Searching for an end-to-end experimentation solution

On the build vs. buy debate, Don commented, "People shouldn’t underestimate the amount of time it takes to build a homegrown solution.” He emphasized that since SoundCloud aimed to instill a culture of treating product changes as experiments, they needed to ensure the tool adoption process was as straightforward as possible.

SoundCloud conducted a thorough evaluation of several vendors before ultimately selecting Statsig for two primary reasons:

• An end-to-end platform with robust SDKs for feature rollouts and experiment execution
• A cost-effective solution that aligns with their event volume requirements

“We evaluated Optimizely, LaunchDarkly, Split, and Eppo, but ultimately selected Statsig due to its comprehensive end-to-end integration. We wanted a complete solution rather than a partial one, including everything from the stats engine to data ingestion.”

Don highlighted that the absence of an end-to-end integration would have introduced significant adoption challenges. He emphasized, “Our goal was to maximize the adoption of experimentation, especially as we were in the early stages of cultivating this culture, and Statsig played a crucial role in minimizing adoption barriers.”

“Statsig’s client SDK component, in particular, impressed us and we ultimately went with Statsig because it provided a full-stack solution at a cost that aligned with our requirements.”

One year with Statsig: Incremental progress in experimentation velocity

The technical implementation of Statsig proceeded smoothly and swiftly. SoundCloud’s primary focus was on ensuring that the SDK was deployed in all areas and that each client team set up the SDK. In Don’s own words, “It was easy, taking less than 2 weeks, and the documentation was concise, with excellent examples. It was a turnkey solution.”

As more people adopted Statsig, the creation of tests gained momentum. Don explained, “We started with one test in the first month, followed by 5 in the next quarter, and then 12 in the subsequent quarter, eventually surpassing 20 experiments by the end of the year.”

This progress was significant, as Don pointed out, “Our experimentation was virtually non-existent before Statsig. We lacked both the tooling and the cultural inclination, and we had to advocate the concept to 40 internal product team members.”

Ultimately, Statsig empowered SoundCloud to transition from conducting zero experiments to conducting 20 in their first year!

Rolling out their flagship news feed with Statsig

A noteworthy milestone in SoundCloud’s journey with Statsig was when they introduced their entire news feed using Statsig.

“All testing and iterations for our flagship news feed release were carried out using Statsig. It played a crucial role in helping us make key decisions about how the feed should look.”

Don Browning

SVP, Data & Platform Engineering

Don highlighted the scale of SoundCloud and the diverse user groups they needed to cater to. “We have a large monthly user base of 50 million MAU, and every user gets bucketed in an A/B test at some point because we conduct different tests across web, iOS, and Android platforms. Statsig enables us to cover all these modalities seamlessly.”

Experimentation as a driver of revenue growth and profitability

SoundCloud has integrated experimentation with Statsig as a core element of its product development process, driving its pursuit of revenue growth and profitability. Zachary Zaranka, Director of Product, stated that they achieved a directly attributable ROI from their experimentation efforts with Statsig.

“Leveraging experimentation with Statsig helped us reach profitability for the first time in our 16-year history.”

Zachary Zaranka

Director of Product

He highlighted their utilization of experimentation in expanding their fan subscription business, a fundamental driver of growth. “Statsig supported the experimentation that led to a 30% YoY increase in annual revenue for our fan subscription business.”

Working with Statsig

“I’ve worked with a bunch of vendors, but Statsig is one of the best. You all leaned in from day one," Don expressed. He and his team had an excellent experience collaborating with the Statsig team during implementation and onboarding.

“Statsig was always asking how they could assist, monitoring our progress, and checking in with us. Jiakan (Enterprise Engineering) conducted a couple of walkthroughs for different audiences, including managers and directors who would be impacted by experimentation.”

When asked about the road ahead for experimentation at SoundCloud, Don shared their aspirations to conduct more experiments and exercise greater selectivity in the features they roll out. "We are making progress. Now, major product features undergo experiments, and we assess their performance against cohorts. Ultimately, we aim to identify and kill more features that have a negative impact on metrics.