Building features centered around design and UX

Notion’s product culture has been instrumental in distinguishing itself in the competitive SaaS market for collaboration tools. Wendy Jiao, a Software Engineer on the data platform team responsible for integrating the SDK and ensuring seamless metric recording, captures what sets Notion apart:

“Notion is not just a tool; it’s a dynamic collaboration platform that empowers users to tailor workflows and unlock their productivity.” Wendy’s role empowers data scientists and involves collaboration with various vendors, including Statsig.

Mengying Li, a Data Science Manager who collaborates with various teams to drive data-driven business decisions, says, “Our culture revolves around design and prioritizes user experience. We embrace innovative technologies that amplify our platform’s capabilities.”

The role of experimentation in Notion’s product development

At Notion, the emphasis is on rapid feature deployment without compromising long-term value. “We maintain a very agile development cycle, but our standards for feature releases remain very high,” says Mengying. "In this dynamic environment, engineers enjoy the freedom to experiment, but the spotlight is on learning rather than quick wins.”

Data-driven decision-making is central to their product development strategy, and early on, Notion recognized the importance of effective feature flagging and experimentation."

Before Statsig: The challenges of an in-house experimentation platform

However, they faced a challenges in running exclusive experiments. “Before we got Statsig, we found it time-consuming to add feature flags to every feature and couldn’t run experiments without overlaps”, said Mengying.

As one of the first data scientists on the growth team, she took on the task of exploring third-party vendors to solve the challenges of their existing in-house experimentation tools. “We couldn’t run many experiments with our in-house tool – which slowed us down. It didn’t make sense for us to spend time building a platform by ourselves,” said Mengying.

Key challenges

• Engineering resource constraints to scale their in-house tools
• Limited Data Science resources to analyze a large volume of experiments
• Inability to run multiple experiments in parallel due to inefficient traffic allocation

Mengying also highlighted specific pain points from a Data Science perspective:

• It was taking too much time to locate the correct feature flag, which delayed analysis
• There was a lot of manual work such as performing statistical analyses, p-value calculations by themselves

“It would take 2 months from set up to launch to analyze an experiment and it was very hard to debug.”

Why Notion embraced Statsig

In Notion’s quest for an ideal experimentation tool, Mengying outlined two fundamental guiding principles: “First, we needed a tool capable of effective feature flagging, and second, it had to empower both Data Scientists and non-technical users to self-serve experiment insights.”

After conducting an exhaustive evaluation of available market options, Notion selected Statsig for several compelling reasons:

• Pricing flexibility: Statsig’s pricing model (which includes no fee for seats) ensured that the entire team, including non-technical users, can access and benefit from experiment data.

• Seamless integration: “The integration process was seamless, tailored to Notion’s unique requirements,” Wendy explained. “Statsig showed dedication to building features that aligned with Notion’s needs.”

• Holistic metrics analysis: Statsig’s “Scorecard View” for experiments provided a holistic perspective on metrics. As Mengying explained, “Notion’s data analysis spans a range of product metrics. With Statsig, we gained the ability to review all metrics within a single view, a stark contrast to other tools that required us to navigate through individual metrics one at a time.”

Above all, what set Statsig apart for Mengying and Wendy was the exceptional experience of collaborating with the Statsig team. “We had unique requirements concerning SDK integration and metric ingestion that the team at Statsig understood well. They proactively developed features based on our specific requests.”

“Statsig’s people and vision aligned with our needs,” Wendy added. “The team exhibited dedication, humility, and responsiveness, which instilled in us the confidence to forge this relationship.”

Achieving 30X experimentation velocity

"We transitioned from conducting a single-digit number of experiments per quarter using our in-house tool to orchestrating hundreds of experiments, surpassing 300, with the help of Statsig."

Mengying Li

Data Science Manager, Notion

Notion began leveraging Statsig for a broad spectrum of experiments, encompassing everything from refining onboarding to enhancing activation and boosting engagement.

Notion’s experimentation journey witnessed a remarkable transformation, surging from single-digit experiments per quarter to over 300, incorporating hundreds of feature gates and dynamic configurations.

This exponential growth in experimentation didn’t require a proportional increase in engineering support. Remarkably, just one Engineer effectively manages the platform—a testament to Statsig’s user-friendly nature.

Wendy quantified the impact on productivity, stating, “The gains are substantial: a single Engineer devotes around 5% of their time covering Statsig SDK integration, as well as metrics ingestion pipelines. Otherwise, we might have required a full team of 4 Engineers.”

Driving core growth metrics with Statsig

"We've successfully launched over 600 features by deploying them behind Statsig feature flags, enabling us to ship at an impressive pace with confidence."

Wendy Jiao

Software Engineer, Notion

Notion effectively drives four key growth metrics throughout the customer journey via experimentation:

• Sign-up rate (new users)
• Activation rate
• Engagement (active use)
• Retention

Here’s a selection of notable use cases:

1. Using holdouts to measure aggregate impact: Leveraging a holdout group within Statsig, Notion’s growth team executed a series of experiments aimed at enhancing the activation rate by 10-15%, subsequently bolstering engagement and retention rates. Mengying highlighted the remarkable level of control they had in designing and executing experiments. “Statsig allows us to use different units for experiments, such as user IDs and device IDs, a feature not widely supported by many platforms.”

2. Uncovering unexpected yet impactful learnings from experimentation: An interesting experiment entailed randomizing the order of workspaces that were available for a new user to join during onboarding. Such experiments provided valuable insights for Notion and led to significant enhancements to the mobile interface and the sign-up flow.

3. Protecting against metric regression: Wendy, emphasizing the platform’s impact on protecting against regression of core metrics, stated, “We’ve successfully launched over 600 features by deploying them behind Statsig feature flags, enabling us to ship at an impressive pace with confidence.” She also lauded the level of control it granted, highlighting the effectiveness of dynamic configs in optimizing code at runtime.

4. Improving new user onboarding experience: Notion harnessed Statsig’s capabilities to fine-tune its onboarding process. They iterated on the sequence of questions presented to users, carefully selecting the order in which they were asked during onboarding to maximize the information gathered and personalize the user experience. This approach significantly contributed to improving the success of the onboarding and activation processes.

A thriving experimentation culture and the path forward

“We are an experimentation-driven company, and our leadership embraces this vision. We routinely evaluate the holdout and report our progress to the finance and leadership teams on a monthly basis.”

While the growth team was the first team to embrace the experimentation culture, more teams are starting to use experiments in their product decision-making process. As Statsig interfaces seamlessly with customer.io, marketing teams have also actively engaged in running experiments, particularly in email campaigns.

The organization is progressively becoming more self-sufficient as the culture of experimentation permeates various teams. Wendy shares her vision for the future, stating, “Our aim is to continue advancing our self-serve capabilities in experiments, accelerating the pace of experimentation across all teams.”

Notion’s overarching vision is clear: to move swiftly, learn continuously, and foster innovation.