A massive opportunity for experimentation

Parafin is a full-stack embedded financial services company that serves tens of thousands of small businesses on marketplaces like Amazon and Doordash. Brooks Taylor is the company’s Data Science Lead. His team specializes in data modeling, predictive analysis, and scaling a hypothesis-led culture.

Brooks recognized the significant business potential of experimentation. “With a large base of end-users (merchants) through partnerships with companies like DoorDash, Amazon, and Mindbody, we saw a high-yield opportunity for experimentation in underwriting. We could test various offers: their size, duration, and cost.”

Given that Parafin underwrites hundreds of thousands of policies, Brooks acknowledged the wealth of data available and the potential to iterate and optimize their business metrics. However, he also highlighted the nuances of their business model, stemming from having limited direct access to customers until they demonstrated high intent.

The need to upgrade existing tooling

Parafin had been using LaunchDarkly for feature flagging but encountered roadblocks in increasing experimentation velocity and developing a hypothesis-led culture across the organization.

“Before Statsig, we were doing a lot of manual, ad-hoc work, limited to basic feature flagging. There was a lot of shoulder shrugging and muddied analysis of outcomes,” explained Brooks.

Parafin recognized the need for more sophisticated, end-to-end tooling that would allow for logging events, conducting analyses with statistical rigor, and bringing cross-functional teams into the experimentation process.

“Our previous feature flagging tool wasn’t integrated with the rest of the business. Engineers would push merchants through with feature flags, but data wasn’t reliably logged and documented for us to analyze,” Brooks elaborated.

Initiating the experimentation journey with Statsig

Parafin ultimately chose Statsig due to the platform’s ability to comprehensively measure business outcomes beyond simple events like clicks or page views.

“We had a significant opportunity cost in not being able to swiftly align teams and make decisions that would positively impact the business metrics,” explained Brooks.

Parafin initially self-served and started using Statsig for the following reasons:

• Developing a source of truth for metrics
• Codifying the analysis process
• Sharing results with the broader generation outside of Data Science

Parafin’s North Star for their experimentation program was clear: grow revenues and increase profitability. Brooks outlined three areas where Parafin began experimenting to drive impact: the underwriting engine, product back-end, and front-end.

Explaining why they consolidated capabilities into a single platform, Brooks said, “We don’t want to pay for too many tools. With Statsig, we were able to set up a single data integration and view all our exposure events.”

“Statsig is a powerful tool for experimentation that helped us go from 0 to 1.”

Immediate positive impact with Statsig’s experimentation

Brooks wanted his team to focus on getting their first successful attempts off the ground as quickly as possible and build experimentation velocity.

“We had 5 Data Scientists on my team and the goal was that each would create an experiment.” They saw an instant impact to their top-line with their first experiment.

“Our first Statsig experiment produced an 18% increase in loan origination, therefore helping us capture more revenue.”

Brooks Taylor

Data Science Lead

Scaling the experimentation culture: More wins, more learnings

As they began running more experiments and increasing their velocity, Parafin made more data-driven decisions that positively impacted core business metrics. Consequently, Parafin increased investments in their experimentation program and upgraded their Statsig license.

“As a growing company with many questions from Product Managers and Marketers, we recognized the value of having a platform that serves as a central source of truth. This enables everyone to self-serve, saving unnecessary back and forth. Otherwise, our Data Scientists would always be on the hook for storytelling, which has a significant time cost.”

Brooks also highlighted the value of unsuccessful experiments, stating, “Many people bring ideas from their previous companies, which may not be replicable here. They wanted to focus on retention and dedicated resources to reactivation, but we discovered that there wasn’t much upside in that area.”

Addressing unique challenges

“Money is a fundamentally different kind of product”, quipped Brooks, highlighting various considerations in his mind:

• Adverse selection: Brooks explained, “Despite what you may know about a prospective borrower, they possess information you don’t. There are invisible risks, and they need to be managed.”

• Behavioral differences: These are based on criteria such as whether merchants have taken a cash advance or are repeat customers.

• Confounding metrics: Variables such as the health of their business and how quickly they are paying off loans determine whether they will return in the future.

• Visibility gaps in the customer journey: “There is limited visibility in the early stages. By the time customers materialize for us, they exhibit extremely high intent,” explained Brooks.

“There is an inherent growth versus risk tradeoff—and risk can manifest in various ways. We may perceive low risk and end up biasing our approach if we give an offer to everyone, which could inadvertently increase our risk profile. On the other hand, if we wrongly predict a merchant as risky and make them ineligible for the offer, then we could be missing out on business.”

Making data-driven decisions with Statsig’s Experimentation + Analytics

Parafin leveraged Statsig’s sophisticated statistical capabilities to inform core business logic—testing prototypes and fine-tuning their models to approve offers for borrowers.

“With Statsig, we’ve been able to set up holdouts in a controlled manner, modulating the volume of offers and how much we are willing to invest in learning and optimizing our models,” Brooks explained. This approach helped his team validate a prototype and determine whether redesign was necessary.

Brooks detailed how his team uses the built-in analytics in Statsig to delve deeper into metrics and guide the decision-making process. “We can explore metrics across the customer journey and utilize charts like ‘funnels’ to effectively prioritize efforts and shape our roadmap decisions—deciding, for example, between focusing on retention vs acquiring new users.”

Given that Parafin was starting from scratch, Brooks noted the importance of being able to build a reliable catalog of metrics, “Statsig helped us log events to build out the library of metrics – which is especially important for startups – and that helped us get off the ground faster.”

On working with the Statsig team, Brooks said, “When we’ve had technical questions about things like latency, we’ve gotten good insights, quickly.

“I like that when interfacing with Statsig support, you get to talk to someone who really knows the product—the Engineers who are building it—and not just a Support Rep.”

Brooks Taylor

Data Science Lead