In most of the evaluations I help customers with, they are deciding between building/improving their in-house system or buying Statsig.

I’ve observed some patterns and developed some standard questions that I ask these customers, which I think objectively helps them begin to understand which of the build and/or buy options is right for them.

Longer term I plan to create a calculator or survey that can help people understand which option they should prioritize, but in the meantime I want to share some key factors to consider and questions to ask yourself.

Technical expertise

Building an experimentation platform requires specialized engineering, data science and data engineering skills and resources.

If your team has the necessary technical expertise, building a platform may be a viable option. However, if your team lacks the required skills, buying a platform can be a faster and more cost-effective solution. These are the main questions I use to help teams determine if they have the technical expertise to build an end-to-end experimentation platform in-house:

Have your team members built similar systems before?

What kind of infrastructure and architecture is your team familiar with?

An in-house experimentation platform requires a solid infrastructure and architecture that can handle large amounts of traffic and data. This includes SDK design and implementation, robust data pipelines and scalable and accessible storage. How heavily does your team rely on other teams to access, understand and manage data?

If your team relies on other central teams, what would that mean for your experimentation platform’s access to data and maintenance? How much cultural and organizational shift would have to happen?

SAAS providers have dedicated teams focused solely on developing and improving their software, which allows them to adopt new use cases and technologies than an in-house team might be able to. SAAS providers can also leverage economies of scale to build and maintain robust infrastructure that is often more reliable and cost-effective than what an individual organization might achieve on its own.

On the flip side, if you have a highly unique infrastructure, you may have trouble finding a SAAS platform that can eloquently integrate and grow with you.

After surveying your team, if the answer to this question doesn’t inspire confidence, you should consider buying, or a hybrid build and buy.

How familiar is your team with statistical analysis and experimental design?

• Which types of tests will fit your business’ needs?

• Can you implement those at scale?

• Can you automate power analysis?

• How long would it take you to build a manual v1 in a notebook?

• Can this be implemented in such a way that trusted experimental results can be understood quickly?

Over-engineering the stats engine can be a big pitfall. It should be well-designed and engineered but it should also be simple and usable. If your data scientists are running multiple types of tests and it takes too much time to interpret results, you’ve missed the mark. Experimentation is now a bottleneck and innovation is stifled.

While academic materials are a great starting point to learn about experimental design, they may not provide all the insights needed to achieve optimal results in real-world settings. There's a wealth of practical knowledge on running experiments at scale that can be found within successful companies like Google, Amazon, Microsoft, DoorDash, Pinterest, Linkedin, Uber, and Netflix.

Access to this community is crucial in order to learn how to strike a balance between being pragmatic and rigorous in your approach.

Time to market

Time to market is fairly straightforward, but important nonetheless. Building a scalable end-to-end experimentation platform from scratch can be a time-consuming process—I would say at least a year from 4 engineers, a data scientist and a data engineer just for a minimum viable product.

On the other hand, buying a platform can provide a turnkey solution, albeit not exactly tailored to your needs, that can be implemented quickly and easily. If time-to-market is a critical factor for your business, buying a platform may be the best option.

Customization

Companies that have unique or complex requirements may find that building a platform is the only way to achieve their goals. However, if your requirements are relatively standard, buying a platform can provide sufficient customization options to meet your needs.

The key advantage of building your own system is the ability to address problems that are unique to your business or systems. Most people worry primarily about the stats engine, which is important to get right, but in my opinion, a usable and pragmatic experimentation system trumps all.

When evaluating for customization, think about more than just the stats engine. For example, the experimentation platform should integrate seamlessly with the engineering processes and tools of your organization. The platform should help democratize data within your organization and add velocity to your product release cycles.

An important consideration of building is that the solution you create may prove to be too specific, which can limit your ability to adapt to new or slightly different problems over time. This can result in additional time, effort, and resources needed to solve these new problems, leading to potential delays or inefficiencies in your operations.

If your use cases are narrow but highly custom, you’re probably a good candidate for build. If you have broad use cases, high scale throughput, and value time to market, but your team has strong customization needs, you may be a good candidate for a build-and-buy hybrid model.

Maintenance and support

Building a platform requires ongoing resources and maintenance to ensure that it remains up-to-date and meets your business needs. On the other hand, buying a platform typically comes with ongoing maintenance and support from the vendor.

Companies that lack the resources or expertise to maintain and support a platform may find that buying a platform is a better option. These are the main questions I use to determine if a team may struggle to maintain and support an experimentation platform in house:

Can you objectively describe your team's current workload and how it might affect their ability to maintain and support an in-house experimentation platform?

Or, more simply put, does your team have the bandwidth? If not, you’ll have to add headcount. You’ll also need to factor in ongoing support and troubleshooting; this is a cost to your team and a cost that may slow other teams from shipping products.

What kind of support will your team need from other teams within the organization to maintain and support an in-house experimentation platform?

Request a demo

Statsig's experts are on standby to answer any questions about experimentation at your organization.

Grab a Demo

Cost

Cost is a critical factor to consider when deciding whether to build or buy a platform, but one of the most difficult to project accurately.

Building a platform can be expensive, requiring significant resources and expertise. For building a high-scale system from scratch, my guess is that it takes at least a year from 5 engineers and a data scientist. What would that cost your organization?

Furthermore, do you have those resources? Build/buy hybrid scenarios are much more bespoke and harder to generalize on cost. Buying a platform can be more cost-effective, as it eliminates many of the upfront costs associated with building a platform. However, companies should consider the ongoing costs associated with buying a platform, including licensing fees and support costs.

Closing thoughts

In general, establishing a strong experimentation culture is the most critical aspect to achieving success. However, this is a challenge that must be tackled by each individual company. It requires significant effort and investment to build a culture that supports effective experimentation.

Focusing on undifferentiated infrastructure can divert resources and attention away from this crucial task, hindering the development of a robust experimentation culture.

Determining where your organization is on the build vs. buy spectrum is not easy. It can be political. Sometimes you’re talking about sunsetting a system built by someone you respect. Sometimes you can’t find a vendor that meets all of your needs. Further, determining the total cost of ownership of any option may prove impossible.

This blog isn’t the perfect answer to the build vs. buy conversation, but it hopefully helps you build out a mental framework that makes you feel more confident about the conversation, and helps you keep your team on the same page about how to make the decision!

Questions to ask:

This is a summary of the questions that are typically asked when making a decision on whether to buy a solution outright, or build one in-house. If this is a conversation your company is having, or may have in the near future, these questions can help frame the discussion in a useful way.

• Have your team members built similar systems before?

• What kind of infrastructure and architecture is your team familiar with?

• How familiar is your team with statistical analysis and experimental design?

• Which types of tests will fit your business’ needs, and can you implement those at scale?

• Can you automate power analysis?

• How long would it take you to build a manual v1 in a notebook?

• Can this be implemented in such a way that trusted experimental results can be understood quickly?

• Can you objectively describe your team's current workload and how it might affect their ability to maintain and support an in-house experimentation platform?

• What kind of support will your team need from other teams within the organization to maintain and support an in-house experimentation platform?

Create a free account

You're invited to create a free Statsig account! Get started today with 2M free events. No credit card required, of course.

Create my account