While the panel at Mixpanel's DDC with Chris Lambert of Lyft, Stan Chudnovsky of PayPal, Adam D'Angelo of Quora, Jared Fliesler of Matrix, and Patrick Collison of Stripe was titled 'The Secret Weapon every Startup should use: Data', the panelists were more cautionary if anything, noting the increased reliance on data and the potential for this to lead to poor decision making. Without guidance from larger, high-level statistical models tied to a solid understanding of the vision of the product by a company's leadership (often guided by one 'north star' metric), startups must be cautious in how data-heavy their approach is. The benefits still outweigh the potential downfalls however, and the panelists recommended that everyone should start collecting and looking at their data even in their product's very early days.
One (carefully chosen) metric
When you work with one basic number as your 'north star', you have to be really, really careful how you choose that number. Patrick of Stripe gave a specific example of how Stripe's 'Average Number of Transactions' would never make sense as a metric to align their company around, and actually isn't useful at all - that number is 0. It's so easy to set up a Stripe account that there are a lot of people who simply set up an account just to look around at the product. Perhaps they're happy with what they see, but they're not actually handling any online transactions so they don't need the service. This leads to an extremely skewed distribution for this one metric, but in no way indicates anything about the health of the business, the retention of users, the value of new users, etc.
Jared from Matrix spoke about his time at Square, where his first job was to come up with a single metric, their 'north star' that the entire company could really optimize for. On the merchant side, they knew that number of transactions was an important metric to think about. They also noticed that once merchants reached a certain number of transactions, merchant churn dropped drastically. It was this insight that then led to their decision around growth to focus on that one metric.
Alignment to a statistical model
As companies get bigger, problems sometimes emerge where all teams are pushing for their own discrete metric. Adam of Quora spoke about the importance for the company to get to the best outcome overall, maximizing the entire company not just individual metrics. If those metrics are too narrow, another problem is that they won't account for externalities inflicted upon other teams. The best way to deal with this issue is through employing a top-level statistical model to understand how the different metrics of different teams add up and affect the company wholistically. At Quora, they look at how much knowledge will be shared in the world. This lets Quora's leadership trade off different movements that different teams can make, and the model makes sure everyone can maximize the big picture. Stan of PayPal also believes in the power of statistical models, and testing if your model holds as data comes in - If it doesn't, then maybe your business is changing, maybe you need to adjust the model, or maybe there's something else going on that you need to understand. He suggested building your predictive model early, and see if it holds or not over time by constantly feeding collected data into it and checking outcomes.
With respect to internal dynamics and culture related to growth, Adam noted that it's often easy for the growth team to measure what they're doing, but hard for others. You can potentially end up with a super-data driven growth team that focuses solely on short-term, local optimizations. This is something that others in the company can come to mistrust and not like or appreciate, and in the long term may end up resisting anything related to data. This is an extremely counterproductive state - the growth team ends up doing all of the work and having more impact than the rest of the company. Adam recommended spreading the use of data across the company - while there's one model, one consistent way to evaluate progress, this is shared by everyone, not just the growth team.
Start small, and proceed with caution
Stan also spoke about how Data Scientist often request and feel the need for very large data sets in order to start deriving any insights. This leads to some founders and CEOs feeling like they're never quite there - they're too small to start collecting and analyzing data (let alone employing a Data Scientist), and think the potential ROI is negative. One piece of advice from Stan was to go on 'data discover trips' in the early days of your product, i.e. randomly write queries and see what interesting clusters, trends, and insights emerge. You can get a lot of incremental answers with a simple approach like this, while it also helps get around your bias of assuming you know what you're looking for. The panel noted on several occasions that it's very easy to end up with a systematic bias by measuring and fixing things simply because they're easy to measure. Serendipity and measuring things that are difficult to measure shouldn't be undervalued at any stage.