Test, confirm, and evolve decisions: Five questions for Hunter Walk
This article is part of a series called “The Data-Driven Product Manager,” interviews with product experts to help you use data to improve your product.
Hunter Walk began his product career at Linden Lab and moved on to Google, where he led product development at YouTube before forming Homebrew.co, a seed-stage VC firm. He’s become an important voice in the product community through his blog, which sports the memorable tagline “99% Humble, 1% Brag.”
As a VC at Homebrew, Hunter has made a series of successful investments, including Layer, Shyp, and The Skimm. As a product leader, Hunter advocates for humility, ethical practices and value to customers and the world at large. As a writer, Hunter has perfected the “Five Questions” interview, so we were excited when he agreed to answer five of our questions for him.
Since our mission at Notion is to help teams use data to better communicate and collaborate with their company, team and leadership, I wanted to talk with Hunter about his approach to data as a product person and investor, to learn what has worked for him.
I first asked Hunter where he saw practices around being data-informed or data-driven going in the next few years for himself and the companies he’s working with.
Given that Homebrew takes a concentrated investment strategy at seed stage (8–10 companies per year), I’m not sure we’re ever going to use standardized data-driven filters as rigorously as perhaps an accelerator or later stage growth investor would. That said, we track and crunch a lot of data to help us identify topics we want to discuss about our performance, potential blind spots in our investment practices and so on.
As for the startups we back, the average founder has embraced the increasing availability and ease of use of data tools. Because of SaaS and API/SDK toolsets, it’s not uncommon to see a 2016 seed stage company with the kind of realtime business intelligence dashboard that just a decade ago you would have needed to be a Fortune 500 company to implement.
In recent years, many influential software leaders have advocated the use of a “one metric.” After writing a SaaS-focused KPI guide, called the AARRRT of Pirate Product Metrics, I was curious how Hunter feels about the concept and what his experience with that approach has been.
I believe that KPIs are essential to a company and that over time, shifting KPIs can be one of the most powerful levers in ensuring organizations focus on big goals. For example, YouTube’s first six years or so were all about maximizing daily playcount. Then they shifted to trying to maximize amount of video viewed per day as a better proxy for engagement and satisfaction. It was the right change at the right time.
KPIs are essential to a company and that over time, shifting KPIs can be one of the most powerful levers in ensuring organizations focus on big goals.
The Lean Startup has been influential in guiding tech leaders to adopt measurement into the growth cycle. I asked Hunter to talk about how the “Build-Measure-Learn” philosophy impacts the companies he funds.
We don’t enforce any particular methodology on the founders we back but we do evangelize rate of learnings vs rate of doing. That is, we don’t think just getting a lot done is proof of progress if it’s not in a “hypothesis -> test -> refine” learning loop.
I wondered what mistakes Hunter has seen when companies try to use data to learn.
They lose sight of the bigger picture and get trapped in a bunch of local optimizations and multivariate results which overall don’t turn cohesively into a product. I don’t think you can A/B test your way to an amazing product. You can use data to test, confirm and evolve decisions.
When we were scaling YouTube, one decision we made was to basically give every product/engineering team the ability to run a 1% test without asking for permission. We built an experiment framework which made it easy to run these tests in a standardized way. That way any debate could eventually be settled with “why don’t we just test this out” versus the chilling effect of making it hard to combine quantitative results alongside intuition.
I don’t think you can A/B test your way to an amazing product. You can use data to test, confirm and evolve decisions.
Has data ever played a part in a decision not to invest?
One example that comes to mind is that while we don’t expect seed stage founders to over-model their business — a detailed five year forecast is silly when you’re just starting out — I do want teams to understand the assumptions which matter and sensitivities.
For example, a few years back we spoke to a food-delivery service that asserted their cost of customer acquisition would be zero. They were making this assumption because in early tests it spread virally among their friends and friends of friends. After congratulating them on this early success, I noted that really no business had ever gotten to scale on just word of mouth. If they were the first to do so, that would be awesome, but what would happen if they plugged in average CAC estimates into their model? Not only did they think this was a waste of time, but they also had no idea what the average CAC was for their more mature competitors. This company did successfully raise a seed round, but they’re now out of business.
Finally, as a useful takeaway, I’ll mention Hunter’s own blog post on Objectives & Key Results (OKRs) for tech companies. OKRs, developed at Intel and popularized by Google, are a great way to get your team aligned with data. Hunter suggests specific goal-framing for startups, summarized briefly here:
One Month — “What are we building this month?” is the key question.
“N+12 Months” — [Ask] “What will our product and business look like a year from now?”
Quarterly/Annual KPIs — Keep a very narrow grasp on what you actually want to measure — just key drivers of business — and set quarterly targets. There can be a reality check — do these quarterly targets get achieved given what we’re building?
Thanks again to Hunter for answering our questions! If you’d like to learn more about data, check out our last articles in “The Data-driven Product Manager” with UX expert Laura Klein and product consultant Jock Busuttil, and sign up for The School of Little Data, our free email courses to learn how to use your data to collaborate with your team. Notion is a new tool for teams who want to communicate with data, and we’d love your feedback on how we can best serve your needs.
