Ben Gilbert: Tracking Network Effects
This article is part of a series called “The Data-Driven Product Manager,” interviews with product experts to help you use data to improve your product.
I met Ben Gilbert at the Industry Conference in Cleveland in September, where he presented a talk called “Universal Truths in Birthing a New Product — at a Pre-seed Startup or the Biggest Software Corporation in the World.” He hypothesized that companies with the ability to scale fast (and therefore be attractive to investors) usually incorporate “network effects.” These are ways that growth tends to lead to exponentially more growth, because the value of the product for the user depends, at least in part, on more people using the product. Great recent examples would be Airbnb, Slack, Uber and Amazon, to name a few.
Ben is a founder and partner at Seattle’s Pioneer Square Labs, whose unique approach develops business ideas and pairs them with the right entrepreneurs to bring them to market. Building on the Lean Startup principles of iterate and fail fast, PSL tests if ideas are marketable, and then rolls out companies that have a head start in the market. To do this, they rely on both instinct and data to move forward and measure success.
Before joining PSL, Ben co-founded Madrona Venture Labs, and ran The Garage, Microsoft’s Grassroots Innovation program for side projects. Ben has a background in iOS and web development, with past stints at Microsoft, ExactTarget and Cisco. Along with a passion for innovation and entrepreneurship, Ben’s an Eagle Scout who loves outdoor adventure and is also the host of the Acquired podcast.
We had a great conversation and I’m excited to share some of Ben’s thoughts with you here.
Data tracking should start immediately
I first spoke to Ben about how the companies he works with at Pioneer Square use data to develop their product and their market fit.
In all the companies that we start, we’re absolutely vigilant about full data instrumentation from the beginning. Before, there’d be this weird three-month gap around data at the beginning of the company, so it was like, “Well, we have analytics on,” but it wasn’t like we really thought it through. For example, a company would say, “Yeah, we had Google Analytics set up,” but now it’s like, “Did we actually put events in, did we properly add UTM parameters to everything so we understood when people are coming from this source, and they had this kind of lifetime customer value?”
We’re actually using this company, right now, called Segment that’s helped a lot. It’s unlike every other analytics platform I’ve ever used that has some non-ground truth version of your data that they store and you don’t ever get to directly query it. With Segment, it actually lives in my production database. That enables massive developer power. It can help inform people across the organization that aren’t going to be writing code. It acts as the broker to all the other services and keeps all the historical data, so if later on you’re like, “I wish I had been using KISSmetrics the whole time,” you could just replay all your data into KISS, or into Mixpanel, or into Amplitude, or any analytics tool.
Cohorts and Correlations
Next, Ben and I discussed what metrics were important to the very early stages of a startup and how that impacts a company’s ability to scale later on.
Now what we try to do in all the companies that we start is attribute as much as we possibly can to the marketing source. Whatever we can possibly learn about when a user came in, how we got them, really looking at cohorts, not only by time, but understanding what groups of people, later on, become very valuable customers.
A lot of start-ups get to this point of assessing “cost of customer acquisition” and they don’t consider lifetime customer value, the other half of the equation. It’s really hard to do that at first. You don’t have enough data to consider the lifetime customer value, so you just default to “how cheaply can we acquire people?
A lot of start-ups get to this point of assessing “cost of customer acquisition” and they don’t consider lifetime customer value, the other half of the equation.
You’re kind of screwing yourself over later if you didn’t track who your customers are, and how do some customers perform versus others a year later, looking at transaction history. This tells us how we acquired the most valuable customers, and if we can spend twice as much or ten times as much acquiring them if they’re going to be a power spender on our platform.
Data for everyone
We also discussed how important it is that everyone on the team can access data, and how non-engineers can understand the data that’s being tracked.
We have two worlds with data. Analytics from your production data for engineers, and analytics from your tools that you don’t need engineering to access. We’re seeing an interesting trend now where those Venn diagrams are merging so teams have the ability to do all the dirty work, but can also use another tool or build their own tool on top for everyone to use.
I’m a developer also, and I’d get asked a question by someone at a meeting when we’re looking at data, deciding if a company is viable and we should spin out. Like when we’re looking at the Mixpanel dashboard, and I’d be like, “Uh, I don’t think that’s tracked in analytics. I mean, I guess we have it in all the timestamps from user signups and actions they took in our app. I can try and figure that out from our database.” I’d be doing that, and then I’d get another question that was available in the analytics.
It seemed like these worlds were stupidly separate, and I was frustrated with that. I’m still new to this newer stack with our Segment implementation, but I think tools that allow everyone access are the future, where you get to answer all the questions from one place instead of bouncing back and forth between two, requiring two completely different skill sets.
I think tools that allow everyone access are the future, where you get to answer all the questions from one place instead of bouncing back and forth between two, requiring two completely different skill sets.
Teams tend to divide along data, with marketing always tracking acquisition and product always tracking engagement. I think the success of companies comes in marrying those two things together and understanding that your acquisition doesn’t matter unless there’s engagement. What really matters is how long you can extend that user lifetime.
What we know we don’t know
I asked Ben what kind of pitfalls he’s encountered on his road to being more data-driven.
In general, I’ve become much less trusting on any sort of acquisition data, because things change. I’ve over-trusted and I’ve over-represented the cost to acquire a customer without thinking about how deep the well is.
One thing that I definitely think about that I didn’t used to think about is “what are the unique conditions right now that I’m not factoring into my tests that are affecting all the numbers that are coming out of it?” A lot of that is like seasonality or things I don’t yet know about the business. Whenever you’re starting a new company, there’s so much about the space that you don’t necessarily know that could be impacting it.
“What are the unique conditions right now that I’m not factoring into my tests that are affecting all the numbers that are coming out of it?”
We’re working on a business right now called LumaTax that actually just spun out. LumaTax has an extremely seasonal period, because it’s small business sales tax filing. Nobody cares about sales tax until they get the notice from the government that they have to file their quarterly sales tax, so you get these really weird burst volumes where you actually should have all your marketing spend in one month. If you’re prototyping during a tax filing month versus any of the other ones, it’s completely different. And I didn’t know before if it was quarterly or monthly filings, so I couldn’t have even predicted that there would be a heavy month.
Customer Data FTW!
Ben shared a great example of a company that’s used data to their great advantage by tracking key details about their customer value.
I heard a really good anecdote about Zuily in the early days. They had a range of customers that they knew they could acquire for 50x more than other customers, because they were so good at understanding what the later behavior of those customers would be. They literally were spending 50 times more on some customers because they had confidence it was worth it!
I think the vast majority of start-ups would not know that. They would have one or two numbers, and they’d go, “Well, for this type of customer, we’d pay this; for that type of customer, we’d pay that.”
The vast majority of start-ups would not know that. They would have one or two numbers, and they’d go, “Well, for this type of customer, we’d pay this; for that type of customer, we’d pay that.”
I think what I’m aiming to do in all the companies that we start is have the ability to really slice and dice as much as we can know about a person so that we can say, “Yeah, it’s worth paying 100X the acquisition cost for this customer compared to that customer.”
Thanks again to Ben for taking the time to talk with us. Check out some other recent posts in the series here and stay tuned for some other great conversations from Industry Conf. You can also learn much more about using data for product development in our School of Little Data, a free email course we created at Notion to help you get started.
