Got Metrics? Yes, but are they Meaningful?

Robert Brodell
Product Coalition
Published in
5 min readFeb 23, 2019

--

Metrics that use business relevant data alongside intuitive trending and straightforward analysis provide a strong foundation when managing strategy

It took about thirty minutes to get everything together. Seven slides showing product metrics that required data pulled from six different systems be graphed and heavily footnoted eliminating any confusion on what the numbers represent. In only three minutes my executive derailed the effort by asking: “We have metrics, right?” After a stunned silence, he spoke up again: “I see the numbers and graphs, but what are we measuring and why?” As silence took hold once more, I realized that our “metrics” were meaningless.

Product managers seek meaningful metrics that test our assumptions and tell us something actionable about our products. Sometimes metric data overlooks relevant product and business information. Other times, convoluted trending and analysis obscures relevant data. Each case results in a well-intended metric that nobody but the product manager can comprehend. Such meaningless metrics lack business impact or are too nuanced for the team to act on.

Meaningful metrics communicate business relevant information in simple terms. A suite of meaningful metrics drives action by telling us where our products succeed and where they fall short. Meaningful metrics depend on business relevant data. Some companies define certain data points as business relevant, but as product managers we should consider our unique product and ensure no data is overlooked.

Identifying Business Relevant Data

Frameworks help us assess the relevancy of different types of data. Google’s HEART framework is a great place to start. “HEART” represents five categories of metrics Google teams consistently use:

Metrics built on relevant data empower us develop impactful strategy

1. Happiness

2. Engagement

3. Adoption

4. Retention

5. Task Success

Happiness data tells us about user attitudes. If customer satisfaction is vital to business performance happiness metrics are useful. Engagement data illustrates user behavior and level of participation with a product. Engagement data should inform metrics when increased product use drives success. Adoption data quantifies baseline user growth and helps us create metrics for products that need to expand market share. Retention data enumerates users who continue to use or abandon a product over a set period. Retention metrics are beneficial when gauging baseline use of a product over time.

Task success data quantifies product performance. Most product teams find producing and intuiting task success data easier than data in the four other categories. Consequently, task success data often feeds meaningless metrics. Only when paired with another HEART metric does task success data illuminate performance’s impact on user behavior. This business relevant data is meaningful. It helps us understand how performance drives the bottom line.

Product managers often find the HEART framework helps identify meaningful data to feed metrics. HEART categories do not carry equal importance. In fact, some categories may have no bearing on an individual product’s business. Using the framework as a guardrail, not a check list, helps us consider all available data.

Developing Meaningful Metrics

After selecting data relevant to our business, we can develop meaningful metrics. Business critical data alone does not constitute meaningful metrics. We need to distill this data into understandable statements that drive action.

Relevant data fails to tell us anything actionable without context provided by trending and analysis

Yet another framework can act as a guardrail to ensure we produce simple and actionable metrics. Remember the fill-in-the-blank story books we filled out as children? They included simple statements like:

(name 1) and (name 2) went to (verb) in the park.

Applying that same structure to metrics ensures simplicity. For example, an adoption metric with this structure provides data, trending, and analysis needed to inform actionable steeps in just one sentence:

(#) customers onboarded to the product last week which is (up / down) (%) since our last release meaning customers (may like / may not like) the new (feature name).

These metric stories tighten feedback loops by telling us good, or bad, news so appropriate action can follow. Using the story above we can create actionable good news:

246 customers onboarded to the product last week which is up 15% since our last release meaning customers may like our new UI features.

After reviewing this metric, the product team may elect to expand marketing for the new UI. Using the story above we can also create actionable bad news:

246 customers onboarded to the product last week which is down 10% since our last release meaning customers may not like our new UI features.

The team may elect to rework some of the UI after reviewing this metric.

In each example our product onboards 246 customers after releasing new UI features. That data alone does not drive action to improve adoption. Trending on percent change since the prior release paired with a brief analysis informs what action we can take to improve adoption.

Ultimately, the issue is not one of metrics versus judgment, but metrics as informing judgement…

- Jerry Z. Muller, The Tyranny of Metrics

Unfortunately, many metrics overuse caveats and visualization which obscure trending and analysis past the point of intuitive understanding. Trending and analysis become overcomplicated when well-intended product managers attempt to manipulate data to safeguard the product’s image or validate an existing strategy. The fill-in-the-blank framework ensures our metrics speak for themselves.

Benefits of Meaningful Metrics

Connecting business relevant data, intuitive trends, and straightforward analysis shortens our feedback loops and empowers us to adjust strategy as needed. Meaningful metrics do this. They rely on business relevant data and present that data alongside simple trending and analysis to drive action. We may need to use visualization and footnotes to emphasize the metric’s message. But using the HEART and fill-in-the-blank story frameworks to produce metrics should eliminate the need to explain our metrics to an executive.

--

--

I'm a product manager & freelance writer. My writing explores best practices, product mindset, and complex product challenges. RobertBrodell.com @RKBrodell