Making mid-course corrections to your roadmap.

How to use data when plans inevitably change.

If you are a product manager, you have probably spent some uncomfortable time trying to manage a roadmap. You might face problems when product doesn’t measure up to the plan, or struggle to know what you should work on next.

With data playing an increasing role across all business decisions, it’s no surprise that it can be helpful with your roadmap as well.

Owning a roadmap with data exists on two planes — planning and adjusting.

Planning

The first plane of owning a roadmap involves strategic alignment. Does everyone on the team believe the features you are planning to build will get you to the monthly, quarterly, or yearly goals you set out to reach? What metrics should you track to define success?

Jim Semick of ProductPlan wrote recently on the difficulty of being data-driven with new products and features:

Metrics at the early stages can be challenging because there’s rarely a history of data. In some cases, if the product has recently launched, there might be a flood of data, but no structure or focus on the right metrics to use.

If you find yourself building an early-stage product roadmap, you’ll definitely appreciate his post. Clearly defined success metrics early in the process are critical to establishing short-term goals, long-term strategies, and team alignment around your roadmap.

Adjusting

The second plane — that is, making adjustments to your roadmap — requires a different approach to data. It’s full of quick calculations and fast thinking amid a fervent flood of questions. Can we squeeze in features to reduce our user churn? Why is the dev team focused on paying down tech debt when the sales team is demanding new features to sell?

Mid-course corrections are at the front line of data-driven product management.

When the success of your product requires you to be able to answer these questions and adjust your roadmap when needed, data becomes an invaluable tool to have at your disposal.

Thankfully, with just a little bit of data, you’ll make those corrections faster and more confidently. Looking at our questions from above, we now have answers.

Can we squeeze in features to reduce our user churn? Our dev team is currently maxed out on throughput and is already sacrificing time to fix bugs in order to keep up with feature demand. So squeezing in new features is unlikely.

Why is the dev team focused on paying down tech debt when the sales team is demanding new features to sell? Take a look at our cycle times. We have a bottleneck related to reviewing code. Even if we were finish more features, we wouldn’t be able to support them through the rest of the workflow. Once we resolve this, we’ll be able to ratchet up our throughput.

Using little data to make better, faster product decisions.

Before making any decisions that will change your roadmap, you should have a solid understanding of your dev team’s current productivity and if there are any weaknesses or known risks to be mindful of.

It’s worth keeping in mind that there is no secret metric that will work for every team. The trick is to find the team metrics that define your agreed-upon levels of team success.

If you are unsure of where to start, we find it’s helpful to ask two questions: How is the team doing? and How can the team get better?

Some metrics that help explain how your team is doing

Cycle Times
As with velocity, the amount of time spent delivering tickets is unique to each team—so it’s good to use cycle time metrics not to measure success, but as indicators for consistency. Watch trends and keep an eye out for unexpected variance. An unstable team makes it difficult to forecast future throughput.

Commitment Reliability
This is another great metric to watch to understand your team’s accuracy. Commitment Reliability looks at the delivered work in a period of time against the committed amount. Your level of comfort promising the delivery of last minute features might be tied directly to the reliability of the team.

Iteration Flow and Throughput
Watching the flow of tickets through the development workflow is a great way to identify bottlenecks. To help build consistency and accuracy, keep an eye on the cadence of your work cycles. If a lot of work comes in right before a release, how does that affect your quality or release teams? If there are peaks and valleys in your throughput, were they expected or do you need to dig deeper to better understand hidden risks?

Team Morale
Quantitative data usually only tells half the story, so it is important to get a pulse of your team health. This can be done through simple surveys rated on a scale of 1–10. “How productive we think we were this sprint? What is our confidence level in code quality? How well did we communicate as a team this sprint?” Sometimes this info comes out if you run retrospectives, but having it as chart is great to find correlations.

Delivered Value
This is another metric that can benefit from quantitative and qualitative data. Since feedback from your customers is a lagging indicator, asking your team how much value they feel they are delivering is a great way to measure expectations vs. reality. If you have a successful track record of perceived value aligning with actual value, you’ll be more in tune with roadmap adjustments.

Some metrics that can help a team get better

Defect Removal Efficiency (DRE)
Think of this as a better way to tracked escaped bugs. Dividing the number of bugs the team is finding and fixing before a release by the total number of bugs found (including the ones found in production after a release) provides the team with a measuring stick to improve their code quality.

Story Churn
While sales and marketing are worried about user churn, reducing the amount of story churn is a great way to improve your dev team. If stories are being rejected, it might be the result of product requirements needing further definition. Or perhaps your documentation flow is broken, making it easy to miss feature specifications. Reducing churn at this level results in a more productive team across the board, meaning you’ll be ready to adapt to roadmap changes more easily.