How to build a data driven culture: Metrics to learn, not to manage
I once designed an application for a large construction and manufacturing company to help factory workers track the efficiency and quality of their production line. The business goals were pretty clear: we need more data about the production line to make more informed decisions about what to improve. Do we need more staff? Do we need to refine our processes? Do we need to upgrade or replace our machinery?
All of these decisions were currently based on ad hoc feedback coming from the supervisors or team leaders from the factory shop floor. The data-driven German in me was stoked — more data, more measurement, awesome! It sounded like a no-brainer to me.
The vision was to invest in automation, for example by using more sensors in the machinery in the long term, but first we wanted to test the concepts through a tablet application for the production line workers. Through this application, the shop floor staff would be able to manually enter the output of a certain time period, machinery outages or issues, or raise other quality related feedback. A dashboard for team leaders and business stakeholders would then consolidate these inputs into efficiency and quality scores for reporting.
When I went into the factory to test an early version of the prototype with the shop floor staff however, we didn’t get quite the reaction we were hoping for. While I could clearly see benefits for them as well, for example they could now raise frustrating issues and get them resolved much faster than before, I could sense a general hesitation when I asked them for feedback.
One of the staff finally asked: will this be used to track our breaks or time spent on the machinery? Will our team leaders use this to compare our outputs against each other? What happens to our jobs if our numbers are not good?
The actual design of the application was not the issue. We really had to go back to the drawing board and reconsider the impact on the staff, their managers and how it would affect performance measurement and escalation processes across the company.
The “danger” of sharing your metrics
I have since come across similar patterns when working with product teams that try to become more data driven. Instead of producing physical items, their “production line” of course ships product features or updates. On one hand, more data about how those features are performing would mean they could improve their features and be more effective in their jobs. On the other hand, product teams often fear that the data could be used against them if the numbers don’t look good.
Similar to our ambitions for the shop floor tracking application, the initial intention was to learn. But instead of Build-Measure-Learn, many companies tend to replace the word learn with manage. Metrics are labelled success metrics that come with a certain expectation: if your new feature or product is not performing well, you haven’t succeeded and you’ll be in trouble.
Past experiences have made teams wary of how data will be used internally, so it’s often easier for them to share as little data as possible. If only a very small percentage of users adopt your new shiny feature, it’s easier to not have this metric showing in your company wide dashboards in the first place. The result is that teams rather keep those metrics to themselves. For important presentations, teams then carefully curate certain metrics to minimise any uncomfortable questioning.
The comments on a post from tech entrepreneur Dan McGaw stating “You can’t manage what you can’t measure” showed just how burnt some product people are from their past experiences: people were very quick to point out to be careful what to track as it could be used against you.
Dan’s statement in itself is not wrong, as ultimately the point of being more data driven is to know whether you’re doing well (or not) and to use this insight to inform future decisions. The problem lies in how companies typically treat their teams when they share those metrics and learnings.
Working with high levels of uncertainty
Building products is hard, and no matter how much research and testing you do upfront there is still a high level of uncertainty about whether something will work once you launch. See Marty Cagan’s inconvenient truths about building products (at least half of our ideas are not going to work) or this graph from Strategyzer that highlights the uncertainty product teams tend to work with, especially in the early stages:
The reason for this is that there are a large number of both internal and external factors that can have a huge impact on your success from when you first came up with an idea to when you actually launch: execution speed and quality, competitor moves, industry or regulatory changes, broader market and economic shifts to name a few.
Using metrics to learn, not to manage
My point is that even if you do all your research and test as much as you can upfront, we have to acknowledge that product teams in technology companies work in very fast paced and rather uncontrollable environments. Using metrics to punish teams when launches are not successful ultimately leads to teams becoming more risk averse, experimenting less and sharing their metrics less often which is usually the opposite of what those companies are trying to achieve.
The difference between those teams and the most data driven teams I’ve worked with is how much the company and stakeholders emphasise the importance of learning. Yes, you can’t manage what you can’t measure. And yes those insights will tell you whether a team, feature or product is doing well, or not. However, assuming the teams have followed the right process with early validation and research, companies need to provide a safe environment where learning is encouraged if they want to build a data driven culture .
In truly data driven teams that do this well, I’ve often observed the following habits or mindsets:
1️⃣ A Build-Measure-Learn instead of a Build-Measure-Manage mindset
2️⃣ Curiosity: asking “Can you help us understand?” instead of “Why didn’t this work?”
3️⃣ Showing appreciation for teams for sharing their metrics openly, good or bad
4️⃣ Reminding the teams in the day to day that the focus is on learning and improving
5️⃣ Viewing data as a way to capitalise and learn from failures
To expand further on the last point, I always love to share this quote from my interview with data leader Crystal Widjaja that I believe sums it up perfectly:
“When people make mistakes (failed experiment, failed deployment, etc), you’ve already paid that cost. We should think of data as a way to capitalize on mistakes and learn from them. Rather than “paying the tuition of the failure” and firing the individual, use the data insights to tell us WHY it was a failure, learn from it, and leverage it for the next iteration to be 10x better than the first.”
- Crystal Widjaja
Final 2 Cents
If we don’t openly share our metrics and data across the company, we are missing out on the many benefits of gathering insights to help us make better decisions and build more impactful products.
To build a truly data driven culture, we need to make sure teams feel safe to openly share their metrics and findings with us. We need to be aware of the high levels of uncertainty that product teams have to work with, assuming of course they have already done what they can to at least reduce the risk through research and early validation.
Data is ultimately not only about measuring how our production line or new feature is performing. Product metrics can help us capitalise on mistakes, increase our learnings and with that build a more data driven culture across the company — but only if we provide the right environment that encourages teams to share their metrics in the first place.
— Originally published on https://www.productandsystems.com on September 12, 2022.
Want to read more about product metrics?
The Insights Driven Product Manager is your essential guide to mastering product metrics and uncovering true data insights. The book hit #1 on Amazon’s new releases in two categories and was recognized as one of the top books for Product Managers by Amplitude!
Get the ebook
Get the paperback or Kindle
Includes real life examples, frameworks and templates on which metrics to track, how to run experiments, when to use qual vs quant data & more. 📈
