When to use quantitative vs qualitative data
I’ve been a huge advocate for gathering more data about how customers are using our products to drive better product decision making. In particular, I have focussed my recent writing on how to define better metrics and gather quantitative customer behaviour data as this is one of the biggest gaps I’ve seen in Product Manager skillsets. However, I would not do my job well if I didn’t also highlight the importance of gathering qualitative data to truly understand the motivations, frustrations or fears of customers.
A common theme I’ve seen in FinTech products for example is a drop-off when the application asks for ID verification or access to previous bank account data during onboarding. This is typically done to reduce fraud risk and assess the customer’s eligibility to access certain services. The easy answer to why users may be dropping off here would be that these steps simply take additional effort — the user might not have their ID or the password to link their bank account handy.
When actually talking to customers about their experience, product teams often notice there is a lot more behind those drop-offs: the customer might wonder why this information is needed, what this data will be used for, and if they can even trust the application to handle this data securely?
Not addressing those customer fears in the onboarding process would be a big missed opportunity to improve the experience and increase the onboarding conversion. Even though I love looking at clearly presented numbers, I’ve learnt that quantitative data is not a silver bullet on its own.
In this article I want to highlight the key differences between “quant” and “qual” and provide some concrete examples of when to use which.
The difference between quant and qual
In its definition, quantitative data is a numeric quantity like an amount or measurement. It’s clearly structured data that tells you the hard numbers on what is happening. Structured data means the data follows the same format and can be easily searched, aggregated or manipulated. Qualitative data on the other hand needs interpretation to be aggregated.
A practical example of quantitative data is when you look at the customer onboarding funnel of your product as in the FinTech example I just mentioned. You will typically have a funnel report in your dashboard that shows you exactly how many users perform each step of the onboarding journey and where your largest drop-off points are. It might look something like this:
Qualitative data on the other hand describes attributes or characteristics. It can tell you why something is happening. Qualitative data is non-numerical and unstructured, such as written or verbal language. I couldn’t just put it into a spreadsheet and easily turn it into a graph, I would need to go through each sentence to then aggregate a report on the overall sentiment. While this can be automated to a certain extent for example by using Natural Language Processing, it still needs some sort of interpretation.
While the onboarding funnel report above tells you that there are big drop-offs - you might want to look into the set password step in this case - you will most likely need qualitative data to tell you why customers are dropping off at this point.
I’m saying most likely because sometimes your dashboards might show you some pretty obvious factors that could lead to drop-offs such as the app always crashing at a certain point in the onboarding. But in most cases, it won’t be as obvious and this is where we’ll need to get actual feedback from our users to understand where the friction lies.
There are many easy and effective ways to get qualitative feedback. Many teams reach out to customers who have dropped off to arrange an interview or set up automated emails, in-app messages or survey links to ask for feedback using engagement tools like Intercom or Userpilot. You can also analyse customer reviews that mention frustrations with particular parts of the product. When analysing the onboarding performance, you can now add this qualitative feedback to your numbers which will make this funnel graph a lot more insightful.
In the earlier stages of building a product, for example if you’re testing the product market fit of a new B2B offering, joining sales calls can be another effective way to gather qualitative data. You can hear first hand from customers which parts of your value proposition resonate with them, or which hesitations they may have before choosing your product.
How to determine when to use which
When building products, you will always need both qualitative and quantitative data. Depending on the lifecycle stage of your product or product initiative, you might start with one over the other, but you will always need both to get true insights from your data.
For example, in the early research of a new product or potential new feature, product teams need to establish a deep understanding of the customer pain points and specific needs. Conducting customer interviews (i.e. gathering qualitative data) will typically be the best starting point to really dive into the problem space and uncover opportunities for the product.
Once you’ve identified a clear problem to solve and you’re trying to test and narrow down different solutions that will work best for your target customer, you will want to add in quantitative data as well. Products like Netflix do this all the time, for example by A/B testing different variations of a new feature with a subset of their users before rolling out the final solution to their entire user base.
Here are some common scenarios and questions you can ask yourself when trying to decide which type of data to use:
- Are you trying to go wide? Do you want to explore a problem or opportunities where you need human insights? Start with qualitative data for example through customer interviews or joining sales calls.
- Are you trying to narrow down? You understand the problem and opportunity but need to find the best solution. Quantitative data will help you with this, for example you could test different value propositions with landing pages or A/B test different versions of a feature with a subset of your customer base.
- Are you trying to understand what is happening? Do you want to find out what customers do in your product and whether you’re doing well or not? Quantitative data will give you an unbiased view e.g. by measuring your key metrics in your analytics dashboard.
- Are you trying to understand why something is happening? Gather qualitative data to dive deeper into problems or opportunities you’ve found in your dashboards, for example by sending surveys or conducting interviews with customers who have stopped using your product.
You can also use this handy overview to remind yourself of the key differences between quantitative and qualitative data:
Final Thoughts
The key takeaway is that if you want to get more insights from your data, you will need to combine both quantitative and qualitative data effectively. Just focussing on the “hard numbers” alone will likely lead to you missing big opportunities to improve the customer experience and impact on the product and business success.
To get started, you can use the questions and the cheatsheet above to determine when to use which type of data. You’ll find that with more practise it will become second nature and you will be able to intuitively decide when to apply which.
This post uses excerpts from my ebook The Insights Driven Product Manager.
The Insights Driven Product Manager is your essential guide on how to track less and get more insights from your data to make better product decisions. It includes practical frameworks and worksheets like the “quant vs qual” overview above that you can apply to your products straight away. Download the ebook below to get access to all chapters:
