Building Better Products Using Quantitative & Qualitative Data

Disclosure: This was an internal post I wrote and shared with my colleagues back in 2015. While cleaning out my Confluence space, I came across it and thought that I'd share it with everyone.

There is a much more recent and polished post by my good friend Glenn Block on Mind the Product that you can read here: https://www.mindtheproduct.com/2018/01/need-quantitative-qualitative-data/

What Data Is/Not

• Data cannot compensate for bad design or replace good design, but it can inform designers.
• Data cannot compensate for listening to actual users, but it can inform the kind of user research we conduct.
• Data cannot tell us what to build, but it can inform what we should investigate.

Example

If you can visualize a funnel report where visitors are leaving a buying process during various stages of the journey, quantitative data can give you information about where you may have a problem.

In the above example, the quantitative data (QT) informs you that 36 people are falling out of the cycle during the Payment Info page/stage. 

Quantitative and Qualitative Data

QT is useful for identifying that there is a problem, but it doesn't tell you why the problem exists or specifically what the problem is.

Qualitative data (QL) is useful for telling you why there is a problem.

So it can be helpful to think:

• QT = What
• QL = Why

A Process for Using Both QT and QL

1. Identify the Biggest Problem you need to solve. You can use QT to inform you of what this may be. Look at the number of users affected, the largest opportunity loss, the highest count of critical failure points, etc. An example of a good QT tool is a funnel report.
2. Once you've identified the problem area, use QL to understand what is causing the problem, such as observational research.
3. Based on the QL, propose a solution hypothesis. For example: "We see 36% of the people falling out of the funnel. We observed that a lot people leave the site when they get to the promo code box and search online for promo codes, and then end up finding the same product offered on other sites and go purchase it there. So one proposed solution is to remove the promo code box."  Of course, you have to understand the reason why things were done in the first place before making cavalier proposals.
4. Learn & Iterate. A/B test the solution. Repeat and tweak until satisfied.
5. Return to step 1.

Another Process for Using Both QT and QL

1. Review QL data/observations to formulate a problem hypothesis. i.e. "5 customers don't like this feature."
2. Use QT to study the relevant metrics related to the hypothesis.
3. Follow steps 3-5 above.

Why invert 1 and 2? Because you might want to use QT to support or refute QL findings. Sometimes a squeaky wheel is not representative of the real problem.