The concept of measurement
The mathematical physicist Lord Kelvin, famous for inventing the absolute scale of temperatures and formulating the second law of thermodynamics, remarked that:
I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.
The ability to measure the things we need in our daily lives is really important. Without effective measurement, you would not know:
How much gas is needed in your car to make a journey.
What time of day it is.
How much money you have in the bank.
How much you weigh.
Which batteries fit into the remote control for your TV.
The nature of measurement
It’s also critical for product managers to measure the impact and value of what they create. Before we look at how to do that, it’s important to keep the following in mind: all measurements are approximations. Because we use tools to measure objects, the accuracy of our measurement is limited to the precision of the tool we are using.
Let's say that you step on the scale in the morning and it reads 159 pounds. The following are all possible:
The scale is functioning perfectly, and your weight of 158.953 pounds has been rounded to 159 pounds.
The scale is not calibrated perfectly and always adds a pound to the "true weight." Therefore, your real weight is closer to 157 pounds.
The scale is extremely inaccurate. Your real weight is closer to 115 pounds.
Let's look at how to measure the effectiveness of products you will work on as a product manager.
A product is only effective if:
Customers engage with the product (because they like it).
Business targets are being met.
A business model is a conceptual model that describes the value that the business creates for customers, how it delivers this value, and how it captures value through making profits.
A business model describes how your product helps:
The organization to achieve its targets.
The customer to effectively get some "jobs done" that they care about.
Note that both of these factors are important. If you delight customers but don't help the organization to meet its targets (including making profit), then you will go out of business. If you help the business to achieve its targets but do not provide a useful service to customers, then they will soon stop using your product and profits will diminish.
Thus, product managers need to measure that:
Your product is making a contribution to business targets being met.
Customers are engaging with, and appreciating your product.
The need to measure these two factors explains why a data-driven product management approach is an effective way to align the performance of a product to the desired strategic outcomes of the organization.
How to build effective products
We apply a process called data-driven product management to help build effective products.
This approach:
Defines the success of the product in measurable outcomes (before building the product).
Prioritizes features that best achieve these outcomes.
Measures the impact of these features when built to see if these outcomes were achieved.
Improves under-performing features until they achieve the intended outcomes.
Applications of data-driven product management
Product managers often find themselves asking questions like:
Should I build this product idea?
If I build this product, which features should it have?
If I need to build a set of features, how will I know I have built them effectively?
Data-driven product management can help product managers to answer each of these questions above. Let's examine how!
1. Should I build this product feature/idea?
Data-driven product management specifies that you should build a certain feature or make a certain improvement if this feature helps the product team meet the predefined outcomes that were set as targets.
For example, if the main outcome this year is to reduce fraud, then the team may not need to build a feature for sharing content on social media but may want to analyze suspicious activity instead.
2. Which features should my product have?
Data-driven product management provides a structure that allows team members to debate which features will help reach a set of targets. The measurable outcomes are set as targets for the product team to help frame the right discussions.
If a key target for the team is to "increase signups by 10%," then improving the emails to existing users (who have already signed up) typically won't help to achieve this goal. However, a better idea may be to improve the signup page.
In order to know which features to put into the product (or the next version of the product), ask yourself questions like:
What 'jobs' are my users trying to do?
How can I help them?
Which actions do I want to users to do in the product (that help the business achieve its goals!)?
How can I encourage users to perform these actions?
3. How will we know we have built a feature well?
Data-driven product management helps you build the right thing well. A new feature does not equal success! Getting users to adopt and use features is how you should measure success.
Sometimes, a product team has a roadmap of features to build within a certain time frame. They may "build according to the plan." However, developing 10 features that no customers will use does not help the business. The focus should, therefore, be on improving key metrics.
Instead of building a feature and moving onto the next one, a data-driven product manager will know what contribution each feature should make before it is built, and will then measure the performance after it is released. For example, if you redesign the signup page and the conversion rate on that page stays flat, the data-driven product manager does not move on to the next feature. Instead, she will try to understand why the feature is under-performing and see if some small tweaks can help.
Often a major feature will not perform well initially but a small modification like changing the text on a button will make a big difference!
Summary
Measurement assigns a number to the characteristic of an object (or event).
All measurements are approximations.
Product managers need to measure that:
business targets being met
customers love the product
Data-driven product management is an approach to building products that:
defines the success of the product in measurable outcomes (before building the product).
prioritizes features that best achieve these outcomes.
measures the impact of these features when built to see if these outcomes were achieved.
improves under-performing features until they achieve the intended outcomes.
Additional resources
Intercom product blog on finding the metrics that matter.
