Photo by Hans Reniers
Sadly, most teams never measure the value of what they create, and thus they continue to invest in building those aspects that aren’t delivering value. Measuring value allows the product owner to direct the team’s focus toward the 20 percent of features that hold most of the value.
The first step in measuring the value that we are producing is moving from assumptions to hypotheses. An assumption is something that we believe based on limited evidence. A scrum team might capture their assumptions like this:
As a [type of stakeholder],
I want [what we believe the stakeholder wants]
So that [some value is created].
When developing a hypothesis, we want to ask ourselves, “How will we know our stakeholder got that value?” We want to identify measurable criteria that will answer that question. We can add this criteria as a fourth line to our user story:
As a [type of stakeholder],
I want [what we believe the stakeholder wants]
So that [some value is created].
This will be confirmed when [criteria that will confirm the value].
This last line is not an acceptance criteria though there are similarities. Acceptance criteria let us know when the team has built what was intended, that they created the expected output. The fourth line above tests whether that output has created the desired outcome, the actual value.
Example Product Hypothesis
Photo by Saad Khan
One of their user stories is:
As a businessperson,
I want a self-serve business center with printing, copying, and faxing capabilities,
so that I can get a little work done without leaving the resort.
We can turn this into a testable hypothesis by adding one more line.
As a businessperson,
I want a self-serve business center with printing, copying, and faxing capabilities,
so that I can get a little work done without leaving the resort.
This will be confirmed when at least ten guests do printing, copying, or faxing each week during the first month that the self-serve business center is open.
This additional line gives us a way to measure the value of what the team has built. The hypothesis statement should include:
- The data we will collect
- How long the experiment will last
- The expected result from the experiment
The data we gather will inform decisions about what to build in the future. If we get the predicted result:
At least ten guests do printing, copying, or faxing each week during the first month that the business center is open
then we can feel justified in continuing to invest in this area of our product.
But what if we build a self-serve business center and nobody uses it? Should we still move forward with upgrading it to full service? Probably not! The data doesn’t support our original belief that our customers want to work while on vacation.
Techniques To Measure Value
Here are some approaches to measuring the value of what the scrum team has built. Any of these can be used as the means to test a product hypothesis.
A/B Testing
A/B testing involves directing some users to the new version of what the team has built and the rest of the users to the pre-existing version. We then measure how the two groups behave with regard to whatever value our product hypothesis is testing. For example, we might be looking for 20% more users to make a purchase when they get the new page vs. the pre-existing page.
Learn More About A/B Testing
- What Is A/B Testing
- Wikipedia Entry on A/B testing
- A Refresher on A/B Testing from Harvard Business Review
Multivariate Testing
Multivariate testing is A/B testing made really complicated. Instead of testing one change, we are testing a combination of changes, trying to find the best combination. In a website, we might be experimenting with page layout, images, and ‘Buy Now!’ button design all at the same time.
Learn More About Multivariate Testing
- Multivariate Testing: What, Why, How
- Wikipedia Entry on multivariate testing
- Video: A/B & Multivariate Testing
Usage Tracking
When we implement a new functionality in our product, we add the ability to track how often it gets used. For example, if we added an AI-enhanced grammar checker to a word processor, we could track how often users activated the feature and how often they accepted the suggestions from the grammar checker.
Learn More About Usage Tracking
- Software Usage Analytics Guide for Product Managers
- What Is Product Usage Analytics And How To Implement It
Web Analytics
Web analytics is a specific type of usage tracking for websites. It enables us to see what parts of the site are most visited, how people navigate around the site, where they enter our site, and where they leave it.
Learn More About Web Analytics
Usability Testing
Usability testing involves watching users as they try to accomplish a task using our product. It enables us to understand how they interact with the product, what they expect, and where they get stuck.
Learn More About Usability Testing
User Interviews
A simple way to know what parts of your product are valued by users is to ask them. After deploying a new feature, you can ask users if they have used it, how they have used it, what they like/dislike about it, and so on.
Learn More About User Interviews