The Learn Step of the Lean Cycle
The third step of the lean cycle is the Learn step.
When your experiment ends, you will collect the results and retrieve the data. In the Lean Startup context, there are two possible outcomes:
The pre-defined success threshold of your experiment was achieved
The pre-defined success threshold of your experiment was not achieved
An important question is "what do I do next?"
When the outcome of your experiment is success, then you need to determine your next experiments and future improvements. When the outcome of your experiment is a failure, you have a set of options to consider for what to do next.
Learning from Success
Existing Products
When you are working on an existing product, a successful experiment gives you two options
Move on to new areas of improvement
Seek further improvements in the same area (double down on your efforts to improve certain metrics).
Let's say that an experiment generates a 5% increase in retention. Your options would be to move on to improving another area of the business (e.g. revenue) or to continue to try to improve retention to see if you can achieve an even bigger improvement with further experiments.
Validating Business Models
When you are working on validating a business model, then successful experiment results means that you really have something nice to celebrate, and you have validated "the riskiest part of the plan". Here is a lean canvas that I have created for the music streaming service Deezer.
For the sake of example, we might complete the canvas and then identify Channels as the riskiest part of the plan, but after some experiments we might validate that we can acquire online traffic profitably. In that case, we would typically regard the Channels section (marked in yellow above) as the riskiest part of the plan and ask ourselves "what is the next riskiest part of the plan". Perhaps we would identify revenue streams as the next most risky. Then our upcoming experiments should try to validate any assumptions we have made in the Revenue Streams section of our business plan (marked in green below).
Learning from failure
Although it is great when we achieve success with our lean experiments, most of the time we will experience failure. This is perfectly normal and to be expected. We are operating in terms of extreme uncertainty when we build new digital products and discovery is our goal.
Thomas Edison, inventor of the first commercially viable light bulb, took over ten thousand experiments to achieve success. We can see his attitude from his quotation below:
I have not failed 10,000 times. I have successfully found 10,000 ways that will not work.
When our experiment fails, we have several options:
Change experiment
Change hypothesis
Postpone learning
Change Experiment
As Thomas Edison has illustrated above, we cannot always just quit if an experiment doesn't work. Sometimes learning what doesn't work can be really valuable. Just because one experiment doesn't prove our hypothesis true doesn't mean that another experiment (or an improved experiment) could not have the desired results.
Our options are:
Improve the existing experiment - let's say we were using a landing page MVP and results were poor. We might instinctively know that using better design, testimonials, real imagery, current client logos, money back guarantees, videos are all options for improving the landing page. Making the landing page experiment better might be easy and might yield quick results and learning.
Choose a new experiment (to try and validate the same hypothesis) - let's say that we want to prove that we can acquire a paying customer for less than $10. After doing a number of experiments to see how many customers we can acquire with Google Ads, you might try to see what the results are with Facebook Ads. This is a new experiment that tries to validate the hypothesis "we can acquire a paying customer for less than $10 on average".
Change Hypothesis
There does come a time when we have run so many experiments without success that we conclude that maybe we don't understand the world as well as we thought. Our hypothesis should predict the future results of experiments and if it does not, then perhaps the hypothesis is not true.
Changing hypothesis is also known as a pivot. By changing your assumptions and business model, you try to prove another hypothesis. The 10 different types of pivot that we saw earlier in the course are good options to consider if you feel like changing the product/business model/hypothesis.
Postpone Learning
There are also times when you just move on. It doesn't mean giving up for good on proving a certain hypothesis true - you just aren't going to spend any more resources proving it true right now. You might move on to another element of the pirate metrics or tackle other areas of your roadmap and come back to this learning later.
Summary
When your experiements are successful, you can either seek further improvement in the same area/metric or you can move on to improving or validating some other area.
If you are validating a business model, once you have validated one part of the plan (one section of the lean canvas), then you should move on to the next "most risky" section of the lean canvas and define an experiment to validate that next section.
If your experiment failed, don't worry! Most experiments fail. Try to improve the experiment or try a different experiment that validates the same hypothesis.
At some point, if you cannot prove your hypothesis true, then you may decide that maybe your assumptions are true after all and you may develop new business assumptions (called a pivot) and try to prove those true instead.
