Climb to the middle and the top of the testing pyramid
Where’s a great place to get a fantastic soy vanilla latte? How would you decide if your local coffee shop produces one that’s up to your standards? To be confident that their soy milk is up to scratch, you might ask to try some. You might also taste a little of their vanilla syrup. And their espresso, of course!
The quality of the individual ingredients matters, but does trying all the separate ingredients of a latte really tell you if the whole latte is any good? Of course not! We’d have to actually try the final product, which brings all those pieces together.
When checking if our app works, we’ve got to be just as thorough. So far, we’ve checked individual parts, but that means we’ve made a lot of guesses about how all those parts fit together; especially by using mocks wherever classes would normally collaborate. Now, we need to be sure that it continues to behave as expected as we put the code together.
Fortunately, by bringing our code together in different ways, we can test it for different things - and this is a job for tests a little bit further up the pyramid! Here is an overview of the types of tests that bring code together in more complex tests:
Pyramid | Test | Question being answered | Example |
Middle | Component integration tests | Do our unit tested classes actually play well together? | We’d write a JUnit test which checks that our CalculatorService can call methods on our ConversionCalculator, and to check representative happy and sad paths between these two classes. |
Middle | System integration tests (SIT) | How can we quickly test that our running application would collaborate with the outside world? |
|
Top | End-to- end user journey testing | Integration between our running app and a real user achieving real outcomes on the site. | Test that a user can log into our system, select a type of calculation, enter some values, and correctly convert them. |
You can see that each of these has very specific uses, that can help in different ways. Make sure you ask yourself: What test do I really need here? In the next few chapters, we’ll go over how to write and implement each of these tests in turn.
Acceptance tests
Hold on! I told you about acceptance tests which describe the business requirements.
Sure, I remember...but do you have an example? 😇
Okay, what about an acceptance test for a website with registered users? It might be something like: "logged in users should be able to delete their profiles so they can unregister from our site." This involves testing that a user can be logged in and use the application to unregister. Subsequently, we can assert that they are no longer registered users. This the sort of thing which would be decided when viewing your software as a commercial product. It's not directly testing any components, so much as our business' expectations. Acceptance tests are written in business language, but you implement them using the other types of automatic tests you've already seen.
As you can imagine, acceptance tests (also called business acceptance tests) are a source of great confusion among many engineers. Are these system integration tests? Yes. Are these end-to-end tests? Yes. What about unit tests? There are many opinions based on differing patterns you’ll see in the industry. So, where do acceptance tests live?
Traditionally, (manual) acceptance tests were the last thing tested, and always at the top of the pyramid against a running system. That was then. Some think that this is where they belong in the automated world, but manual tests at the end are dog slow. That's why many teams choose to write acceptance tests which run close to the system integration tier of the testing pyramid. Modern frameworks, such as Spring and Cucumber, make it easier to write them this way.
I don't get it. There isn't a fully running system at the integration tier! That means that we’re still making guesses about the user. Shouldn't we only use end-to-end tests? 🤔
To quote John Fergusson Smart, author of BDD in Action,
“The provenance of a test actually has little bearing on how it is implemented. For example, only a small part of the business focused acceptance tests would be implemented as end-to-end UI tests (the ones that illustrate typical user journeys through the system).”
Basically, acceptance tests can be implemented in more than one way. You don't have to use end- to-end tests exclusively. In fact, it's unlikely that you will. Martin Fowler, a founder of the Agile movement (who is far more of a proponent of the testing pyramid than John), clarifies this further, describing acceptance tests sitting across the testing pyramid:
“It's good to understand that there's technically no need to write acceptance tests at the highest level of your test pyramid. If your application design and your scenario at hand permits that you write an acceptance test at a lower level, go for it. Having a low-level test is better than having a high-level test. “
In other words, you need some tests that take your business level test-cases and run them as fast as possible. This way, you can have fast feedback you’re confident in. These tests might not be on the same level of the pyramid. They’ll probably have a bunk-bed between two levels. Our table can be completed with:
Pyramid | Test | Question being answered | Example |
Middle | Component integration tests | Do our unit tested classes actually play well together? | We’d write a JUnit test which tests that our CalculatorService can call methods on our ConversionCalculator, to check representative happy and sad paths between these two classes. |
Middle | System integration tests (SIT) | How can we quickly test that our running application would collaborate with the outside world? |
|
Between middle (SIT) & Top | Acceptance tests | Integration between business requirements and the needs of real users. | A test describing business requirements, not components. Implementation will vary. |
Top | End-to-end user journey testing | Integration between our running app and a real user achieving real outcomes on the site. | Test that a user can log in to our system, select a type of calculation, enter some values, and correctly convert them. |
But how do we decide when and where we actually need acceptance tests?
There isn’t a right answer. Here’s how I’d recommend you decide:
Ask whether the business behavior already has an end-to-end test which will use the methods and classes you’d need to deliver your acceptance test.
If the answer is no, you might need to write one.
After this, write all your business acceptance tests at the system integration level with as much realism as you can afford without slowing down your test. By realism, I mean real collaborators instead of test doubles. The reason for this is that having a representative test at the E2E level gives you the confidence to write faster, partially mocked, tests. It’s also better if all your acceptance tests are in the same place in your source code. This makes it easier to find them when trying to grow or understand your application.
What’s beyond the top of the pyramid?
If you were to pull out your telescope and gaze above the top of your pyramid, you’d be blinded by the chaos of the real world. For the most part, the pyramid tests predictable scenarios. As you hit the top and go into the world, it is only then that you really know how your product behaves. Fortunately, you’ll have a lot of confidence by the time you reach this point.
This doesn’t mean that you’re done with testing or can stop there. So what can you go on to test and learn and sense in the world? Check out these tests:
Test | Question being answered | Example |
Beta | What happens when I put a version of my app in front of real users? | If I release my calculator out to some real students at the local school, is it performant and usable? |
A/B testing | I have two great ideas for this UI or a complicated bit of code. Which one should I use? | You might release two versions of your application. One which sends all calculations to Google, and one which solves them itself. Compare which is better and more loved by your users. |
Smoke | Integration between your running application (all the code) and the platform it runs on. | The application is running on your production platform. Fast tests check that its configurations are correct and it seems to be handling a happy path or two. |
Why should you care about testing after your code is released? Building part of a system is your contribution to solving a bigger problem. A question you need to keep asking is what problem am I solving? As you solve that problem, you need to keep asking if you’re building the right thing. That is, are you correctly solving that problem in a way which works for your users, your business, and other software in the real world?
So, what does all this mean for our pyramid?
Here’s the final pyramid. It hasn’t really changed. You should just have a better feel for the types of things you’re trying to prove in your unit, integration, and end-to-end tests. Also, remember that testing is an activity you can continue and bring into the world.
This final pyramid still doesn't show everything and answer all questions:
Where on this pyramid would you put tests to check that your software is fast enough (performance testing)?
Or tests which make sure that it doesn’t fall over when a lot of people use it with a lot of data (load testing)?
What about tests that check if someone can exploit your software and gain access to data they shouldn’t (penetration testing)?
There are clearly many more types of tests you can do. What you actually choose to test beyond your units should come down to the question of what else you need to have the desired level of confidence that your system behaves as it should. In general, most good projects should have at least unit, component integration, and end-to-end tests. Anything you do above and beyond comes down to your business’ need for confidence.
Last thing: Tests don’t come for free. Remember that every test is like a puppy. 🐕 You need to commit to caring for it and invest time in giving it the on-going love it needs. Does your codebase have space to house another puppy? “Woof” said the test.
Let's recap!
For unit tests to be fast and thorough, they make many assumptions. Each mock is itself a guess about a collaborator.
Component integration tests validate that a small number of units can collaborate.
System integration tests validate that your units can collaborate in a partially running system. This is a compromise on the slower end-to-end tests.
With end-to-end tests, you run the whole system and validate your assumptions.
Acceptance tests run perpendicular to the pyramid and can cross layers.
