The pyramid approach to quality software


I recently read a blog post from Mike Cohn on “The Forgotten Layer of the Test Automation Pyramid”: I thought this was a great way to conceptualize the different forms of testing and their combined value.

**Unit Testing**Unit testing is your foundation for quality, it requires pragmatism and you don’t need a 100% coverage for it to make a difference. I have also found from my own experience it is not necessarily about catching bugs in the future (which it does help) but more about driving out good design and giving engineers the confidence to refactor in the future. Don’t underestimate the value in the latter point, you wrote the code so you understand it, but will someone joining the project in 6 months time have the confidence to change your code?**Integration Testing**The next layer up is integration testing. Before I go any further I want to interject at this point and briefly discuss semantics. A unit test should test the smallest possible piece of software, normally a single class, without any dependancies. An integration test should test a component or a service with its dependancies, for example a call to a remote service or a local database. I tend to reserve integration tests for when there is an infrastructure dependancy. My preference is also to differentiate between the two and put them under different source packages. The intent is different so be explicit.Back on subject. I have found integration tests to be invaluable. They provide you with a huge amount of confidence if you can test a call to a service and assert the expected result. It also helps problem determination if you can isolate the data tier of your application and run the integration test suite. There is a higher cost to integration testing though. The actual test in itself is often easier to write than a unit test, but their dependancies, set-up and tear-down introduce complexity. For example, consider a call to the database, for the test to be repeatable you need to start with a known state, when you complete you need to revert back to that state. You also find your tests are more susceptible to the order in which they run. For example, what if I have a service for creating a user, another for adding a project and third for permissions on a project. You will need to run them in that sequence if they are going to pass. Take a look at the “Cairngorm Persitence Library”: for an example of unit tests and integration tests.**Functional Testing**The top of the pyramid is functional testing. If we can automate our functional tests we have huge opportunity to quickly assert the system is working as expected and to catch bugs early. Functional testing requires a lot of manual effort and time from a Quality Engineering team. If we can automate we get a payback on that effort and are able to reduce the testing cycle. When investing in functional testing is also worth considering the customer satisfaction. I am sure we have all been in the position where we deliver a working feature only for it to break it in the next release. This often erodes confidence with the customer who can’t understand why you broke something that was previously working. If we have a repeatable mechanism for running our test suite we can apply a consistent level of testing. The downside to functional testing is typically the cost, the tooling costs (although there are opensource alternatives) and there is the investment in time. I would encourage all projects to invest in functional testing from day-1. Too many projects don’t start with automated functional testing as “they don’t have enough time”. As they reach a critical mass of functionality the value becomes apparent and they do a u-turn. At this stage you are playing catch-up and it difficult to realize the value – it’s often too late. Quality is an investment. Your velocity will be slower at the start but you will gain momentum and you will see payback in terms of effort, a reduction in defects, higher quality and ultimately a happier customer.**Summary**High quality comes from a strategy that combines unit testing, integration testing and functional testing. Doing one on its own doesn’t assure quality. 100% test coverage comes the aggregate.

Comments are closed.