Switching to continuous deployment isn’t that difficult.The only necessary step is to deploy every successful build to production instead of the QA/test environment. But can you be sure that all successful builds will be good enough to be deployed to production? Will they cause failures and user dissatisfaction if you deliver a broken app multiple times per day?
Ensuring quality is the trickiest part of continuous deployment, which is different from continuous delivery when all successful builds are automatically deployed to production. With continuous delivery, you can deploy all successful builds to production.
However, to do that, you have to wait for the end of the day or work sprint, run a manual or automated regression testing, UAT, performance testing, and other types of tests to ensure a decent application quality before deploying to production.
The bottom line is – it’s usually not the technology or lack of infrastructure that stops us from deploying all builds to production. It’s the testing part that is lacking. Can I trust my automated tests to catch all issues without someone running all manual regression or doing exploratory testing? That’s the question that buries the hope of deploying new features to production multiple times per day.
Some teams manage to go over this blocker, but how? What do they do different? Even though some well-known companies like Google, NASA, or Amazon manage to roll out multiple application deployments daily, it’s not the size of a company that matters. As long as you have the right mindset and great risk management, you can get amazing results. So, let’s go over some of the main points.
You will not be able to catch all the possible bugs in the application, and it is OK.
Test engineers are usually perfectionists (I speak from my experience), that’s why it’s difficult for them to take a risk of issues possibly reaching end users. This approach has a hefty price tag and impacts the speed of delivery, but it’s acceptable if you deliver only once or twice per month.
The correct approach would be automating critical paths in application both from a business perspective and application reliability. Everything else can go to production without thorough testing because with continuous deployment, you can fix issues within hours or minutes.
For example, if item sorting and filtering stops working in production, users might complain, but the development team could fix this issue quickly. Would it impact business? Probably not. Would you lose a customer? Probably not.
These are the risks that should be OK to take if you can quickly fix issues in production. Of course, it all depends on the context – if you’re providing document storing services for legal investigations, it would be a good idea to have an automated test for sorting and filtering.
UI E2E tests are not the only reliable source for automated testing.
This is by far the biggest and most expensive mistake in automation I usually see. Even though they imitate user behaviour, they shouldn’t be used to test everything. If you want to make sure that price calculations work correctly (it applies to taxes, discounts, shipping, etc.), you don’t need to be logged in to the system, you don’t need to have a browser, a valid user, add items to the cart, and go through all the stages of purchase to reach the last one with final prices.
This can be tested in the backend – somewhere in the code there is a part that is responsible for calculating price, and it can be covered with unit or integration tests to run through hundreds of test cases in a matter of seconds.
You should have at least one happy path of purchase scenario in UI E2E tests covered, but any more would spend 95% of the time just repeating the same tests and would end up wasting valuable time.
Testing must adapt to the new paradigm shifts in application development and support accelerating delivery. The current approach isn’t perfect, and frankly speaking, never was. But we should still rethink our approach to managing risks by including new possibilities and looking at the big picture.
Everything from speed to business, quality, and business needs must work together to move a product forward. If these elements work separately, caring only for their individual gains, other related areas will be negatively affected.
