Test Quality in CI/CD – Expert Roundup

Home/Blog/Test Quality in CI/CD – Expert Roundup

Test Quality in CI/CD – Expert Roundup

Bar KofmanDigital Marketing Manager | December 19, 2017

Software is everywhere! Gaining a competitive edge is based on the ability to develop and release software quickly. Adoption of CI/CD addresses exactly this with both technology-oriented change and process and organizational change.
Technologies like CI/CD, microservices, feature toggles, test automation and processes such as shifting left, shortening the feedback loop, fail fast, distributed and autonomous teams, no-QA CoE enable to move faster. The main capabilities teams should adopt to overcome the bottleneck are:

  1. Development of automated tests across the entire application beyond unit tests. The selected test frameworks and automated tools must provide stable tests, that are not easily broken.
  2. Visibility and insight on what code and features are actually being tested. Pushing untested code towards production is a sure path to decrease in speed, as issues in production are the number one reason for holding teams back from releasing software quickly.
  3. Optimizing regression test suite, for maximum coverage while shortest execution time.
  4. Consolidated view of all quality activities across different teams, environments, and technologies.
  5. Ability to prioritize quality activities based on data to mitigate risk when tradeoffs are required.

Adopting the right set of tools, processes, and mindset is the path to maintain high quality while increasing speed in a CI/CD environment.

Avishai Shafir
Contributed by Avishai Shafir, SeaLights

What should CD testing look like? If you ask me, it all comes down to a properly integrated test strategy.
That strategy is having both developers and testers know who tests what so that there are no gaping holes in the overall test coverage (to prevent quality leakage) and no double work is done (to prevent unnecessary wasting of time). Only once you have a clear idea on who tests what in the CD process, then you can decide whether or not to automate the accompanying tests and on what level (unit, integration, end-to-end). Creating a well-functioning integrated test strategy requires adaptation from both testers and developers:

  • Developers should become even more aware of the importance of testing and start looking beyond plain happy-flow testing in their unit and integration tests. This removes a lot of potential defects that are otherwise detected later on in the process, if at all… If you’re a developer, please do read this article on Simple Programmer to see what I’m trying to get at.
  • Testers should start getting closer to developers, partly to better understand what they are doing, partly to help them with refining and improving their testing skills. This might require you to get comfortable reading and reviewing code, and if you’re into test automation or willing to become so, to start learning to write some code yourself.

Link to full article: https://www.ontestautomation.com/first-steps-into-testing-in-a-continuous-delivery-setting/

Bar Kofman
Contributed by Bas Dijkstra, OnTestAutomation ,

A year or so ago, there was a tweet that appeared on my timeline multiple times across many weeks…or maybe it was months…I’ve slept since then. Clearly, my memory fails me on the specifics. What I do remember is that the question posed by the tweet was something like the following: is automation sufficient without a strategy? When discussing strategy, I like to ask two broad questions:

  • What do we want to accomplish?
  • How do we plan to accomplish it?

The “Whats“—these are our goals. Every organization has business goals. Clearly, product development must be in line with the goals, meaning testing goals must be in line with them as well. For instance, are we:

  • trying to reduce defects in a product?
  • trying to release faster or more often?
  • adding support for a new operating system to a product?
  • trying to gain market share?

Our testing strategy must be consistent with these goals, thus our automation goals must be as well. If they are not, we’ll just be doing the wrong thing faster. Having a good relationship with our extended audience, in this case, “the business”, can help us have a positive impact on achieving these goals. The “Hows“—these are our methods.

  • Do we need a small, fast, shallow smoke suite of tests that are executed on each deploy to each environment?
  • Are there some areas of our product that are difficult to test?
  • Could humans benefit from applying technology to these problems?

Regardless of the methods we choose, we must be sure they are in line with our goals. It’s important to note that the thoughts expressed above are not automation-focused; they focus on business strategies and testing. This focus is intentionally away from automation specifics to help ensure that automation is supporting our goals; automating is not a goal, it’s only a means to an end. So, back to the question of “is automation sufficient without a strategy?”; clearly, my opinion is “no, it is certainly
not sufficient”. Link to the full article: https://responsibleautomation.wordpress.com/2017/01/11/your-first-move-automation-strategy/

Bar Kofman
Contributed by Paul Grizzaffi, Principal Automation Architect, Magenic. Author of Responsible Automation,

Joe: How does testing fit into a continuous delivery? Is it an important piece that you need to test before a build is promoted to the next stage and before it’s delivered to users? How does that process work?

Sahas: Testing/the whole quality is absolutely an important aspect. I strongly believe in having a high-quality product, and I strongly believe in some of the practices associated with Agile quality.  The world has moved on. We were shipping products once every six months, maybe, once a year, once every two years. Now, everybody is thinking about shipping the product in a reduced cycle time. Ship it once in two weeks, ship it once in four weeks, ship it once in a month kind of thing.

Everything changed around that. The business needs changing, so now, it’s not only the development that has to shift gears towards Agile. Everything has to shift gears towards delivering that product, which means quality aspects also have to change. At a high level, I’m a big believer in Agile quality. How would we fit,
in a quality mindset, into the product-building lifecycle.

Regardless of what you do, you have to build quality in. You shouldn’t be thinking about quality as an after-the-fact, which unfortunately has been the way we’ve worked so far. We worked first to get the requirements out and we designed, I did some coding, and finally, we will say, “Okay. I’m ready to bring in some testers. Let’s start doing testing.” If we want to deliver product continuously, we have to continuously evaluate the quality at every stage. Testing throughout the cycle is a critical part of building quality, and fitting quality into continuous delivery.

We need to engage the quality evangelists, who think from a quality perspective, to test our understanding. The process goes like this: here is the feedback that we got from the customer and here’s what we built. We ask did we build what he wanted, instead of taking the tester directly into a screen and saying, “here’s my screen. Go and test its functionality.” Instead of checking the functionality, we need to check our understanding: this is the second aspect. Did we understand the problem correctly? Did we build the right solution, the right thing, the right way?

The third aspect we focus on is finding bugs. However, in a shorter cycle if I’m finding a bug, how likely is it that I would go and fix the bug before it goes into my customer’s hand? Instead of finding a bug, pair with a developer, be a part of the whole thing. Pair with your product owner. Try to understand what your customer really wanted, and focus on preventing bugs. Employ techniques, for example, check with the developer while he’s designing, while he’s coding, and try to influence the developer to think like you from a quality perspective. Hence, to cover more areas while the developer is on the line of the code.

In some places, quality used to be a separate organization. We might create a quality department or outsource it somewhere to some country, and QAs and testers and test automation experts would test the quality. We have to move away from that, though, and the whole team has to open to quality. As the developer, I write the code. But I wouldn’t call to the other side of the window and say, “Hey, QA, go and test for me.”

We, as a team, have to own quality. We as a team have to, for example, think about quality aspects while we’re thinking about design aspects. We also have to think about what is the expected load on this particular service? How do I verify that particular character that my product
owner wants? You’ve got to go back to a product owner and speak with them, “You’re asking me to build this new service. What if this comes up in the Super Bowl ad? What if I get 30,000 hits in three seconds? How do you expect the service to react?”. It has to be more collaborative. The whole team owning quality in order to deliver what would delight our customer: that’s the fourth aspect.

The fifth aspect is not automating absolutely everything. Instead, of automating everything, I believe you cannot really get rid of manual verification. You cannot really clone your brain, the way your brain thinks, the way the product owner would see the product, the way your customer, the way millions of customers would see the product. Rather, you should use automation for its strengths. This means automating in a more rationalized fashion.

We have to think more in-depth. As a team, why do we want to repeat something if that has been automated at the unit test level. If something has to be automated at the unit test level, why should we automate at the GUI level? If something can be automated at the web service level, why would you automate that at the GUI level? We have to take some sort of rationalized approach even to automate things. That’s the fifth aspect. I call it rationalized automation.

I would say, wherever I go, wherever I try to implement automation, wherever I try to implement quality, we would bank on these five things.

Link to the full podcast/transcript: https://www.joecolantonio.com/2015/10/15/episode-73-continuous-delivery-automation-podcast/

Bar Kofman
Contributed by Joe Colantonio, founder, TestTalks. Author of the UFT API Testing Manifesto (interviewing Sahaswaranamam Subramanian) ,

Many applications start out simple. While trying to find a market fit and iterate around the product, the code base is still small and full of hacks that are used just to ensure things work. The tests are still narrow and on point but not exactly tuned for speed. After all, the overall build is still in the realm of minutes.

Once the application and product start to gain traction, things get more complex. The test suites start accumulating cruft. Builds start taking ten, twenty, or thirty minutes. After a couple of years, it takes hours to run a full build. At this point, there’s a unit test suite, integration tests, and user interface tests. Together they take significant time to run, slowly but surely reducing development velocity and seriously hampering a team’s ability to ship fast.

Test complexity is a silent productivity killer that’s hard to pinpoint when it reaches a critical mass. The increasing test complexity impacts velocity and makes the overall build more brittle and prone to breaking due to only small changes.

Test quality takes on a different meaning at this point. The tests themselves might be great, the code base well covered, but the sheer complexity starts getting in the way. At that point, teams will find their ability to continuously deliver value to the customer is hampered.

It seems inevitable that a team reaches this point when they strive to maintain good test coverage while also trying to avoid the increased complexity of microservices. Yet microservices also offer an answer to the problem these teams are now facing. They allow breaking out specific functionality into separate applications with separate test suites. Smaller pieces allow for test suites to stay small, which brings back the velocity that the team was used to in the early days

Microservices come at the price of increasing complexity in managing them at runtime as well as making sure that the services, and the teams building them, communicate well with each other. But for the ability to ship faster that complexity can be a worthy trade-off. It helps to avoid one of the biggest test quality issues with continuous integration delivery: a complex test suite and a slow build.

Bar Kofman
Contributed by Mathias Meyer, CEO, Travis CI,

When you work on free software, what you get is a wild diversity of tools, preferences, languages, personalities. Testing in Ubuntu is about combining all those into a consistent, happy family, to build an operating system that is at the same time usable, useful, cool and transparent for our community. The challenge is enormous, but the opportunities that this diverse and distributed community present are even bigger.

The role of our community of upstream developers is to develop software giving their best efforts. The role of our users is not to remain passive, but to start owning their technology by contributing with their feedback, their ideas, requirements and maybe even their time and abilities. Our role is to make sure that the process for developers to get a new stable version published and ready to install is super simple, from commit to millions of users in a snap. We try very hard to make this enjoyable and fun for everybody, including us.

After 10 years of developing Ubuntu, learning from our community and adjusting to the new projects that appear every day, what we now know is that to release with high quality we all need to be flexible. This means adapting ourselves to the processes of the projects that build the different parts of our system, injecting ourselves in those projects with a very low-maintenance method that will get the project compiled, packaged and installable, and helping those projects complete automated user acceptance testing and crowd testing with our community of early adopters.

We now have build.snapcraft.io, a secure build farm that will build software and put it in the Ubuntu store, in the edge channel, every time that a new commit lands to the master branch. In this channel, it’s not visible to all the users, only to the early adopters who want to contribute testing. From here, each project can choose any path they want to assure the quality of their software, usually with a combination of automated and manual tests, running sometimes in isolation, sometimes in integration with the other projects that are following the same journey to stable. Once the testing is done, the project is moved to stable where it is visible to millions of users. We encourage the developers to be welcoming to new contributors, and we encourage our community to help us make an amazing, free and open operating system.

Leo Arias
Contributed by Leo Arias, snapcraft developer at Ubuntu,

Don’t leave failing tests unattended—everyone knows that tests can fail. Sometimes, fixing a particular test is a hard job and we leave it failing (but promise to ourselves that we’ll fix it later). The result is often the same: configuration becomes red and everyone starts to ignore any new failed tests in this configuration, which ultimately eliminates the effect of your tests. So, your tests are run, but the result is generally ignored.

To solve the problem, there must be a certain level of discipline in the team. If a test is failing and it’s temporary okay, you can skip these tests (i.e. throw SkipException in a code) or have the tool to mute the test, so it doesn’t impact the real situation. Don’t forget to leave a proper comment.

Another important thing is to review builds from time to time to locate tests that started to fail. Usually, a single build doesn’t contain lots of changes and it’s possible to identify who broke the test or the whole build. It’s a good practice to review builds with your changes to identify whether you’ve broken any test.

CI/CD tools must provide a way to notify that a build has failed. In the simplest case, it’s just notification via email or IM. A more advanced case is when everyone can see that someone is taking care of the failing test or build (for example, investigations in TeamCity).

So, the bottom line is: without a certain level of discipline, the efficiency of tests significantly drops. CI/CD tools can help to locate the reason and provide current status to others, but it cannot fix tests for you.

Sergey Pak
Contributed by Sergey Pak, Developer, JetBrains

The CI/CD pipeline needs to incorporate functional requirements, performance, and security, with valid test results and reports.  As the software development world has moved towards using various sophisticated methodologies, CI/CD has become an important part of development.

Testing is added to the CI/CD pipeline to ensure that the software delivered is stable and 99% bug-free. Reporting of issues that occur during testing in CI/CD is key. The main reason to adhere to the CI/CD approach is to save time and have the whole deployment process automated. Most of the CI/CD processes tend to happen at night and we need to have full reports on how the tests have run, which tests failed, and why they failed.

CI/CD should definitely include performance and security testing even though such tests don’t need to run as frequently as the functional tests. However, performance and security testing should be a part of the pipeline that runs roughly once every two sprints. JMeter/Blazemeter are two good performance testing tools, which can both be useful. The JMeter “Performance Plugin” provided by Jenkins can help to understand performance trends. JMeter also provides sophisticated reports which base performance tests on the specifications inputted by you that detail how well you expect the system to work. These reports can give you a detailed look and feel of the performance of the system.

JMeter 3 now works with Selenium Grid to help understand how the software performs when accessed by various systems from various browsers. With the help of Maven and Jenkins, these tests can now be part of your Continuous Deployment, and the end reports also include APDEX (Application Performance Index). With the help of Selenium Grid, JMeter can achieve real results of how the system works when accessed by various different browsers concurrently. My talk will run you through on how these challenges can be achieved from the very beginning so that you can avoid unexpected load clashes when hitting the brim.

When talking about functional testing, obtaining screenshots and screencasts are very helpful to visualize when your tests have failed in the CI/CD pipeline. Using Extent Report with the help of screenshots and screencasts can help you gather important information and understand the failure and successes of the systems.

Security can be added to your CI/CD pipeline with the help of OWSAP’s own ZAP API at a very early stage so that you can find the security glitches in your system and fix them as early as possible.

CI/CD provide a vast set of practices and processes that you can use to get the most out of software testing—whether functional, UI testing, API testing, performance testing, or security testing, all can be a part of your CI/CD pipeline to help release reliable, solid software.

Christina Thalayasingam
Contributed by Christina Thalayasingam, Zaizi,

Agile methodologies teach the breaking down of software development into smaller tasks known as “User Stories”. This enables early feedback which is useful to align software features with market needs.

With the widespread adoption of agile practices, teams are able to deliver functional software in smaller iterations.

Continuous Integration (CI) is the practice of checking in code regularly. Each feature is integrated and tested several times a day from a shared codebase. Though it gave a push for many smaller and frequent releases, test deployment and releases became strained, which ultimately affected the end goal.

Jez Humble’s breakthrough book, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation, talks about treating the entire software lifecycle as a single process—and one that could be automated. While agile addressed the needs of developers, an investment in DevOps initiatives and Continuous Delivery offers businesses much higher returns.

How to do automation right?

When selecting the right processes to automate, always ask yourself, “Does this need to be automated now?” The following checklist will give you the answer:

  1. How frequently is the process or scenario repeated?
  2. How long is the process?
  3. What people and resource dependencies are involved in the process? Are they causing delays in CI/CD?
  4. Is the process error-prone if it is not automated?

CI/CD is not just about the tools! If you’re looking at the tools without thinking about the people, processes, and the company structure, your CI/CD is never going to succeed. Implement a process using the tools, not the other way around. It’s crucial to understand the process and combined requirements of the organization and then choose the right set of tools to fulfill technical requirements.

Seamless coordination between CI and CD

CI feeds CD. The toughest aspect of CI/CD is the human factor involving collaboration between various teams – development team, quality assurance team, operations team, scrum masters etc. Collaboration and communication cannot be automated. To measure the level of coordination, benchmark your CI/CD processes against the best in the business.

Keep the goal in sight.

Design a meaningful dashboard by assessing what data everyone wants and establish a standard narrative for what that data means. Do not obsess over substance at the expense of appearance. Progressive assessment is important before metrics and dashboards. CI/CD is ultimately essential because it meets business goals. Failed releases lead to customer dissatisfaction and lower revenues.

CI/CD is not possible without continuous testing. In order to build quality software, we need to adopt different types of tests—both manual and automated—continually throughout the delivery process. CloudQA automates your regression testing, cross-browser testing, load testing, continuous monitoring, and seamlessly fits into your CI/CD pipeline by providing out of the box integration with CI/CD tools like Jenkins, CircleCI, TravisCI, JIRA etc.”

Arun Kulkarni
Contributed by Arun Kulkarni, CloudQA,

Up until the last few years, Agile really stopped at the custom app development team’s door. But now organizations realize that they need to apply the same principles to their mission-critical enterprise applications that run the majority of their business (e.g. ERP, Finance, Supply Chain, HR, etc). Organizations are looking at how to make Agile and DevOps work across this bigger landscape.

CI/CD systems require effective automation that can be run on-demand for continuous testing as part of the overall continuous delivery process. This is much more challenging in the enterprise application space where you are looking at end-to-end tests that can have 1,000s of steps and traverse multiple systems (SAP, Salesforce.com, Outlook, etc.). A 2017-18 World Quality report cites that just 16 percent of end-to-end business scenarios are executed with test tools. The majority of end-to-end tests are still being done manually and many times by business users/domain experts.

This brings up another core challenge when it comes to implementing CI/CD across enterprise applications. Transferring the knowledge needed from domain experts to automation specialists can take months. The more companies try to work around these legacy systems by developing customized surrounding apps, the more the problem compounds. Customized apps still need to be tested against the supporting business processes, and all gains achieved on the custom dev side from the Agile process are quickly lost.

The challenges of end-to-end business process testing don’t end with the availability of automated tests. Running tests as part of a CI cycle also presents a number of challenges when it comes to scheduling. Unlike custom apps for which virtual DevTest environments that can be spun up in minutes, tests for systems like SAP need to be run many times in pre-production where there is limited availability. Executing UI-driven tests at scale also requires an active user session which means installing local agents and logging into machines. All this adds to the complexity.

Lastly, UIs are also evolving. SAP Fiori is a great example of where tools like Selenium fall short. Being able to create a continuous testing strategy that includes UI and end-to-end business tests is critical to delivering a CI/CD strategy for mission-critical enterprise applications. Worksoft understands this better than anyone. For more on how to build automated tests for mission-critical processes go to https://www.worksoft.com/products/worksoft-certify.

Shoeb Javed
Contributed by Shoeb Javed, CTO, Worksoft,

Software testing is usually performed as an isolated and independent activity confined within an allocated time span and set of resources. Once the development is done, testers are expected to find any issues and report them back for fixes. This process might have worked decades ago. However, in the internet age, the landscape is much different.

Application complexity and size coupled with the pace of change makes the traditional method of testing not only inefficient but highly ineffective for delivering a quality product. Particularly with larger applications, it is impossible for testing to be performed in a confined time and space with an expectation of decent coverage.

If testing cannot be done just during the testing phase, how else can it be done? Quality was always supposed to be considered as not a specific group’s problem, but something every person related to the product should be responsible for. In theory, it sounds great but the practical implications of this idea were very limited.

Continuous testing is just that. Instead of confining the testing process to a group and time, it’s done by everyone and all the time. In some way, every person is involved in fueling the testing efforts directly with tangible and measurable results, making quality everyone’s daily focus. Sales, marketing, support, development and product management – all contribute in providing market insights, customer feedback on what’s working, and technical help – together creating a continuous testing cycle and great feedback loop to continuously test, measure and refine the product’s quality efforts.

Ali Khalid
Contributed by Ali Khalid,  Quality Spectrum LLC,

Test automation is an essential part of CI/CD, but it must be extremely robust. Unfortunately, tests running in live environments (integration and end-to-end) often suffer rare but pesky “interruptions” that, if unhandled, will cause tests to fail.

These interruptions could be network blips, web pages not fully loaded, temporarily downed services, or any environment issues unrelated to product bugs. Interruptive failures are problematic because they (a) are intermittent and thus difficult to pinpoint, (b) waste engineering time, (c) potentially hide real failures, and (d) cast doubt over process/product quality.

CI/CD magnifies even the rarest issues. If an interruption has only a 1% chance of happening during a test, then considering binomial probabilities, there is a 63% chance it will happen after 100 tests, and a 99% chance it will happen after 500 tests. Keep in mind that it is not uncommon for thousands of tests to run daily in CI – Google Guava had over 286K tests back in July 2012!

It is impossible to completely avoid interruptions – they will happen. Therefore, it is imperative to handle interruptions at multiple layers:

  1. Secure the platform upon which the tests run. Make sure system performance is healthy and those network connections are stable.
  2. Add failover logic to the automated tests. Any time an interruption happens, catch it as close to its source as possible, pause briefly, and retry the operation(s). Do not catch any type of error: pinpoint specific interruption signatures to avoid false positives. Build failover logic into the framework rather than implementing it for one-off cases. Aspect-oriented programming can help here tremendously. Repeating failed tests in their entirety also works and may be easier to implement but takes much more time to run.
  3. Third, log any interruptions and recovery attempts as warnings. Do not neglect to report them because they could indicate legitimate problems, especially if patterns appear. It may be difficult to differentiate interruptions from legitimate bugs. Or, certain retry attempts might take too long to be practical. When in doubt, just fail the test – that’s the safer approach.

Andrew Knight
Contributed By Andrew Knight, LexisNexis,

A couple of years ago I worked on a big product where we used Specification by Example (Behaviour Driven Development) extensively. There were 5 developers on the team, a product owner, and a business analyst. We worked for several years on the product. We started to use Specification by Example to solve communication problems between the product owner and the developers – we needed a way to bridge the communication gap, and Specification by Example proved very effective.

After a while, we began to automate our scenarios, creating an executable specification. We added a Living Documentation (that’s what started my involvement with Pickles, the open source Living Documentation generator) and integrated the results of the automated test runs into that Living Documentation. We had a pretty cool automated build with virtual machines where we deployed the software and ran our battery of automated scenarios. Productivity reached an all-time high. The number of user stories that were rejected by the product owner at the end of the iteration became zero.

Gradually, problems started to appear in our setup. We simply had too many scenarios: we began to focus on quantity of scenarios, not on the quality. The scenarios became more technical and less easy to read, so they lost their power to explain the workings of the system. The scenarios took a long time to run, so the running time for the whole suite increased to several hours. Due to timeouts, on average, 0.5 percent of the scenarios might fail – but we had 400 scenarios so there was a failure in every run. The value of our automated verification setup decreased severely.

What I learned from this: when doing automated scenario verification, focus on quality and not on quantity. If you want lots of tests, write good unit tests that run in the blink of an eye. But for your integration tests, or end-to-end tests, or scenario verifications, pick a small set of important scenarios and make sure they run reliably and reasonably fast. That way you will get the most value from those tests in the long run.

Dirk Rombauts
Contributed by Dirk Rombauts, Pickles,
*The open source Living Documentation generator

3 Barriers to Continuous Delivery

Continuous delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. Sounds pretty great, right? New features can be added quickly, and there is always a usable release ready to go. So why are only 65% of organizations implementing continuous delivery? Studies show that the barriers for companies moving to implement continuous delivery are integrating automation technologies, people, and collaboration platforms.

Integrating automation technologies

Change always comes with growing pains. Integrating new technologies with legacy software can be challenging, but getting to continuous delivery, releasing better software faster, is worth the challenge.


The race for IT talent is fierce, and finding the right fit for your team is a challenge. Continuous delivery is cutting-edge technology and there’s naturally a limited number of people with experience.

Collaboration Platforms

Because continuous delivery aims to streamline the development process, it means your collaboration platform needs to be able to keep up with the flow of information between teams.

These barriers look pretty intimidating when you begin the process of continuous delivery, but having the people, process, and tools to help you release the best software for your customers is the key to success.

Sarah Burns
Contributed by Sarah Burns, Marketing Specialist, Celtic Testing Experts,

CI/CD is like a chain, and testing is one of its essential links. There are many types of testing and one of them is performance testing. If we want to be sure that our application meets its SLA requirements, we have to run performance tests with every build. A good practice is to automate these tests and put them into our CI pipeline. A good performance testing tool, dedicated environment, and trending reports are crucial elements in this process.

Thanks to the cloud, container technologies, and virtualization it is quite simple to prepare a testing environment even before each test run. The tricky part, however, is to integrate the test tool with our CI tool to easily run load tests. We need to run test scripts, collect data, and lastly, display results. Due to differences between test and production environments, we usually do not focus on absolute numbers but rather relative comparisons. Choosing the right load test tool could save us a lot of work.

At SmartMeter.io we are aware of this. Therefore, reporting in SmartMeter.io is as simple as possible (there are literally “one-click” reports). Reports also contain Trends analysis in clear graphs and tables, not dependent on any plug-in or CI tool, which makes it possible to use our favorite tool or a tool preferred by our client. If we want to be sure that the metrics of our application meet our business SLA, we can use acceptance criteria provided as a core component of SmartMeter.io. Every report tells us which criteria passed or failed. Any failed criterion marks the whole test run as failed, so any CI could recognize that load tests failed. There is no need to check every report. Rather, you can focus your work on the things that matter.

Martin Krutak
Contributed by Martin Krutak, SmartMeter.io

The essential component required to enable Continuous Integration/Continuous Delivery is automated testing. This is, of course, a correct statement, but only in part. Yes, automated testing is an essential component for enabling Continuous Integration/Continuous Delivery.  However, the problem with automated testing is that no matter how good the automated testing solution is, the creation of the automated testing scripts or scenarios will always lag behind the development work having been completed.

Often this is because the automated testing scripts cannot actually be written until the development work is completed i.e. the field or button on the screen needs to be added to the screen before an automated test can be written to test it. Because of this delay, automated testing lags behind development by as much as one, two, or even more sprints.  In the worst case, due to project schedule pressures, automated testing is abandoned for a period of time to meet a milestone or deadline, with the idea that it will be reinstated for the next phase of delivery.  Of course, all this does is build up a bigger automated testing debt, which has to be paid off in the next phase.

Testing needs to lead development, not lag behind it, for us to remove that lag and debt build-up. Enter Test Driven Development (TDD) or Acceptance Driven Development (ADD).  With these approaches, the requirements are written in the form of the automated tests that will be used to test the system and deem it acceptable.  Developers make changes to the system based on these definitions and acceptance criteria.

Once the automated tests and regression tests pass, the developer knows his work is completed and the system can be delivered immediately and continuously.  There is no lag at all between development and testing because automated testing scripts are written as part of the requirements definition process. The biggest change we need to make to enable Continuous Integration/Continuous Delivery is for testing to lead development, not lag behind it.

With this in mind, we can say that the essential elements required to enable Continuous Integration/Continuous Delivery are automated testing and a Test or Acceptance Driven development and testing approach.  Only when these two components are used in combination can the dream of Continuous Integration and Continuous Delivery become a reality.  Visionary companies in this space like AutotestPro offer solutions which combine TDD and automated testing so that testing does not lag behind development; instead, testing leads the development.

Paul Chorley
Contributed by Paul Chorley, Co-Founder and Managing Director, AutotestPro Ltd,

DevOps testing is the portion of the DevOps Pipeline that is responsible for the continuous assessment of incremental change. DevOps test engineers are the team members accountable for testing in a DevOps environment.  This role can be played by anyone in the DevOps environment such as QA personnel, developers, or infrastructure and security engineers for their respective areas. The DevOps test engineer can be anyone who is trusted to do the testing.

In a non-DevOps environment, independent testers from QA team test the products passed on to them by developers, and QA then passes the product on to operations. In a DevOps environment, there is a need to pass on an error-free code in small chunks (e.g. microservices). This means that there is a need for testing more frequently throughout the process, end-to-end across the development and deployment cycles. The time for each increment in DevOps is very short.

The combination of short increments and the spreading of tests across the end-to-end pipeline requires fast, automated tests and end-to-end test results coordination. Everyone in IT needs to learn to test, both manual and automated and know how to read the test results.

Testing maturity is a key differentiator of DevOps maturity:

  • Many organizations automate integrations, builds, and delivery processes but have trouble with the subtleness of test orchestration and automation
  • There is a vital role for testing architects and testing teams to offer their expertise in test design, test automation, and test case development with DevOps
  • Whether the organization is using a test-driven development methodology, behavior-based test creation, or model-based testing, testing is a vital part of the overall DevOps process — not only to verify code changes work and integrate well — but to ensure the changes do not mess up the product
  • Testing is an integral part of product development and delivery

There needs to be constantly testing so that error-free code can be merged into the main trunk and we can get a deployable code from the CD/CI. This needs people to plan for the environment, choose the right tools and design the orchestration to suit the need.

Effective DevOps testing requires Development, QA and IT Operations teams to harmonize their cultures into a common collaborative culture focused on common goals. The culture requires leaders to sponsor, reinforce, and reward collaborative team behaviors and invest in the training, infrastructures, and tools needed for effective DevOps testing

A DevOps testing strategy has the following components:

  • DevOps testing is integrated into the DevOps infrastructure
  • DevOps testing emphasizes orchestration of the test environment
  • DevOps tests are automated as much as possible
  • DevOps testing goal is to accelerate test activities as early in the pipeline as possible

The 5 Tenets of DevOps Testing are:

  • Shift Left
  • Fail Often
  • Relevance
  • Test Fast
  • Fail Early

The test strategy requires also to design the application in a loosely coupled architecture. It is very important to have a good design before moving onto the automation. Test result analysis is also another key activity to be performed to ensure that proper testing takes place with the right coverage.

Some examples of open source DevOps testing frameworks are Jenkins and Robot.

Examples of a commercially licensed testing framework are Cloudbees, Electric Cloud, and Team City. For further detailed learning, the DevOps Test Engineer Course is recommended.

Niladri Choudhuri
Contributed by Niladri Choudhuri, Xellentro,

Our approach to continuous integration testing for Fareboom Mobile relied on four types of tools: Git (alternatively – Bitbucket) for code tracking and collaboration, TeamCity (alternatively – Jenkins) for test management, HockeyApp for build distribution, and finally – Slack, integrated with the TeamCity and HockeyApp, for notifications and troubleshooting.

There are numerous great instruments, and the key is to find the ones that work best for you. For instance, we prefer TeamCity because it allows our QA engineers separately test required branches of code.  It’s important to remember that the numerous benefits of the CI/CD approach can conceal some threats you should be aware of. However, you can get around these threats if you’re prepared. Thus, our recommendations for testing in the CI/CD process would be:

  • Don’t strive for the highest code coverage, since you don’t know if the test was performed on the relevant functions
  • Adopt the practice of test-driven (test-first) development, where tests are written before the code. This allows you to prevent situations where the test is engineered around possible bugs
  • Conduct research before introducing any tools
  • Establish process management to ensure the team is following strict guidelines

Niladri Choudhuri
Contributed by Alexey Balykov, AltexSoft,

*AltexSoft is a technology consulting company with expertise in software engineering, data science, and UX design.

If you’ve been working as a tester for any length of time, you can’t have failed to notice the shift

towards CI/CD in many projects and organizations. Businesses, projects, and operations teams all want to try and take advantage of at least some of the perceived benefits of being able to quickly and consistently release new builds to production, at the push of a button. In the meantime, testers will likely have found that the CI/CD model has a big impact on how they need to approach testing.

Most of the CI/CD pipeline has development, QA, staging, and production environments where certain tests are run to ensure that the code which has been written is safe to push ahead. An automated test is the most important part of any CI/CD pipeline. Without proper automated tests that run fast, have good coverage, and no erroneous results, there can be no successful CI/CD pipeline. The automated tests are usually divided into multiple “suites”, each with their own objective.

The list below gives a small overview:

  1. Unit tests: This is the suite that is run first, often by developers themselves before they add their changes to the repository. Unit tests normally test individual classes or functions.
  2. Integration tests: After unit tests come integration tests. These tests make sure that the modules integrated together work properly as an application. Ideally, these tests are run on environments that are similar to the production environment.
  3. System tests: These tests should test the entire system in an environment as close as possible to the real production environment.

Testing in a development environment

In the development environment, smoke testing is done. Smoke testing, also known as “Build Verification Testing”, is a type of software testing that comprises a non-exhaustive set of tests aiming to ensure the most important functions run properly. The results of this testing are used to decide if a build is stable enough to proceed with further testing.

To implement smoke tests, the testing team will develop a set of test cases that are run when a new release is provided by the development team. It will be more productive and efficient if the smoke test suite is automated or it can be a combination of manual and automation. To ensure quality awareness, the smoke test cases are communicated to the development team ahead of time, so that they are aware of the quality expectations. It is important to focus on the fact that smoke test suite is a “shallow and wide” approach towards testing.

Testing in a QA Environment

In a QA Environment, regression testing is done. Regression testing is the type of testing carried out to ensure that changes made in the fixes or any enhancement changes are not impacting the previously working functionality. The regression packs are a combination of scripted tests that have been derived from the requirement specifications for previous versions of the software as well as random or ad-hoc tests. A regression test pack should, at a minimum, cover the basic workflow of typical use case scenarios.

Best practices for Testers in CI/CD

  • Perform standard actions defined in the testing procedure & check the desired responses for correctness. Any failure of the system to comply with the set of desired responses becomes a clear indicator of system regression
  • Do a careful analysis of every defect based on the previous test scenarios to avoid a slip in regression testing
  • Ensure that the regression tests are correct are not outdated

Testing in a Stage Environment

In the stage environment, (similar to the production environment) performance testing is done. Any application performance test result depends upon the test environment configurations.

Performance testing is often an afterthought, performed in haste late in the development cycle, or only in response to user complaints. It’s crucial to have a common definition of the types of performance tests that should be executed against your applications, such as Single User Test, Load Test, Peak Load Test and Stress Tests. It is best practice to include performance testing in development unit tests and performs modular and system performance tests.

Testing in a Production Environment

In a production environment, sanity testing is done. Sanity tests are usually unscripted, helping to identify the dependent missing functionalities. These tests are used to determine if the section of the application is still working on a minor change. Sanity testing goals are not to find defects but to check system health. An excellent way is to create a daily sanity checklist for the production testing that covers all the main functionalities of the application. Sanity testing should be conducted on stable builds to ascertain new functionality/ bugs have been fixed and application is ready for complete testing, and sanity testing is performed by resters only.


This blog points out which environments are part of the CI/CD pipeline and how it is configured to successfully deployment of an application. It also explains the best testing types and approach used in each environment and its best practices.

Devendra Date
Contributed by Devendra Date, DevOpsTech Solutions Pvt Ltd. ,

Implementing Continuous Integration (CI) provides software development teams the ability to adopt a regular software release schedule with an automated error detection process for a more agile, safe, and low-cost DevOps approach.

When applying this approach in data management, automated testing is important for some of the same reasons as it enables teams to execute with the 3 testing drivers: agility, accessibility, and accuracy.

Data Agility Testing

By leveraging modern data management tools, the data ingestion process can be deployed at a more rapid pace (with metadata driven workflows or drag-and-drop code generation). Agility testing helps ensure proper front-end configuration is in place, which may appear daunting, but with appropriate environment access and testing jobs, the process can be quite simple. Data agility gives teams accurate data ingestion to produce data for use or storage immediately.

Data Accessibility Testing

To start, this process tests database connections and file URLs for accuracy. In advanced models, data dictionaries and glossaries are also checked for valid entries against ingested data. This driver forces governance practices to be in place before ingestion for fewer deployment and activation problems.

Data Accuracy Testing

This testing takes place downstream of the ingestion process ensuring validation, transformation, and business rule logic is applied. It’s often considered the most difficult testing to visualize and implement at the right scope.

Tackling CI may seem complex on the surface, but by following these 3 testing drivers, teams can ingest, transform, and apply business rules faster and with fewer issues while reducing manual touch points. Once CI is configured and test-driven development is in place, you will probably wonder what you ever did without it.

Robert Griswold
Contributed by Robert Griswold, Vice President of Professional Services for TESCHGlobal

When it comes to CI/CD systems, properly designing the overall structure of your system can often more effectively test your applications than poorly designed systems with excellent tests.

When you start designing your CI/CD pipelines, the first thing to do is break apart your application into as many logically independent components as possible. For example, your application might have a frontend, backend, and middleware layer. Your first instinct might be to create a pipeline for each component, but you usually want more granular control.

For example, let’s say that you deploy your software in a Docker container. You’ll want to independently test your software (ex situ), and also test the software inside your Docker container (in situ). This allows you to catch errors specific to your code, and errors related to your deployment platform. In this instance, these ex-situ and in situ tests require their own pipelines to comprehensively test both the application and the application’s deployment.

The next thing you need to consider is upstream dependencies. Upstream dependencies are triggers that cause a pipeline to execute. As a rule, each pipeline should have one primary upstream dependency, and zero or more secondary upstream dependencies. A primary upstream dependency is usually source code (but not always). Understanding the entire set of upstream dependencies for a given pipeline will make sure that your application always has the latest code, and will identify upstream changes that break functionality.

For example, let’s say that you have source code deployed as a Docker container. The git repository that contains your source code is a primary upstream dependency. When there are new commits to the repository, it triggers a pipeline that tests the individual code. This pipeline might also subscribe to another pipeline that contains dependencies, such as security scanning software.

After you understand how to structure your upstream dependencies, you need to consider downstream dependencies. Downstream dependencies are additional pipelines that are triggered when a pipeline successfully executes. When you start designing the structure of your CI/CD system, you need to take into account all downstream dependencies for every pipeline.

For example, if multiple pieces of software depend on a common module, anytime that common module’s pipeline executes, it should trigger those additional pieces of software. This guarantees that each source code component has the most recent version of all libraries, and it will identify any problems as early as possible.

Let’s go over an example to showcase how designing the high-level actions and organization of your CI/CD system will enable you to structurally find problems, even with suboptimal testing.

An engineer at our hypothetical organization issues a patch to fix a security vulnerability. After he pushes the code to git, this triggers a pipeline which successfully tests the patch. This then triggers 5 additional pipelines, because each of these pipelines depends on the new patch. 4 of these pipelines complete successfully, but one fails. The successful pipelines push their build products to QA. An engineer notices that the last pipeline is now red, and sees that the security patch worked, but it exposed an integer overflow bug in a Java library that needs to be patched. The engineer patches the library, and the CI/CD system automatically builds, tests, and deploys that code to QA.

This pipeline structure enabled our hypothetical organization to deploy the security patch to 4 applications in QA automatically, and show engineers exactly what is wrong with the last bit of code. Without understanding the proper upstream and downstream dependencies, applying the software security patches would have been extremely time-consuming, and it would not have identified the fact that one of the apps has a bug that directly conflicts with the patch.

This CI/CD system is able to effectively test large sets of highly dependent applications simply because of how it is structured.

David Widen
Contributed by David Widen, Pre-Sales Systems Engineer, BoxBoat Technologies,

The setup could be fast, but the maintenance may be hard—this is the truth of Continuous Testing.

Here is a straightforward story of continuous testing started from scratch. The tester first climbed through some learning curves in scripting an automated test, maybe an API test, maybe a UI test. He gained hands-on experience in using test frameworks and libraries (e.g. Selenium, Mocha, Calabash, etc.). Then he found some ways to run the tests in a clean environment, understood the concept of building an image and running in a container, say using Docker. After some trial and error…Pass, Pass, Pass. Great!

The boss saw his work and said, “Let’s run all the tests once an hour, send me an alert when any test fails.” He went to Google to search some keywords: Pipeline and Scheduler. Jenkins, GitLab, Heroku—lots of systems are providing a pipeline service. By choosing any system, he could run the tests all at once right after the deployment stage. A schedule is even handier to trigger the pipeline periodically. At the same time, he saved the test results to some kind of database, so he could use those results to compare the records with previous runs. Finally, when a test failure was detected, the program would send an email to the boss.

From then, this guy no longer needs to repeat the same set of manual tests every day, every few hours, overnight, marking ticks and crosses on a long long list, being bored and making mistakes easily. But one day, sad news came. The website broke with NO alert sent. Okay…Let’s check what’s going wrong. The test script? The built image? The pipeline? The test runner? The scheduler? The machine? The database? The alert sender? That’s why I said, the maintenance may be hard. Yet, if your project is going to be a long one, it’s really worth it.

Contributed by Joyz Ng, QA Engineer, Oursky

Continuous Delivery is a key practice to help teams keep pace with business demands. True Continuous Delivery is hard to do—it is impossible without confidence in the quality and fitness of the software—the kind of confidence only deterministic and repeatable automated tests can supply.

It is common for teams who have historically been dependent on manual testing to start with an inverted test pyramid in their automated build pipeline. This is better than no tests at all, but the fragility of these types of tests tends to surface quickly. False positives, false negatives, and little improvement in escaped production defect counts tend to erode trust in the test suite and, ultimately, leads to the removal of the tests.

I recommend that my teams start with a solid core of fast running, highly isolated unit tests. Every language has a popular unit testing framework which gives teams a high level of confidence in the quality and correctness of their code in mere seconds. This should be the first quality gate in a Continuous Delivery pipeline. If these tests fail there is no point in moving on to the next stage. Unit tests show that the code does things correctly.

Building upon the unit test suite should be a thin layer of integration tests. This proves that components play nice togethe