Like many software development managers looking to increase velocity, I use CI. When I first started with CI, I definitely saw an increase in my team’s speed of development because it prevented integration problems and allowed them to catch bugs earlier on in the development process. After our codebase matured and grew, and our regression test suite also grew, CI actually began to damage our development velocity and efficiency. 

As a software development manager, there is a neverending demand to speed up my team’s releases. However, as I described above, many times my hands are tied because testing is a bottleneck and there is nothing I can do to speed up the QA process. 

Long Continuous Integration and Regression Test Cycles cause:

Long Wait Times: Waiting for your own CI cycle is difficult, but the wait becomes even more unbearable when there are other developers in front of you waiting for their CI cycles to finish, lengthening the wait time. 

Lengthy Feedback Loops: All you want is immediate feedback about the code you developed so you can continue developing new features quickly and efficiently, or alternatively, quickly fix a production issue. Instead, you need to wait for the CI to finish before you can move forward.

Context Switch: Because you can’t sit idly by while the CI is running, you start new tasks which you then inevitably need to stop as soon as the CI completes so that you can go back and fix whatever bug the CI found. This context switch is difficult and makes completing tasks challenging.

Sometimes I feel like Tom Cruise in the movie “Mission Impossible” (cue the music….). To increase velocity my team needs to test less, but if they test less then our software quality drastically decreases causing even more bugs in production. 

See!? Mission impossible! Until now….

Test Impact Analysis: Stop Waiting for Tests You Do Not Need to Run 

One way to deal with long regression and CI cycles is Test Impact Analysis. This theory states that you should only run those tests which are necessary to ensure the quality of a build. The tests you must run in any given cycle are:

• Impacted Tests: Existing tests impacted by recent code changes 
• Failed Tests: Tests that didn’t pass in the previous test cycle(s)
• Pinned/Flagged Test: Tests that are marked as important to run in every test cycle
• New Tests: New tests that have not yet run in a test cycle

This short list of tests drastically decreases the CI cycle and solves so many challenges that developers experience on a day to day basis. But, if you try creating this list of tests manually, how can you be sure that you include all of the necessary tests? Isn’t this just as time-consuming as waiting for your CI to finish?

SeaLights Test Impact Analysis: “Smart” Test Recommendation and Execution

In August 2019, SeaLights released a new feature: Test Impact Analysis. Using Machine Learning, SeaLights Test Impact Analysis automatically selects and executes only the relevant tests required to validate a given code commit, generating a shorter, more focused test cycle and providing faster feedback to developers per the quality of their code changes. 

All the tests listed above (Impacted, Failed, Pinned, and New) are automatically recommended to you and you even have the option of SeaLights automatically executing these tests for you. 

An additional element of SeaLights’ Test Impact Analysis is that you can schedule full test runs so that you don’t miss anything and don’t end up compromising your software’s quality. You decide when they run: daily on the first build, weekly on the first build, or customize this feature and choose to run it every x test cycles.

Mission Accomplished: Simultaneously Increasing Velocity and Quality

Since my team started using SeaLights Test Impact Analysis our CI time is down from 30 min to 10! Because our CI has to work less, there are fewer bottlenecks and faster feedback loops; we can fix our code faster and discover quickly if our code is of high quality. Our velocity is actually increasing without harming the product’s quality – something that software teams strive to achieve daily can be easily accomplished with this new feature.

One of my team members recently told me enthusiastically “it is AMAZING when I can see the Test Impact Analysis in action on our CI! When you compare how long the CI took before we used this feature and now, it’s unbelievable!”

You can try SeaLights Test Impact Analysis for yourself – fill out the form below and one of our product experts will get back to you shortly.