Like many software development managers looking to increase velocity, I use CI. When I first started with CI, I definitely saw an increase in my team’s speed of development because it prevented integration problems and allowed them to catch bugs earlier on in the development process. After our codebase matured and grew, and our regression test suite also grew, CI actually began to damage our development velocity and efficiency.
As a software development manager, there is a neverending demand to speed up my team’s releases. However, as I described above, many times my hands are tied because testing is a bottleneck and there is nothing I can do to speed up the QA process.
Long Continuous Integration and Regression Test Cycles cause:
Long Wait Times: Waiting for your own CI cycle is difficult, but the wait becomes even more unbearable when there are other developers in front of you waiting for their CI cycles to finish, lengthening the wait time.
Lengthy Feedback Loops: All you want is immediate feedback about the code you developed so you can continue developing new features quickly and efficiently, or alternatively, quickly fix a production issue. Instead, you need to wait for the CI to finish before you can move forward.
Context Switch: Because you can’t sit idly by while the CI is running, you start new tasks which you then inevitably need to stop as soon as the CI completes so that you can go back and fix whatever bug the CI found. This context switch is difficult and makes completing tasks challenging.
Sometimes I feel like Tom Cruise in the movie “Mission Impossible” (cue the music….). To increase velocity my team needs to test less, but if they test less then our software quality drastically decreases causing even more bugs in production.
See!? Mission impossible! Until now….
Test Impact Analysis: Stop Waiting for Tests You Do Not Need to Run
One way to deal with long regression and CI cycles is Test Impact Analysis. This theory states that you should only run those tests which are necessary to ensure the quality of a build. The tests you must run in any given cycle are:
- Impacted Tests: Existing tests impacted by recent code changes
- Failed Tests: Tests that didn’t pass in the previous test cycle(s)
- Pinned/Flagged Test: Tests that are marked as important to run in every test cycle
- New Tests: New tests that have not yet run in a test cycle
This short list of tests drastically decreases the CI cycle and solves so many challenges that developers experience on a day to day basis. But, if you try creating this list of tests manually, how can you be sure that you include all of the necessary tests? Isn’t this just as time-consuming as waiting for your CI to finish?
SeaLights Test Impact Analysis: “Smart” Test Recommendation and Execution
In August 2019, SeaLights released a new feature: Test Impact Analysis. Using Machine Learning, SeaLights Test Impact Analysis automatically selects and executes only the relevant tests required to validate a given code commit, generating a shorter, more focused test cycle and providing faster feedback to developers per the quality of their code changes.
All the tests listed above (Impacted, Failed, Pinned, and New) are automatically recommended to you and you even have the option of SeaLights automatically executing these tests for you.
An additional element of SeaLights’ Test Impact Analysis is that you can schedule full test runs so that you don’t miss anything and don’t end up compromising your software’s quality. You decide when they run: daily on the first build, weekly on the first build, or customize this feature and choose to run it every x test cycles.