CI/CD Pipeline Adoption Case Study: 6000 Releases per Month

Alon Eizenman
Alon EizenmanCTO & Co-founder | November 14, 2016

The driver behind Continuous Integration (CI) and Continuous Delivery (CD) has always been speed: release fast and be the first to market. But back in 2007, release automation wasn’t widely established, and the concept of Continuous Delivery (CD) didn’t even have a formal name yet. So while it was possible that a company’s developers, testers, and operations were productive, there was no methodological change in their operations-related processes to facilitate more rapid builds and releases.

CI/CD pipeline has changed that. What is Continuous Integration and Continuous Delivery? CI/CD evolves slow-moving release cycles into rapid software development and delivery that facilitates the release of only the highest quality builds to customers. Automated testing is a cornerstone of this methodology. Done well, it continuously ensures that your application behaves as intended – protecting quality, while supporting fast and streamlined delivery.

In this post, we’ll take a look at a large European bank that adopted a CI/CD pipeline, and examine how development and operations teams – as part of a DevOps culture – can work together to streamline and automate releases, tests, and builds.

Revising Outdated Delivery Practices

When it comes to implementing a new CI/CD-based culture at organizations, there will inevitably be tension between the new way of working, and the way many developers consider the ‘right’ way for Agile teams to work. The source of these conflicts often has to do with how teams approach making software ‘ready for release’ – or being able to define what is “Done”.

CD is designed to provide the same types of results. CD differs from ‘traditional’ Agile in that there is no need to stop and make an effort to create and build releases. And CD does not sacrifice quality either; teams are forced to fix bugs as they’re found, rather than leaving them for later. CD’s success is based on its ability to get faster feedback on changes and create a stable and repeatable process that will successfully deploy to production as needed.

The benefit of cross-functional, multi-discipline teams – accompanied by automated operations – is that they create more focus on what needs to be “Done” from the user’s perspective — delivering value into production. This includes users stories such as the one below – which clearly enhance overall customer experience and satisfaction.

Learn how to increase your UI Code Coverage | Download >>

Ditching Waterfalls

One of the major challenges for our bank was that their software development was tied to an outdated Waterfall model. These outdated development life cycle methods incurred release cycles that took between 12-18 months to complete. To add insult to injury, these releases were bloated and crammed with features, which hurt them significantly as the number of dependent features prolonged the time it took to test the release. It wasn’t uncommon to see three months of rigorous testing go by, and manual deployment was a nightmare. On top of all that, issues would still crop up in production, even after so much testing had been conducted.

Impact on Releases

The R&D and operations teams were able to support an astonishing rate of 60,000 releases a month! They were able to achieve this rate by focusing on the release size, not speed. By making each individual change (release increment) as small as it could possibly be, development teams reduced the release cycle to a span of hours. Features could be taken from request to production in a single day.

Not all of these releases were accepted, of course. A large number were, however, and the ones that weren’t could quickly iterate from failure to success. By rapidly incorporating feedback from business process leaders, the development teams were able to improve on failed releases. Out of those 60,000 releases a month, they began to accept 15 permanent changes per day.

Rebuilding Mobile

One major success for the bank was that they were able to completely rebuild their old mobile application. The old app had been buggy, and outdated and difficult to use—averaging only one star in user reviews.

Under the old development regime, the bank R&D leaders estimated that it would have taken upwards of three years to bring the failed mobile app to a usable condition, with the bank lagging its competitors the entire time. Instead, they decided to do away with the old application and start from scratch.

Using their Continuous Delivery pipeline, they were able to build a bare-bones version of the app in two weeks. Building on the basic application, they incorporated feedback from real customers, and the application was feature-complete in a matter of months, eventually achieving an average five-star rating.

Into the Future: What More can be Achieved from Continuous Delivery?

These achievements in terms of time-to-release and time-to-market are why CD is receiving so much buzz today. And although everyone is talking about CD, not enough people have seen it in action.

What we’ve seen with our case study is that while CD provides tremendous value, there is still room for refinement. Remember, the bank in our example achieved 60,000 releases per month, but still accepted only 15 changes per day. This indicates that although failed releases can quickly be improved to production quality, they still represent a drag on the efficiency of CD as a whole.

Going forward, the next challenge is for CD to deliver more production-quality releases in a single cycle, without revisions. If we can make this a reality, the gains in terms of development speed would be astronomical.

Watch this space for more thought leadership on continuous testing, integration, and delivery, and how SeaLights intends to improve this process by increasing efficiencies.