SeaLights Case Study

How The Pokémon Company International Reduced its Weekly Production Incidents to Less than One a Month with SeaLights Quality Intelligence

Overview

The Pokémon Company International’s software quality group is responsible for the company’s quality infrastructure. It analyzed the organization’s quality difficulties and challenges in order to come up with a comprehensive and high confidence solution. 

They discovered that even though their tests were passing they were still reporting weekly incidents, indicating that risky untested code changes were reaching production. The team was determined to eliminate this problem; it needed a way to gain insight into existing code coverage and identify if new code changes were tested by their regression test suites before a release. 

The company’s technology organization began using SeaLights in June 2018, providing the data-driven insights they were missing and reducing its incidents by 90%.

Down From 2-3 Incidents a Week to Less than 1 Incident Per Month

Manual Tests Cover 100% of All Code Changes

No Missed Untested Code Changes in 5 Months

The Problem

The Pokémon Company International’s team was handling 2-3 incidents a week. They concluded that this was a result of the team’s inability to correlate new code changes with existing legacy code dependencies, and where those coverage gaps were. They needed a tool that could identify untested code changes so the team could focus on developing tests in those areas while shoring up legacy code gaps, preventing bugs from reaching production. 

The team also expressed that without knowing where the untested code changes were located, they were wasting time guessing and developing tests which did not accurately cover all of the changes. They believed that once the team was able to identify the gaps in its tests it could quickly close them, improving the efficiency and effectiveness of its test development.

Additionally, The Pokémon Company International focuses on conducting a lot of manual tests due to the nature of its products. It required a tool that could verify that both its manual and automated tests are actually covering risky code changes.

Why SeaLights Release Quality Analytics?

SeaLights was the only tool that the company found which provided functionalities for both manual and automated tests. SeaLights Release Quality Analytics supplied the team with the data-driven insights they were missing, reducing their incidents from 2-3 a week to less than 1 a month. “SeaLights takes a lot of the analytics work out of it,” said the software quality manager. “We can see what our tests are covering as the code base changes over time.”  

The Pokémon Company International uses SeaLights for:

  • Blocking risky untested code changes before they are released: This data has become an integral part of the company’s SDLC because it automatically identifies untested code changes; the team has full visibility of their release quality and there are no surprises. The software quality manager added, “with SeaLights my team can work independently and make their own decisions about their releases.”
  • Efficient and effective test development: The team is able to track all code changes in every build and every release and verify that their regression tests are actually covering and testing these code changes, making each test more effective and preventing incidents from reaching production.
  • Verifying test quality of both manual and automated tests: The team uses SeaLights to verify that both their manual and automated regression tests are covering all the newly modified code. 

“With SeaLights we know with confidence that we are covering 100% of the code that changes from build to build, and as a result, we have not had any untested code changes in the last 5 months with our legacy app of over 80,000 lines of code.”

Software Quality Manager, The Pokémon Company International

  • Short feedback loops between manual tests and developers: SeaLights automatically shows the company’s untested code changes that its manual tests were missing. With this data, the developers can provide direct feedback to the manual testers so they know exactly which new tests they need to develop in order to quickly and accurately close those gaps.
  • Quick data-driven insights: The SeaLights dashboard is easy to use and can be implemented across the entire SDLC. It allows team leaders to make their own data-driven decisions about their releases and applications, without having to consult with their managers.

“What we really like about SeaLights is that we can quickly see our quality risks and easily understand what our tests are not covering as our code base changes over time.”

Software Quality Manager, The Pokémon Company International

Next Steps

In order to further increase velocity and quality, The Pokémon Company International wants its development team to start using SeaLights Pull Request Analytics as part of its Pull Request Process. These analytics will provide accurate feedback to the developers so that they can block untested code changes even before they merge, saving time and money on future tests.

About The Pokémon Company International

The Pokémon Company International manages the property outside of Asia and is responsible for brand management, licensing, marketing, the Pokémon Trading Card Game, the animated TV series, home entertainment, and the official Pokémon website.

About SeaLights

SeaLights is the #1 Quality Intelligence Platform for software development. The platform provides managers with actionable data analytics for both manual and automated tests so that they can focus their team’s development and testing cycles and make data-driven decisions about their releases, resulting in fewer incidents in production. For more information, fill out the form to the right.below.

With SeaLights we know with confidence that we are covering 100% of the code that changes from build to build, and as a result, we have not had any untested code changes in the last 5 months with our legacy app of over 80,000 lines of code.

Ben Goldsmith, Software Engineering Manager, Pokémon