Case Study - Pokemon

How Pokémon Reduced Their Weekly Production Incidents to Less Than One a Month With Sealights Quality Intelligence

* Down From 2-3 Incidents a Week to Less than 1 Incident Per Month * Manual Tests Cover 100% of All Code Changes * No Missed Untested Code Changes in 5 Months


Pokémon’s Software Quality group is a relatively new organization and they are responsible for the company’s quality infrastructure. Because the organization is new, it analyzed Pokémon’s quality difficulties and challenges in order to come up with a comprehensive and high confidence solution.

They discovered that even though their tests were passing they were still reporting weekly incidents, indicating that risky untested code changes were reaching production. The team was determined to eliminate this problem; it needed a way to gain insight into existing code coverage and identify if new code changes were actually tested by their regression test suites before a release.

Pokémon’s technology organization began using SeaLights in June 2018, providing the data-driven insights they were missing and reducing its incidents by 90%.

“We use SeaLights as part of our go-no-go decisions for every release.”

Ben Goldsmith, Software Engineering Manager, Pokémon

The Problem

Ben Goldsmith is the Software Engineering Manager at Pokémon’s new SDET group and he leads a team of four software development engineers in test (SDET). His team is responsible for building the test infrastructure for the entire test department.  

Ben Goldsmith’s team were handling 2-3 incidents a week. He concluded that this was a result of his team’s inability to correlate new code changes with existing legacy code dependencies, and where those coverage gaps were. He needed a tool that could identify untested code changes so his team could focus on developing tests in those areas while shoring up legacy code gaps, preventing bugs from reaching production.

Ben Goldsmith also expressed that without actually knowing where the untested code changes were located, his team was wasting its time guessing and developing tests which did not accurately cover all of the changes. He believed that once his team was able to identify the gaps in its tests it could quickly close them, improving the efficiency and effectiveness of its test development.

Additionally, Pokémon focuses on conducting a lot of manual tests due to the nature of its products. It required a tool that could verify that both its manual and automated tests are actually covering risky code changes.  

Why SeaLights Release Quality Analytics?

Sealights was the only tool that Ben Goldsmith found which provided functionalities for both manual and automated tests. SeaLights Release Quality Analytics supplied Ben Goldsmith’s team with the data-driven insights they were missing, reducing their incidents from 2-3 a week to less than 1 a month. “Sealights takes a lot of the analytics work out of it,” said Ben Goldsmith. “We can see what our tests are covering as the code base changes over time.”  

Pokémon uses SeaLights for:

  • Blocking risky untested code changes before they are released  This data has become an integral part of Pokémon’s SDLC because it automatically identifies untested code changes; the team has full visibility of their release quality and there are no surprises. Ben Goldsmith added, “with SeaLights my team can work independently and make their own decisions about their releases.”
  • Efficient and effective test development – Ben Goldsmith’s team is able to track all code changes in every build and every release and verify that their regression tests are actually covering and testing these code changes, making each test more effective and preventing incidents from reaching production.
  • Verifying test quality of both manual and automated tests – Ben Goldsmith uses SeaLights to verify that both his team’s manual and automated regression tests are covering all the newly modified code.

” With SeaLights we know with confidence that we are covering 100% of the code that changes from build to build, and as a result, we have not had any untested code changes in the last 5 months with our legacy app of over 80,000 lines of code. “

Ben Goldsmith, Software Engineering Manager, Pokémon

  • Short feedback loops between manual tests and developers – SeaLights automatically shows Pokémon untested code changes that its manual tests were missing. With this data, Pokémon’s developers can provide direct feedback to the manual testers so they know exactly which new tests they need to develop in order to quickly and accurately close those gaps.
  • Quick data-driven insights -The SeaLights dashboard is easy to use and can be implemented across the entire SDLC. It allows Pokémon’s team leaders to make their own data-driven decisions about their releases and applications, without having to consult with their managers.

Next Steps

In order to further increase velocity and quality, Pokémon wants its development team to start using SeaLights Pull Request Analytics as part of its Pull Request Process. These analytics will provide accurate feedback to the developers so that they can block untested code changes even before they merge, saving time and money on future tests.

About Pokémon

The Pokémon Company International manages the property outside of Asia and is responsible for brand management, licensing, marketing, the Pokémon Trading Card Game, the animated TV series, home entertainment, and the official Pokémon website.

About SeaLights

SeaLights is the #1 Quality Intelligence Platform for software development. The platform provides managers with actionable data analytics for both manual and automated tests so that they can focus their team’s development and testing cycles and make data-driven decisions about their releases, resulting in fewer incidents in production.

Want to learn how you can reduce your production incidents?

Fill out the form for more information.

With SeaLights we know with confidence that we are covering 100% of the code that changes from build to build, and as a result, we have not had any untested code changes in the last 5 months with our legacy app of over 80,000 lines of code.

Ben Goldsmith, Software Engineering Manager, Pokémon