Test Metrics

By SeaLights, the quality dashboard that lets you move fast without missing anything
Test Impact Analytics * Release Quality Dashboard * Block Untested Code Changes
Learn more

Software Quality Measurement: 8 Dimensions of Quality and How to Measure Them

Achieving high software quality is a goal of every development organization. However, in order to achieve high quality, you must be able to measure and understand the level of quality you have today. Software quality is complex and difficult to define, not to mention measure on an ongoing basis.

We’ll use the ISO/IEC 25010 standard model to define 8 dimensions of software quality, and suggest some readily-available metrics that can help you measure each one of them. We hope this will help you identify where you stand and start on a road to improving quality in your software product.

In this page you will learn about:

Why is Quality Important? 5 Negative Effects of Low Quality Releases

Software quality measurement helps you understand the level of quality of each release, even each build, produced by your development team. In many cases teams are not aware of the level of quality of a release or build, and do not have a clear indication how quality is progressing over time – is product’s quality getting better or worse?

There are several negative effects of low quality releases:

  1. Reduced productivity—low quality releases are often characterized by poor code quality and a high level of complexity. Software complexity has a negative effect on developer productivity – any change to the software becomes much more difficult, bugs can be very difficult to isolate and fix, and as a result, technical debt can grow.
  2. Critical production failures—beyond the obvious effect of production faults, it also affects developer productivity. Your best developers have to “stop everything” and attend to issues in production, which may be difficult to diagnose and fix.
  3. Employee turnoveremployees who work on high-quality code are usually more satisfied with their work. Working on low quality, complex code and hunting down bugs from the past is frustrating for developers, and can result in significant turnover rates.
  4. Unsatisfied customerssoftware quality directly affects customer satisfaction, revenues, and market share.
  5. Increased costsit is much more expensive to fix bugs after the software has been released to production. Fixing bugs during development can be up to 5X cheaper than fixing them in production.

To learn more about these challenges, see our white paper, Reactive Software Maintenance: The Silent Killer of Developer Productivity.

8 Dimensions of Software Quality

The ISO/IEC 25010 standard provides a useful model of 8 software quality dimensions. Below we’ll explain each dimension and suggest how to measure it in your software project, letting you catch quality issues early and react to them.

Quality Dimension How to Measure
1. Maintainability

How easily you can modify or adapt your software for use by other teams, new purposes or changing requirements. The more complex your code is, the harder it will be to maintain. High quality code is easier to work with and can be easily adapted to new requirements.

  • Lines of Code (LOC)
  • Static code analysis
  • Cyclomatic complexity (and other complexity measurements)
  • Peer review
2. Portability

How easily you can move a software component to another environment. Robust software should be able to run on multiple operating systems and in different environments, for example on the cloud or on-premises, and dependencies on other software should be minimized and well defined.

  • Setting up realistic testing environments
  • Automating tests of installation/uninstallation
  • UI and regression tests to identify problems on specific platforms
3. Functionality

How well software functions compared to user expectations. Functionality is the “what” – if the software does what the user needs it to do, it is functioning properly.

  • Regression tests
  • Integration tests
  • UI automation
  • Acceptance tests
4. Performance

How well software performs under a workload. Performance includes response time and latency, scalability, and stability under maximum expected loads.

  • Stress testing
  • Soak testing
  • Page load times
5. Compatibility

How well software runs on different browsers, devices, or operating systems. Modern applications are expected to work seamlessly across a large matrix of devices and screen sizes.

  • Cross browser testing
  • UI automation
6. Usability

How easily users can understand and use software functionality. UI/UX is a major focus in modern software development. Software needs to be prototyped and tested with usability in mind, and then continuously monitored to see if new usability issues come up.

  • User acceptance testing
  • Website engagement and conversion
  • Real user monitoring
  • Customer satisfaction
7. Reliability

How likely a software system is to fail or malfunction. The more reliable your software, the less likely users are to experience bugs in production.

  • Production defects
  • Unit tests
  • Integration tests
  • Acceptance tests
8. Security

How vulnerable software is to cyber attacks. The more secure your software, the better it protects users against breaches and service interruption as a result of malicious or accidental human actions.

  • Number of incidents
  • Number of breaches
  • Existing known vulnerabilities
  • Mean Time to Repair (MTTR)

Towards a Holistic Measurement of Software Quality

We presented several dimensions of software quality and readily-available metrics you can use to measure them. However, none of these metrics provides a holistic view of software quality. Development teams need actionable information about which parts of their products suffer from quality issues, and where to focus maintenance to have the biggest impact on software quality.

A new category of tools called Software Quality Intelligence can help. For example, SeaLights is a quality intelligence platform that helps visualize test gaps – product features that are used in production, have undergone recent changes, and are not sufficiently tested across all relevant dimensions of software quality. SeaLights can report which areas of the product are at high risk of quality issues – helping you prioritize work in your sprints to quickly improve software quality.

Request a live demo to see how SeaLights can help you visualize and improve software quality.