In the software engineering industry, metrics are essential. Teams use metrics to keep track of a number of things, but the main purpose is to know whether they're doing something right or not and make improvements. Metrics work very well as a group, and there are several metrics groups for different situations. There are metrics for measuring the overall productivity of a software team. We have incident metrics so teams can know how well they can handle incidents that lead to failure. There are also flow metrics to keep track of how much value you can deliver to the end users or customers. Choosing the right set that aligns with your business and development is crucial to ensure success.
‍
In this post, we're going to be looking at testing metrics in agile. These metrics show how effective a team's QA practices really are. QA testing is essential to ensure a team continues to release high-quality applications.
Importance of Testing Metrics for QA
Before we look at the agile metrics, let's learn why they're important in the first place. This quote by Tom DeMarco puts it simply: "If you can't measure it, you can't control it."
Normally, testing is a separate phase in the development process, but in agile, it's a continuous process. In agile, testing starts at the beginning of the project. Starting testing earlier will help you find defects earlier and reduces the likelihood of delays later in the development process. Continuous feedback is a hallmark of the agile development process, and by intertwining testing and development, teams can ensure only quality products get to the end users.
As has been noted, metrics help a team know when they're doing something right; that's the general idea behind metrics. In this case, agile testing metrics give software engineering teams a quantifiable measure of the effectiveness and efficiency of their QA testing activities. These metrics will show if their testing methods need improvement.
‍
The insights from QA metrics help inform the decisions of the team and show how well they're progressing toward their goals. Metrics are always tied to specific goals for business and development; otherwise, they're meaningless numbers and figures. That's why it's important to carefully select the relevant group of metrics to track over a period of time.
QA metrics provide insights into a number of things—for example, the duration of testing activities and the number of defects or bugs detected during the test. They can also show the number of test cases, number of critical defects, and so on.
Simply put, QA testing metrics will show the engineers what areas need improvement and, as such, where they should focus more. So now that you know what QA testing is all about, let's have a look at the metrics.
Â
Agile Testing Metrics
Like all metrics, testing metrics come in two forms: base and derived metrics. The base metrics are simple and straightforward; teams will simply count the relevant parameters. Some examples of base agile metrics include the total number of test cases, number of successful tests, number of bugs and defects, number of failed or blocked test cases, number of severe bugs, number of planned tests, etc.
‍
Base metrics, due to their simplicity, will not offer many insights, and that's where derived metrics come into play. For instance, say your team releases an app. You record a thousand downloads—that's great! But that's just a small part of the bigger picture. Your team will need to receive feedback from the end users based on their experience using the app. This feedback will enable your team to know the number of users who are happy, are satisfied, discover bugs, want more features, etc. This deeper insight into user experiences will have an impact on any subsequent delivery or update.
‍
Getting an absolute number from a base metric just isn't enough to give your team a correct or complete estimation of the effectiveness of your QA testing activities. The agile metrics we're going to look at are derived; you get them by applying specific formulas to the base metrics. These "new" metrics will, in turn, produce more useful data. So, in no particular order, let's have a look at the agile metrics your team needs to measure.
1. Test Execution Coverage
The test execution coverage is the percentage of test cases a team runs compared to the planned test cases. You get this by counting the total number of test cases already executed and dividing by the total number of planned test cases. This metric will show if a team is progressing in their testing efforts.
For example, say your team's original plan was to run 200 test cases, but you end up running 180. This will put the test coverage at 90%.
Teams can automate test coverage, as this will ensure they can find more defects and produce high-quality applications.
2. Defect Discovery Rate
This metric tracks how often the test cases reveal bugs. Basically, it measures the effectiveness of your team's testing activities. The defect discovery rate can be calculated by dividing the number of defects found by the total number of tests run.
So, using the same scenario as before, if out of those 180 test cases, nine bugs were discovered, the defect discovery rate will be 5%.
This rate should be as low as possible at all times. Having a high defect rate will cause delays to production, which will, in turn, lead to unhappy customers and loss of revenue. It can put pressure on the team to quickly fix bugs, and this can lead to accumulating technical debt and producing low-quality applications. This leads us nicely to the next metric.
3. Defect Leakage
This metric measures the quality of the test cases. It allows the team to measure the effectiveness of their QA testing activities. Basically, you're keeping track of how many bugs slip past the team to the end users. You can't know the defect leakage without first knowing the test effectiveness (which can also serve as a separate metric).
You can calculate the test effectiveness by getting the sum of the defects you discover during testing and the defects found after shipping, then using that number to divide the sum total of defects found during testing. So it's bugs during testing plus bugs with the end users over the total bug population.
From the previous example, we had nine defects from the test cases, which is before deployment. Now, if after deployment, the end users find six more bugs, this will bring the total number of defects to 15. The test effectiveness, in this scenario, is 60%. This means the QA team's activities failed to discover a whopping 40% of the defects.
In software engineering, you can't avoid defects, and they come in all forms and leave a bad taste with the end users. With this in mind, it's safe to say a 100% test effectiveness is near impossible, but your effectiveness should still be as high as possible. Keeping track of this metric over time will give your team useful insights into the efficiency of your testing activities and whether you'll need to change or improve the testing processes and methods.
4. Test Tracking Metrics
Test cases can pass, fail, or get blocked. This metric is threefold, as it tracks the percentage of each of these possible outcomes. You can calculate this metric by dividing the sum total of any of these three by the total number of test cases.
Let's say there are 200 test cases, and, out of that total, 170 pass, 20 fail, and 10 got blocked. So the percentage of test cases that pass is 85%, those that fail is 10%, and the blocked test cases make up the remaining 5%.
Now, it's up to the team to decide if these numbers are acceptable or not by comparing them against the goals that align with the metric.
5. Burndown Chart
A burndown chart is the graphical representation of the rate at which a team completes its test cases within a time period or sprint. It's a measure of how fast they can complete story points, which are essentially goals set by the team. This chart will also show how much work is yet to be completed during the sprint.
Â
The image above is an example of a burndown chart from Jira (source). The story points are on the x-axis, the red line is the team's work during the sprint, and the gray line is where they should be, according to their estimates. By looking at this chart, they can predict if the work can actually be completed within the sprint period.
The burndown chart will help teams have a better understanding of their capacity—that is, what they're capable of handling. This metric is easy to track and understand. The insights it provides will help teams properly plan the workflow of the next sprint.
6. Velocity
Velocity is a measure of how much work a team can complete within a sprint. This metric tracks progress and will help the team predict or estimate how long they'll need to complete a project. In a testing scenario, the team's velocity is how often they complete story points, which are, essentially, the requirements needed to complete a particular goal.
So let's say a team completes 25 stories in a three-week sprint, and each story is worth two points. That team's velocity will be 50 story points per sprint.
Having an accurate measure of your team's velocity will ensure you work according to your capacities to avoid work spilling into the next sprint.
7. Code Complexity
This metric measures the complexity of code and reveals if it's readable and testable. In addition, it will show if you can easily maintain that piece of code. You can also call this metric "cyclomatic complexity."
Defects are more likely to be in complex pieces of code, so if that's the case, this metric will reveal when it's time to refactor. To deliver a high-quality product, you must ensure the code has minimal defects. To achieve this, you'll need to have low code complexity, so it's best to keep your code clean and simple. This metric will also reveal the risks of not complying with industry-standard code regulations.
Cyclomatic complexity was developed by Thomas McCabe in 1976. It measures the number of linearly independent parts in the source code of a program. This metric is calculated using a control-flow graph, and its formula is M = E - N + 2P. M = complexity, E = number of edges, N = number of nodes, and P is the number of connected properties.
8. Defect Cycle Time
Defect cycle time is basically the life cycle of a bug. The defect cycle time spans from when a team starts fixing a defect or error to when they resolve all issues and fix the bug.
Now, let's say your team spends a total of 42 hours, on average, fixing bugs across eight sprint periods. If each sprint runs for three weeks, the defect cycle time is going to be 14 hours per sprint.
It should concern teams if they record long cycle times or there's a spike when you compare with previous sprints. Long cycle times mean you're spending too much time fixing bugs and not enough time working on new applications or features. Your team needs to find the root cause of any increase in cycle time as soon as possible.
Tracking this metric consistently over a period of time, across different sprints, will show your team how effective it is at fixing problems.
9. Cumulative Flow Diagram
This is a visual representation of a team's workflow. It's a graph that displays work in progress, completed work, backlogs, velocity, and testing. The cumulative flow graph will help the team identify which item—for example, testing—is a bottleneck.
Let's have a look at an example (source):
Â
An easy way to identify bottlenecks is when a section—for instance, testing—starts expanding vertically. This graph shows the overall work progression of a team.
How to Track Metrics
Now that you know what agile metrics you should track, the next step is knowing how to track them. The first step is knowing what to measure. Only pick those that are relevant to your business and development goals. Metrics can be easy to track, but it can be confusing to choose what to measure. You'll have to choose metrics that cater to both the quality of the product and the process used to deliver that product.
‍
Base metrics, as we've seen, are the simplest metrics to track because they produce absolute numbers. But these numbers don't offer much and will only be useful if your team compares them with numbers from different time periods. It's the same with metrics you derive with formulas; they'll provide clear and useful insights when you collect them consistently over time. Have a system in place to record and store data from metrics during each sprint.
‍
Goals are important when tracking metrics. Your team should have clear targets, which will help you measure progress. Lowering the defect count, hitting more story points, completing more planned test cases—these are some goals or targets you may have. Metrics provide answers to important questions when you ask the relevant ones. Every software engineering team has its own definition of success. Deciding what's important to your team will ease the process of tracking the right metrics.
It's also important that metrics are simple and easy to collect and understand. The number of defects, one metric; the number of critical defects, another metric; the number of defects before deployment, yet another metric. You get the point: metrics don't have to be complex. You can use the same metrics multiple times and they'll remain relevant. Do not spend time and resources collecting what your team cannot use.
.png)
Â
Conclusion
There's a metric for every possible scenario you can think of in software development. There are metrics for measuring productivity, metrics for incidents that cause failure, metrics for speed, and so on.
Agile testing metrics will help your team evaluate and measure the effectiveness of their testing activities. QA testing is important in ensuring that only high-quality applications get to the end users. These nine agile metrics for testing are there to help your teams continuously maintain high standards. Maintaining high quality in your applications will help you keep the users happy and satisfied. Use these metrics to find which areas you can improve and have a healthier software delivery system.