Revenue Protection and Change Stability Value Driver
Perhaps one of the most important QA metrics to track is how much revenue is being protected by QA efforts as a whole. This is most impactful in terms of ROI to the business, outside of time savings.
Defect Escape Rate
Another key DORA metric we track in Quality Hub is the escape ratio of defects (or bugs). Most organizations use tools such as JIRA, Azure DevOps (ADO), or other dedicated Test Management & Defect Tracking tools to log and track work on bugs. Our goal is to allow our users to surface that data in direct relation to the overall testing effort, rather than create another system of record for you to maintain.
With that in mind, our JIRA/ADO integration allows us to import and bi-directionally sync issues and defects so that users can continue to use familiar tooling and only come to Provar Quality Hub for reporting, analytics, collaboration, and integrations. For more information, see Defect Management.
This metric tracks the # of production incidents as a percentage of overall defects raised for the given time period. The metric card displays the current average defect escape rate.
If you click into the metric drilldown for Defect Escape Rate, you’ll get a detailed breakdown of your current escape rate, the average escape rate, total # defects, total production defects, and number escaped this month. This gives you relevant, timely, actionable data to take to your QA teams to focus efforts and show improvements over time.
The insights here are also data-driven and reportable. The example insights above tell us that 88% of our total defects were caught before production, and our current escape rate is a measurable improvement over the current average, showcasing improved testing coverage!
As with other metric drilldowns, you’ll also see the steps to improve your defect escape rate, the current maturity tier, and the KPI thresholds for your organization.
On the whole, defect escape rate is one of the true markers of a high-quality product, and tracking against this metric can convey the value of QA to the entire organization.
Risk Management and Risk Avoidance
Another key aspect of testing and QA is to reduce risk. But what is a risk? In Quality Hub, you can define Risks in a number of different ways. Take the below risk as an example.
Once a risk is defined, it goes through various stages before being considered “Mitigated”. This is up to your team on how you want to utilize risks, but we have created the basic framework for you already. Risks can be created from the Risks tab of the Provar Quality Hub app or directly from Test Plans.
If a risk has its Status updated to “Mitigated”, then the Risk Cost Avoided is calculated as a result based on an internal calculation, which includes the Risk Score * Average Defect Cost.
For example, this risk here has a Risk Level of “Very Low”, and therefore its Risk Cost Avoided is fairly low, but still significant based on the average cost of a defect.
For more information see, Risk Management.
Now that this organization has a few risks marked as “Mitigated”, we can calculate the overall Risk Avoidance metric and display it here on the Quality Hub Executive Dashboard.
This metric card displays the total risk cost we have avoided in our organization, through the enablement of better testing, higher risk coverage, and proper mitigation strategies, which can also be generated by Quality Hub’s AI capabilities. This risk avoidance factor is a calculation of our total risk cost avoided across all risks that have been mitigated multiplied by the Risk Avoidance Factor, which is defined in the Cost Model Configuration page of the Provar Quality Hub app.
Drilling down further into the metric displays the risk avoidance over time, key insights from your data, and a breakdown of your risk numbers (such as average cost saved per risk and total # risks mitigated).
If you scroll down, you’ll see the familiar How to Improve section, with actionable steps to improve your risk avoidance maturity, and your Risk Avoidance KPI thresholds.
