One Key Metric Should Drive Quality Engineering

 If you are in an IT organization, you know how important Quality Engineering metrics are.  Gone are the days that you can talk to a few quality engineers and get their gut feel on determining if a software application has a high degree of quality.  It requires a lot more effort and energy and numbers to figure that out.  Quality Engineering metrics are the heartbeat of any IT organization.  While you should have several there is one that you should spend the majority of your time and effort focusing on.  That Quality Engineering metric is: Defects.  Defects tell so much of the story.  Once you are able to gather that metric and classify it you can do some pretty amazing things.

I have had many quality engineering positions over the years and understanding defects is the first one that I put my energy and effort doing research.  I start to ask a few questions:

1. How does the organization feel about defects?  Is it seen as a positive tool or a negative one?  Do developers take defects personally or do they encourage their quality engineering counterparts to create defects?  This is a really important piece of information because it will help me to understand a lot about an organization and their appetite to influence change.
2. Are all defects entered into a central tool?  This is necessary so that you will be able to capture all defects and not have to hunt through multiple applications to find them.
3. How much technical debt does an organization have?  From what I have researched, most organizations carry a good bit of technical debt.  They are reluctant to spend time and energy in resolving defects.  It creates a negative experience from a business perspective and internal customers often have to workaround issues to get their desired result.
4. Is there a standard for defects?  Once defects are being captured, there are certain criteria that needs to be gathered on each defect so that you can start to see trends and make decisions.  Some of those standards include severity, business priority, root cause, project or sprint, environment, and application.  By gathering this information you can start to classify defects based upon that criteria.
5. Are defects being captured in production?  This is critical.  This metric will help you understand if the applications are stable, and if defects are being captured prior to a production deployment.  Often, production defects are captured in a separate tool, which makes it very hard to consolidate and gain access to for the quality engineering organization.  If they are being captured, what information is gathered?  Is it possible to tie it to a specific release or feature?
6. Which teams are finding the majority of the defects?  Once I can get my hands on this information, I find it extremely helpful.  In one of my previous companies, I did analysis and found that most of the defects were being captured by UAT testers.  This led me to infer that they had the most subject matter expertise on the applications that were being tested.  I began to build a relationship with that team, and did several things to help the UAT testers and gain additional knowledge from that team.  The first thing, was to review the test cases they had created.  While they were at a very high level, my QA team was able to gain some valuable information.  We took that information and incorporated it into our test cases.  Second, we looked at their test cases and mapped those to our test cases.  My team had started automating test cases, so let let the UAT testers see the execution of those scripts and they agreed to let us run the regression test cases for them.  This was a huge boost in productivity for them and it really helped to solidify the relationship.

Using this framework, I did analysis on a company where I previously worked and identified a defect leakage percentage of 38%.  This number was mind blowing and really unacceptable. I established a goal to reduce defect leakage in production and set the target at 8%.

Here were some key focus areas when the team spent the bulk of their energy:

• Reviewed all production defects and developed test cases for those.  Once the test cases were built we incorporated those test cases into our regression suite.
• Leveraged the information gained from the UAT testers to build robust Quality Engineering test cases
• Continued to build more test automation scripts so we could spend more energy building and executing test cases
• Partnered with the development organization and ran our automated scripts sooner in the lifecycle so that we could find more defects upfront

After a year of hard work, the results were impressive.  We were able to get the production defect leakage down to 7%.  This was a huge milestone and everyone was thrilled.  The business was really happy with the improvements and became a fan of the quality engineering team.  While there are many quality engineering metrics that should be captured, defects is the first one that you should start with.