QE Data That CxOs Need Versus The Data They Have
It is an open secret that Quality Engineering (QE) is instrumental to software’s success. With the ever-evolving technological landscape, many tools exist to measure software quality; however, all the data generated by such tools are currently unstructured and in silos.
Even though organizations are working on amalgamating the data and delivering several engineering-specific metrics such as accelerated velocity, the percentage decrease in code churn, the percentage increase in coding days per week, and features delivered, to name a few, they need to be more relevant to CxOs.
No particular dashboard/solution provides CxOs with data on the effectiveness of the quality processes that directly impact the business. Such metrics/data can significantly enable CxOs to make informed decisions regarding product quality and identify areas for improvement with the engineering and quality functions.
Analyzing the Impact of Poor Quality Engineering
In 2013, the HealthCare.Gov site was launched. The website was the official healthcare exchange platform that allowed US residents to compare healthcare plans and identify whether a resident qualifies for federal subsidies. The platform initially covered the residents of 36 states. The launch was plagued by issues right from the get-go.
- The platform went down within two hours of launch due to its inability to handle the load
- The web pages took forever to load
- There were issues with rendering the UX and UI interface across device types
- Data inaccuracies
Before
The schedule for testing was reduced from months to weeks. A strict launch deadline was cited as the primary reason for rushing through security testing and troubleshooting. The incident was highly preventable if adequate and incremental approaches like early testing, beta testing, and regular delivery of a completely tested platform were made in conjunction with development.
Eventually, the issues were resolved at a total cost of $1.7B instead of the initial budget of $93.7M.
Now
Without dedicated and required metrics, CxOs would continue to face several challenges, such as balancing speed with quality, customer-centric personalization with data privacy, and resource gaps with the increasing cost of human capital, to name a few. CxOs must ensure their software is thoroughly tested for quality and meet the standards to deliver a flawless customer experience. In fact, according to a study from the American Society for Quality (ASQ), organizations that effectively implement quality management practices experience an average of 9% increase in sales and a 26% increase in profitability. Moreover, World Quality Report 2022-2023 stated that 88% of its respondents were at medium to high risk of losing market share to a peer, and 90% of the respondents agreed that they were at risk of increased costs for the deployment of new technology solution without a proper QE strategy in place.
The Need to Harness the Power of Metrics
To drive proactive business outcomes, CXOs should be cognizant of choosing and tracking the metrics that are in alignment with their business goals and help them gain insights into their software’s performance, areas/opportunities for improvement, and take steps to identify and resolve potential problems before they become significant issues.
There are several tools to measure the effectiveness of the quality process that helps leaders reduce the time to market. However, most of them are often used by the product and quality engineering teams after the release has been deployed into production and are full of shortcomings. Let’s analyze a few such tools:
- Static code analysis tools
These tools are integral to QE as they can find potential defects early in code development. Static code analysis tools can analyze computer software without actually running the software and automate several manual tasks in the code review process. However, let’s also look at the other side of the coin. SonarQube, for example, can sometimes generate false positives, which results in QE teams wasting time investigating and fixing non-existent defects. The tools can be complex to set up, configure, and maintain.
Furthermore, a code quality index percentage metric provides CxOs with data and insights into the engineering team’s skills, expertise, and proficiency. Thus an absence of such a metric is a significant setback leading to ambiguity in determining release readiness and release velocity.
2. Open Source Automation Frameworks
The open source automation tools gained popularity in the QE field due to their flexibility, innovation, and cost-effectiveness. But the fact that it is an “open source” might make these tools susceptible to security vulnerabilities. Some open source tools can be complex to set up and configure. For instance, Cucumber can be a complex tool for users unfamiliar with BDD.
Test automation is the crux of the CI/CD process; however, it can only be effective once adequate metrics are in place to determine the right level of automation to deliver desired results. A metric like test automation percentage is the key. By tracking and setting goals to maximize the test automation % CxOs can ensure that their product teams respond to market needs in no time while maintaining Quality.
3. Test case management tools
Even though test case tools are a valuable asset to QE as they help efficiently manage, organize, execute, and track testing efforts, it is of little use to CxOs if the tools are not used to derive and compute a metric reflecting the overall maturity and cohesion between the development, test, and operations functions, i.e., the engineering score. Using test case management tools such as TestRail to their full potential, one can also derive traceability percentage, which is an essential attribute of determining software’s release readiness.
Envision this Scenario
CxOs are seeking metrics/data that can guide their decisions regarding product quality and enhancing engineering and quality functions. The need of the hour is a solution that quantifies quality and empowers CxOs with actionable data. This data-driven approach becomes the catalyst for elevating software quality organization-wide. As our blog series unfolds, the next installment will delve into the effectiveness of the quality metrics developing into its practical rendition.
