DevOps is the ideal practice for software development businesses that want to code, build, test, and release software continuously. It’s popular because it stimulates cross-skilling and self-improvement by creating a fast-paced, results-oriented, collaborative workplace. QA in DevOps fosters agility, resulting in speedier operational support and fixes that meet stakeholder expectations. Most significantly, it ensures the timely delivery of high-quality software.
Quality Assurance and Testing (QAT) is a critical component of a successful DevOps strategy. QAT is a vital enabler that connects development and operations in a collaborative thread to assure the delivery of high-quality software and applications.
DevOps QA Testing – Integrating QA in DevOps
Five essentials play a crucial role in achieving flawless sync and seamlessly integrating QA into the DevOps process.
1. Concentrate on the Tenets of Testing
Testing is at the heart of QA; hence the greatest and most experienced testers with hands-on expertise must be on board. Some points to consider: the team must maintain a strong focus on risk, include critical testing thinking into the functional and non-functional parts of the application, and not lose sight of the agile testing quadrant’s needs. Working closely with the user community, pay particular attention to end-user experience tests.
2. Have relevant technical skills and a working knowledge of tools and frameworks
While studying and experimenting with the application is required, a thorough understanding of the unique development environment is also required. This guarantees that testers contribute value to the design stage talks and advise the development team on possibilities and restrictions. They must also be familiar with the operating environment and, most significantly, the application’s performance in the actual world.
The team’s skill set should also include a strong understanding of automation and the technologies required, as this adds rigor to the DevOps process and is necessary to keep it going. The QA team’s knowledge must be focused on tools, frameworks, and technologies. What should their automation strategy be? Do they advocate or utilize licensed or open-source software?
Are the tools for development, testing, deployment, and monitoring identified for the various software development life cycle stages? To avoid delays and derailing the process at any stage of development, it is critical to have comprehensive clarity on the use of tools. Teams with the competence should be able to experiment with technologies like artificial intelligence or machine learning to give the process a boost.
3. Agile Testing Methodologies
DevOps is synchronized agility that incorporates development, quality assurance, and operations. It’s a refined version of the agile testing technique. Agile/DevOps techniques now dominate software development and testing. Can the QA team ensure that in an Agile/DevOps environment, the proper coverage, aligned to the relevant risks, enhances velocity? Is the individual or group familiar with working in a fast-paced environment? Do they have the mechanisms to ensure continuous integration, development, testing, and deployment in cross-functional, distributed teams?
4. Industry experience that is relevant
Relevant industry knowledge ensures that the testers are aware of user experiences and their potential influence on business and can predict potential bottlenecks. Industry expertise improves efficiency and helps testers select testing that has the most impact on the company.
5. The Role of Culture
In DevOps, the QA team’s culture is a crucial factor. The DevOps methodology necessitates that QA team members be alert, quick to adapt, ethical, and work well with the development and operations teams. They serve as a link between the development and operations teams, and they are responsible for maintaining balance and ensuring that the process runs smoothly.
In a DevOps process, synchronizing the three pillars (development, quality assurance, and operations) is crucial for software products to fulfill expectations and deliver commercial value. The QA team serves as a link in this process, ensuring that software products transfer seamlessly from development to deployment. What factors do you believe QA can improve to integrate more quickly and influence the DevOps process?
Cloud computing is now the foundation for digital transformation. Starting as a technology disruptor a few years back, it has become the de facto approach for technology transformation initiatives. However, many organizations still struggle to optimize cloud adoption. Reasons abound – ranging from lack of a cohesive cloud strategy to mindset challenges in adopting cloud platforms. Irrespective of the reason, assuring the quality of applications in cloud environments remains a prominent cause for concern.
Studies indicate a wastage of $17.6 billion in cloud spend in 2020 due to multiple factors like idle resources, overprovisioning, and orphaned volumes and snapshots (Source parkmycloud.com). Further, some studies have pegged the cost of software bugs to be 1.1 trillion dollars. Assuring the quality of any application hosted on the cloud not only addresses its functional validation but also its performance-related aspects like load testing, stress testing, capacity planning, etc invariably addressing both the issues described above, thereby exponentially reducing the quantum of loss incurred on account of poor quality.
The complication for QA in cloud-based application arises due to many deployment models ranging from private cloud, public cloud to hybrid cloud, and application service models ranging from IaaS, PaaS, to SaaS. While looking at deployment models, testers will need to address infrastructure aspects and application quality. At the same time, while paying attention to service models, QA will need to focus on the team’s responsibilities regarding what they own, manage, and delegate.
Key aspects that mandate a shift in the QA approach in cloud-based environments are –
Earlier and to some extent even now, when it comes to legacy applications, QA primarily deals with a monolithic architecture. The onus was on understanding the functionality of the application and each component that made up the application, i.e., QA was not just black-box testing. The emergence of the cloud brought with it a shift to microservices architecture, which completely changed testing rules.
Multiple scrum teams work on various application components or modules deployed in containers and connected through APIs in a microservices-based application. The containers have a communication mechanism based on contracts. QA methodology for cloud-based applications is very different from that adopted for monolith applications and therefore requires detailed understanding.
Security, compliance, and privacy
In typical multi-cloud and hybrid cloud environments, the application is hosted in a 3rd party environment or multiple 3rd party environments. Such environments can also be geographically distributed, with data centers housing the information residing in numerous countries. Regulations that restrict data movement outside countries, service models that call for multi-region deployment, and corresponding data storage and access without impinging on regulatory norms need to be understood by QA personnel.QA practitioners also need to be aware of the data privacy rules existing across regions.
The rise of the cloud has given way to a wide range of cybersecurity issues – techniques for intercepting data and hacking sensitive data. To overcome these, QA teams need to focus on vulnerabilities of the application under test, networks, integration to the ecosystem, and third-party software deployed for complete functionality. Usage of tools to simulate Man In The Middle (MITM) attacks helps QA teams identify and overcome any sources of vulnerability through countermeasures.
Building action-oriented QA dashboards need to extend beyond depicting quality aspects to addressing security, infrastructure, compliance, and privacy.
Scalability and distributed ownership
Monolithic architectures depend on vertical scaling to address increased application loads, while in a cloud setup, this is more horizontal in nature. Needless to say that in a cloud-based architecture, there is no limitation to application scaling. Performance testing in a cloud architecture need not consider aspects like breakpoint testing since they can scale indefinitely.
With SaaS-based models, the QA team needs to be mindful that the organization may own some components that require testing. Other components that require testing may be outsourced to other providers, and some of these providers may include cloud providers. The combination of on-premise components and others on the cloud by the SaaS provider makes QA complicated.
Reliability and Stability
This entirely depends on the needs of the organization. An Amazon that deploys 100,000 times a day – features and updates of its application hosted in cloud vis-a-vis an aircraft manufacturer that ensures the complete update of its application before its aircraft is in the air, have diverse requirements for reliability stability. Ideally, testing done for reliability should uncover four categories – what we are aware of and understand, what we are aware of but do not understand, what we understand but are not aware of, and what we neither understand nor are we aware of.
Initiatives like chaos testing aim to uncover these streams by randomly introducing failures through automated testing and scripting and seeing how the application reacts/sustains in this scenario.
QA needs to address the below in a hybrid cloud setup are –
What to do when one cloud provider goes down
How can the load be managed
What happens to disaster recovery sites
How does it react when downtime happens
How to ensure high availability of application
Changes in organization structure
Cloud-based architecture calls for development through pizza teams, smaller teams fed by one or two pizzas, in common parlance. These micro product teams have testing embedded in them, translating into a shift from QA to Quality Engineering (QE). The tester in the team is responsible for engineering quality by building automation scripts earlier in the cycle, managing performance testing strategies, and understanding how things get impacted in a cloud setup. Further, there is also increased adoption of collaboration through virtual teams, leading to a reduction in cross-functional QA teams.
Tool and platform landscape
A rapidly evolving tool landscape is the final hurdle that the QA practitioner must overcome to test a cloud-based application. The challenge becomes orchestrating superior testing strategies by using the right tools and the correct version of tools. Quick learning ability to keep up with this landscape is paramount. An open mindset to adopt the right toolset for the application is needed rather than an approach steeped with blinders towards toolsets prevailing in the organization.
In conclusion, the QA or QE team behaves like an extension of customer organization since it owns the mandate for ensuring the launch of quality products to market. The response times in a cloud-based environment are highly demanding since the launch time for product releases keeps shrinking on account of demands from end customers and competition. QA strategies for cloud-based environments need to keep pace with the rapid evolution and shift in the development mindset.
Further, the periodicity of application updates has also radically changed, from a 6-month upgrade in a monolith application to feature releases that happen daily, if not hourly. This shrinking periodicity translates into an exponential increase in the frequency of test cycles, leading to a shift-left strategy and testing done in earlier stages of the development lifecycle for QA optimization. Upskilling is also now a mandate given that the tester needs to know APIs, containers, and testing strategies that apply to contract-based components compared to pure functionality-based testing techniques.
In today’s world, embracing inclusivity with accessibility is not only about being humane or legally correct but also makes a lot of business sense. Recent studies have shown that businesses can tap into an additional prospective user base of up to 15% to market their products. Persons with disabilities (PWD) are responsible for 25% of all healthcare spending in the U.S.
Businesses have increasingly become aware of the requirements of people who need accessible technologies to contribute to a work environment or who can also be prospective customers.
“At Barclays, accessibility is about more than just disability. It’s about helping everyone to work, bank and live their lives regardless of their age, situation, abilities, or circumstances. – Paul Smyth, Head of Digital Accessibility, Barclays
What is digital accessibility?
The Web Accessibility Initiative (WAI) states that websites, tools, and technologies should be designed and developed so that even differently-abled people can use them. More specifically, these people should be able to perceive, understand, navigate, and interact with the Web and contribute to the Web.
Web Accessibility enables people with disabilities to participate equally on the Web. Broadly speaking, Web accessibility encompasses all disabilities that affect access to the Web, including:
When an organization removes barriers by making its application accessible to its full potential, it will be an inclusive product, as millions of people with various disabilities can use it.
Mandated by Law – American with Disabilities Act (ADA)
One hears the terms “Section 508”, an amendment to the Workforce Rehabilitation Act of 1973 that requires that all Information Technology assets of the United States’ federal government be accessible by people with disabilities.
Also, ADA (American with Disabilities Act) the Title III requires that all private businesses that are open to the public be accessible to people with disabilities. There is a steady rise in the number of lawsuits filed over the years under this section, resulting from the growing awareness of the ADA Title III. Digital accessibility compliance helps organizations protect themselves against this rising trend of ADA Title III Federal lawsuits.
Foundation for accessibility
The web accessibility guidelines, technical specifications, and educational resources to help make the web accessible to people with disabilities are developed by Web Accessibility Initiative (WAI). They are an integral part of the W3C (World Wide Web Consortium), focusing on accessibility. Over time, the WAI has developed several recommendations, some of which are:
The latest edition of Web Content Accessibility Guidelines (WCAG) 2.1 has additional coverage for mobile and non-W3C technologies (non-web documents and software).
Four principles of accessibility
The WCAG guidelines lay down the four principles which are the foundation for Web accessibility: Perceivable, Operable, Understandable, and Robust (POUR for short).
Perceivable: The objective is to make content available to the senses, primarily vision, and hearing, via either the browser or through assistive technologies like screen readers, screen enlargers, etc. For the visually impaired who mainly rely on a screen reader to have the website’s content read, one needs to add an alternative text that provides a textual alternative to non-text content in web pages that makes the content perceivable. Another example is videos and live audio must have captions and a transcript. With archived audio, a transcription may be enough.
In this video example: Perceivable, the video page uses “voice recognition” and is also updated to use “speech recognition.” “Voice recognition” or “speaker recognition” is a technology that identifies who the speaker is, not the words they’re saying. “Speech recognition” is about recognizing words for speech-to-text (STT) transcription, virtual assistants, and other speech user interfaces. Together they allow a person with visual impairment to enhance their experience of the web.
Operable: The objective is to enable a user to interact with all controls and interactive elements using either the mouse, keyboard, or an assistive device. Most people will get frustrated by the inability to use a computer because of a malfunctioning mouse. Many people prefer to use only the keyboard to navigate websites. Whatever be the reason, either personal preference or circumstance like temporarily limited mobility, a permanent physical disability, or simply a broken mouse, the result is the same: Websites and apps need to be operable by a keyboard. For example, all links and controls on the web page must be accessible using the Tab key on the keyboard.
In addition, Operable principles allow users enough time to use and interact with the content. It also helps them navigate and find content. For example, if all rich, interactive features of the web page like dropdown menus, carousels, modal dialogs, etc., comply with the W3C’s WAI-ARIA 1.0 Authoring Practices recommendations, will ensure that users can easily navigate to the right content.
Understandable: The objective is to ensure that every functionality of web content is easily understandable. A user must be able to understand all navigation and other forms of interaction. To provide a user the best possible experience, every point of interaction deserves careful attention. Navigation should be consistent and predictable throughout the context of the website. Interactive elements like form controls should also be predictable and clearly labeled. For example, Instead of saying: “To postulate a conceit more irksome than being addressed in sesquipedalian syntax is adamantine,” it is better to say: “Being spoken to in unnecessarily long and complicated language is a pain.”
Despite knowing these basics, many websites lack structuring using headings, lists, and separations. Some even use overly complex language, jargon, and unexplained acronyms. It makes these websites difficult and unappealing for many people, including non-native speakers and makes them unusable for people with cognitive and learning disabilities.
Robust: People get familiar and comfortable with different technologies like operating systems, browsers, and versions of browsers with usage and time. Some people like advanced features, whereas many disable them. There are early adopters of new technologies while others are slow to adapt to the rapidly-changing currents in the flow of technological advances.
A User should have the freedom to choose their technologies to access web content. This allows the user to customize the technology to meet his needs, including accessibility needs. In some cases, it might take additional time and effort to develop web content, depending on the specifications of the technologies used. However, in the long run, it will produce more reliable results and increase the chances that the content is accessible to people with disabilities.
You might have experienced the frustration of being told your technology is out of date or no longer supported. Whilst frustrating, you’ve probably found a way around the issue – but what if you couldn’t because you rely on that technology to interact with the digital world.
Beyond the four principles
Web Content Accessibility Guidelines 2.1 are organized into three levels of conformance:
Level A – addresses the most basic features of web accessibility features. Level AA – deals with the most common and biggest barriers for disabled users and is covered in most accessibility regulations globally, including the ADA. Level AAA – the highest level of web accessibility will make the software or product accessible to the maximum number of users. However, these requirements are not entirely easy to conform to and can be focused on if your audience is primarily people with disabilities.
These levels are across all of the previously mentioned principles. Some of the key requirements include:
Google Aces Accessibility
Google’s investment in accessibility provides the company with an innovation edge in a broad array of products and services. Some of the innovations are:
Contrast minimums: a feature designed especially for people with low vision, and the feature also helps everyone see in bright light glare situations.
Auto-complete: initially designed for people with disabilities, now used widely by all users.
Voice control: although initially implemented for users with physical impairments is now widely adopted by millions of users for the convenience it provides
Artificial intelligence: originally integrated to provide visual context to users with visual impairments
Auto-captioning leveraging machine learning designed mainly for deaf users did not see many adopters in that target audience, as many feel it is still inadequate to meet their needs. However, advances in machine learning itself have found broader applications.
The importance of microservices architecture testing
Increased adoption of digital has pushed the need for speed to the forefront. The need to conceptualize, develop, launch new products and iterate to make them better, much ahead of the competition and gain customer mindshare, has become the critical driver of growth. Adoption of agile principles or movement towards scrum teams for increased agility are all steps in this direction. Disruptive changes have also taken place on the application front with the 3-tier architecture of the late 90s and subsequent 2 tier monolithic architecture giving way to one that is based on microservices.
Having a single codebase made a monolithic architecture less risky but slow to adopt changes, the exact opposite of a services-based architecture. Microservices architecture makes it easier for multiple development teams to make changes to the codebase in parallel. By transforming an application into a distributed set of services that are highly independent, yet interdependent, provides the ability to create new services/functionalities and modify services without impacting the overall application. These changes can be achieved by teams cutting across geographies or locations and makes it easier for them to understand functional modules rather than the humongous application codebase. However, the highly distributed nature of services also gives them a heightened ability to fail.
Breaking it down – Testing strategies in a microservice architecture
At the core, a microservices architecture comprises of 3 layers – a REST layer that allows the service to expose APIs, the database layer, and the service layer. A robust testing strategy needs to cover all these layers and ensure that issues are not leaked to production. The further an issue moves across stages, the impact increases on account of multiple teams getting affected. Hence the test plan must cover multiple types of testing like service testing, subsystem testing, client acceptance testing, performance testing, etc. Subsequent paragraphs outline key aspects of service level testing and integration testing in a microservices architecture-based application landscape.
In service level testing, each service forming a part of the application architecture needs to be validated. Each service has dependencies on other services and transmits information to others based on need. In a monolith architecture, since connections are being established from one class to the other within the same Java Virtual machine (JVM), chances of failure are far lower. However, in a services architecture, these are distributed, driving the need for network calls to access other services and makes it more complex.
Functional Validation: The primary goal in services testing is the functionality validation of a service. Key to this is the need to understand all events the service handles through both internal as well as external APIs. At times this calls for simulating certain events to ensure that they are being handled properly by the service. Collaboration with the development team is key to understand incoming events being handled by the service as part of its functionality. A key element of functional validation – API contract testing, tests the request and response payload along with a host of areas like pagination and sorting behaviors, metadata, etc.
Compatibility: Another important aspect is recognizing and negating backward compatibility issues. This happens during the launch of a changed version of the service that breaks existing clients running in production. Changes that happen to API contracts need to be evaluated in detail to understand if they are mandatory and capable of breaking clients in production. An addition of a new attribute or a parameter may not classify as a breaking change; however, changes to response payload, behavior, error codes, or datatypes have the ability to break. A change in value typically changes the logic behind it as well. They need to be uncovered much earlier in the service testing lifecycle.
Dependencies: Another aspect of focus is external dependencies, where one would test both incoming as well as outgoing API calls. Since these are heavily dependent on the availability of other services and hence other teams, there is a strong need to obviate dependency through the usage of mocks. Having conversations with developers and getting them to insert mocks while creating individual services will enable testing dependencies without waiting for the service to be available. It is imperative to make sure the mocks are easily configurable without needing access to the codebase. Usage of mocks also drives ease in automation giving teams the ability to run independently with no configuration.
Understanding Microservices Architecture Testing
Once each service is tested for its functionality, the next step is to move onto validate how the various collaborating services work together end to end. Known as subsystem testing or integration testing, it tests the whole functionality exposed together. Understanding the architecture or application blueprint by discussions with the development team is paramount in this stage. Further, there is a strong need to use real services deployed in the integration environment rather than mocks used for external dependencies.
As part of integration testing, there is a need to validate if the services are wired very closely and talking to each other. The event stream and inter-service API calls need to be configured properly so inter-service communication channels are proper. If the service functionality level testing is proper, the chances of finding errors are minimal in this stage, since the required mocks created in the functionality testing stage would have ensured that the services function properly.
Looking in-depth, we find that the testing strategies in a microservices architecture are not extremely different from those adopted for a monolith application architecture. The fundamental difference comes in the way the interdependencies and communication between multiple services forming a part of the larger application are tested to ensure that the application as a whole function in line with expectations.
A quick scan of the application landscape shows that customers are more empowered, digitally savvy, and eager to have superior experiences faster. To achieve and maintain leadership in this landscape, organizations need to update applications constantly and at speed. This is why dependency on agile, DevOps, and CI/CD technologies has increased tremendously, further translating to an exponential increase in the adoption of test data management initiatives. CI/CD pipelines benefit from the fact that any new code that is developed is automatically integrated into the main application and tested continuously. Automated tests are critical to success, and agility is lost when test data delivery does not match code development and integration velocity.
Why Test Data Management?
Industry data shows that up to 60% of development and testing time is consumed by data-related activities, with a significant portion dedicated to testing data management. This amply validates that the global test data management market is expected to grow at a CAGR of 11.5% over the forecast period 2020-2025, according to the ResearchandMarkets TDM report.
Best Practices for Test Data Management
Any organization focusing on making its test data management discipline stronger and capable of supporting the new age digital delivery landscape needs to focus on the following three cornerstones.
Applicability: The principle of shift left mandates that each phase in an SDLC has a tight feedback loop that ensures defects don’t move down the development/deployment pipeline, making it less costly for errors to be detected and rectified. Its success hinges to a large extent on close mapping of test data to the production environment. Replicating or cloning production data is manually intensive, and as the World Quality Report 2020-21 shows, 79% of respondents create test data manually with each run. Scripts and automation tools can take up most heavy lifting and bring this down to a large extent when done well. With production quality data being very close to reality, defect leakage is reduced vastly, ultimately translating to a significant reduction in defect triage cost at later stages of development/deployment.
However, using production-quality data at all times may not be possible, especially in the case of applications that are only a prototype or built from scratch. Additionally, using a complete copy of the production database is time and effort-intensive – instead, it is worthwhile to identify relevant subsets for testing. A strategy that brings together the right mix of product quality data and synthetic data closely aligned to production data models is the best bet. While production data maps to narrower testing outcomes in realistic environments, synthetic data is much broader and enables you to simulate environments beyond the ambit of production data. Usage of test data automation platforms that allocates apt dataset combinations for tests can bring further stability to testing.
Tight coupling with production data is also complicated by a host of data privacy laws like GDPR, CCPA, CPPA, etc., that mandate protecting customer-sensitive information. Anonymizing data or obfuscating data to remove sensitive information is an approach that is followed to circumvent this issue. Usually, non-production environments are less secure, and data masking for protecting PII information becomes paramount.
Accuracy: Accuracy is critical in today’s digital transformation-led SDLC, where app updates are being launched to market faster and need to be as error-free as possible, a nearly impossible feat without accurate test data. The technology landscape is also more complex and integrated like never before, percolating the complexity of data model relationships and the environments in which they are used. The need is to maintain a single source of data truth. Many organizations adopt the path of creating a gold master for data and then make data subsets based on the need of the application. Adopting tools that validate and update data automatically during each test run further ensures the accuracy of the master data.
Accuracy also entails ensuring the relevance of data in the context of the application being tested. Decade-old data formats might be applicable in the context of an insurance application that needs historic policy data formats. However, demographic data or data related to customer purchasing behavior applicable in a retail application context is highly dynamic. The centralized data governance structure addresses this issue, at times sunsetting the data that has served its purpose, preventing any unintended usage. This also reduces maintenance costs for archiving large amounts of test data.
Also important is a proper data governance mechanism that provides the right provisioning capability and ownership driven at a central level, thereby helping teams use a single data truth for testing. Adopting similar provisioning techniques can further remove any cross-team constraints and ensure accurate data is available on demand.
Availability: The rapid adoption of digital platforms and application movement into cloud environments have been driving exponential growth in user-generated data and cloud data traffic. The pandemic has accelerated this trend by moving the majority of application usage online. ResearchandMarkets report states that for every terabyte of data growth in production, ten terabytes are used for development, testing, and other non-production use cases, thereby driving up costs. Given this magnitude of test data usage, it is essential to align data availability with the release schedules of the application so that testers don’t need to spend a lot of time tweaking data for every code release.
The other most crucial thing in ensuring data availability is to manage version control of the data, helping to overcome the confusion caused by conflicting and multiple versioned local databases/datasets. The centrally managed test data team will help ensure single data truth and provide subsets of data as applicable to various subsystems or based on the need of the application under test. The central data repository also needs to be an ever-changing, learning one since the APIs and interfaces of the application keeps evolving, driving the need for updating test data consistently. After every test, the quality of data can be evaluated and updated in the central repository making it more accurate. This further drives reusability of data across a plethora of similar test scenarios.
The importance of choosing the right test data management tools
In DevOps and CI/CD environments, accurate test data at high velocity is an additional critical dimension in ensuring continuous integration and deployment. Choosing the right test data management framework and tool suite helps automate various stages in making data test ready through data generation, masking, scripting, provisioning, and cloning. World quality report 2020-21 indicates that the adoption of cloud and tool stacks for TDM has witnessed an increase, but there is a need for more maturity to make effective use.
In summary, for test data management, like many other disciplines, there is no one size fits all approach. An optimum mix of production mapped data, and synthetic data, created and housed in a repository managed at a central level is an excellent way to go. However, this approach, primarily while focusing on synthetic data generation, comes with its own set of challenges, including the need to have strong domain and database expertise. Organizations have also been taking TDM to the next level by deploying AI and ML techniques, which scan through data sets at the central repository and suggest the most practical applications for a particular application under test.
The first steps towards a rewarding QA outsourcing engagement
Expanding nature of products, the need for faster releases to market much ahead of the competition, knee jerk or ad hoc reactions to newer revenue streams with products, ever-increasing role of customer experience across newer channels of interaction, are all driving the need to scale up development and testing. With the increased adoption of DevOps, the need to scale takes a different color altogether.
Outsourcing QA has become the norm on account of its ability to address the scalability of testing initiatives and bring in a sharper focus on outcome-based engagements. The World Quality Report 2020 mentions that 34% of respondents felt QA teams lack skills especially on the AI/ML front. This further reinforces their need to outsource for getting the right mix of skill sets so as to avoid any temporary skill set gaps.
However, ensuring that your outsourced QA gives you speed and scale can be a reality only if the rules of engagement with the partner are clear. Focusing on 4 R’s as outlined below while embarking on the outsourcing journey, will help you derive maximum value.
The foremost step is to identify the right partner, one with a stable track record, depth in QA, domain as well as technology, and the right mix of skill sets across toolsets and frameworks. Further, given the blurring lines between QA and development with testing being integrated across the SDLC, there is a strong need for the partner to have strengths across DevOps, CI/CD in order to make a tangible impact on the delivery cycle.
The ability of the partner to bring to the table prebuilt accelerators can go a long way in achieving cost, time and efficiency benefits. The stability or track record of the partner translates to the ability to bring on board the right team which stays committed throughout the duration of the engagement. The team’s staying power assumes special significance in longer duration engagements wherein shifts in critical talent derail efficiency and timelines on account of challenges involved with newer talent onboarding and effective knowledge transfer.
An often overlooked area is the partner’s integrity. During the evaluation stages, claims pertaining to industry depth as well as technical expertise abound and partners tend to overpromise. Due care needs to be exercised to know if their recommendations are grounded in delivery experience. Closer look at the partner’s references and past engagements not only help to gain insight into their claims but also help to evaluate their ability to deliver in your context.
It’s also worthwhile to explore if the partner is open to differentiated commercial models that are more outcome-driven and based on your needs rather than being fixated on the traditional T&M model.
With the right partner on board, creating a robust process and governing mechanism assumes tremendous significance. Mapping key touchpoints from the partner side, aligning them to your team, and identifying escalation points serve as a good starting point. With agile and DevOps principles having collaboration across teams as the cornerstone, development, QA, and business stakeholder interactions should form a key component of the process. While cross-functional teams with Dev QA competencies start off each sprint with a planning meeting, formulating cadence calls to assess progress and setting up code drop or hand-off criteria between Dev and QA can prevent Agile engagements from degrading into mini waterfall models.
Bringing in automated CI/CD pipelines obviates the need for handoffs substantially. Processes then need to track and manage areas such as quality and release readiness, visibility across all stages of the pipeline through reporting of essential KPIs, documentation for managing version control, resource management, and capacity planning. At times, toolset disparity between various stages and multiple teams driving parallel work streams creates numerous information silos leading to fragmented visibility at the product level. The right process should focus on integration aspects as well to bridge these gaps. Each team needs to be aware and given visibility on ownership at each stage of the pipeline.
Further, a sound process also brings in elements of risk mitigation and impact assessment and ensures adequate controls are built into SOP documents to circumvent any unforeseen event. Security measures are another critical area that needs to be incorporated into the process early on, more often it is an afterthought in the DevOps process. Puppet 2020 State of DevOps report mentions that integrating security fully into the software delivery process can quickly remediate critical vulnerabilities – 45% of organizations with this capability can remediate vulnerabilities within a day.
Clear and effective communication is an integral component of QA, more so when DevOps, Agile, and similar collaboration-heavy initiatives are pursued achieving QA at scale. Effective communication at the beginning of the sprint ensures that cross-functional teams are cognizant of the expectations from each of them and have their eye firmly fixed on the end goal of application release. From then on, a robust feedback loop, one that aims at continuous feedback and response, cutting across all stages of the value chain, plays a vital role in maintaining the health of the DevOps pipeline.
While regular stand-up meetings have their own place in DevOps, effective communication needs to go much beyond to focus on tools, insights across each stage, and collaboration. A wide range of messaging apps like Slack, email, and notification tools accelerate inter-team communication. Many of these toolkits are further integrated with RSS feeds, google drive, and various CI tools like Jenkins, Travis, Bamboo, etc. making build pushes and code change notifications fully automated. Developers need notifications when a build fails, testers need them when a build succeeds and Ops need to be notified at various stages depending on the release workflow.
The toolkits adopted by the partner also need to extend communication to your team. At times, it makes sense for the partner to have customer service and help desk support as an independent channel to accept your concern. The Puppet report further mentions that companies at a high level of DevOps maturity use ticketing systems 16% more than what is used by companies at the lower end of the maturity scale. Communication of the project’s progress and evolution to all concerned stakeholders is integral irrespective of the platforms used. Equally important is the need to categorize communication in terms of priority and based on what is most applicable to classes of users.
Documentation is an important component of communication and from our experiences, commonly underplayed. It is important for sharing work, knowledge transfer, continuous learning, and experimentation. Code that is well documented enables faster completion of audit as well. In CI/CD-based software release methodology, code documentation plays a strong role in version control across multiple releases. Experts advocate continuous documentation as core communication practice.
Finally, it goes without saying that setting parameters for measuring the outcome, tracking and monitoring those, determines the success of the partner in scaling your QA initiatives. Metrics like velocity, reliability, reduced application release cycles and ability to ramp up/ramp down are commonly used. Further, there are also a set of metrics aimed at the efficiency of the CI/CD pipeline, like environment provisioning time, features deployment rate, and a series of build, integration, and deployment metrics. However, it is imperative to supplement these with others that are more aligned to customer-centricity – delivering user-ready software faster with minimal errors at scale.
In addition to the metrics that are used to measure and improve various stages of the CI/CD pipeline, we also need to track several non-negotiable improvement measures. Many of these like deployment frequency, error rates at increased load, performance & load balancing, automation coverage of delivery process and recoverability helps to ascertain the efficiency of QA scale up.
Closely following on the heels of an earlier point, an outcome based model which maps financials to your engagement objectives will help to track outcomes to a large extent. While the traditional T&M model is governed by transactional metrics, project overlays abound in cases where engagement scope does not align well to outcome expectations. An outcome-based model also pushes the partner to bring in innovation through AI/ML and similar new-age technology drivers – providing you access to such skillsets without the need for having them on your rolls.
If you are new to outsourcing or working with a new partner, it may be good to start with a non-critical aspect of the work (for regular testing or automation), establish the process and then scale the engagement. For those players having maturity in terms of adopting outsourced QA functions in some way or the other, the steps outlined earlier form an all-inclusive checklist to ensure maximization of engagement traction and effectiveness with the outsourcing partner.
Partner with us
Trigent’s experienced and versatile Quality Assurance and Testing team is a major contributor to the successful launch, upgrade, and maintenance of quality software used by millions around the globe. Our experienced responsible testing practices put process before convenience to delight stakeholders with an impressive industry rivaled Defect Escape Ratio or DER of 0.2.
Trigent is an early pioneer in IT outsourcing and offshore software development business. We enable organizations to adopt digital processes and customer engagement models to achieve outstanding results and end-user experience. We help clients achieve this through enterprise-wide digital transformation, modernization, and optimization of their IT environment. Our decades of experience, deep domain knowledge, and technology expertise deliver transformational solutions to ISVs, enterprises, and SMBs.
As software product development is getting complex with the demand for breakthrough features and functionalities, software testing techniques are getting even more complex. The introduction of new features opens the door for many test case scenarios. Besides, there are various combinations of platforms, browsers, and devices that need to be tested for each scenario.
We know software product testing is not a new fad; it has seen its fair share of transition from manual testing to test automation and likewise testing tools have also evolved. However, as the market is abuzz with innovative products and platforms, businesses are constantly slogging to be front runners in efficiency and customer experience. Testing is no longer a ritual but a market readiness strategy for any respectable product or application.
New age start-ups have taken the market by storm, as the demand to release quality products faster has become increasingly important. Delay in launch of a product and you succumb to laggard status, while a minor defect and you invite customers and social media scalpel.
Given the demand and shift towards high-quality standards, a large number of software testing companies in India are investing time and money to improve customer experience and satisfaction. Be it a cloud platform or on-premise platform, Software product testing companies make use of the latest technologies and tools to get insights and feedback about the quality of their product. You can embed product testing either as an extension to your internal team or engage an independent validator whose approach would be structured and unbiased.
Outsourcing software product testing
The best strategy to address software product testing is to engage an offshore independent testing vendor. It would not only provide an independent eye but also reduce investments in terms of resource utilization. Other benefits would include greater return on quality assurance, augmented efficiency, flexibility, and minimized revenue cycle.
Before outsourcing your projects to software performance testing companies, do thorough research to see if the company has the required skills, experience, and expertise. It is important to find the right vendor with the right profiling in the performance testing strategy, as it would help your company optimize products and meet the high-quality standards of today’s demanding customers.