4 Rs for Scaling Outsourced QA. The first steps towards a rewarding engagement

Expanding nature of products, need for faster releases to market much ahead of competition, knee jerk or ad hoc reactions to newer revenue streams with products, ever increasing role of customer experience across newer channels of interaction, are all driving the need to scale up development and testing. With the increased adoption of DevOps, the need to scale takes a different color altogether.

Outsourcing QA has become the norm on account of its ability to address the scalability of testing initiatives and bring in a sharper focus on outcome-based engagements. The World Quality Report 2020 mentions that 34% of respondents felt QA teams lack skills especially on the AI/ML front. This further reinforces their need to outsource for getting the right mix of skill sets so as to avoid any temporary skill set gaps.

However, ensuring that your outsourced QA gives you speed and scale can be a reality only if the rules of engagement with the partner are clear. Focusing on 4 R’s as outlined below while embarking on the outsourcing journey, will help you derive maximum value.

  1. Right Partner
  2. Right Process
  3. Right Communication
  4. Right Outcome

Right Partner

The foremost step is to identify the right partner, one with a stable track record, depth in QA, domain as well as technology, and the right mix of skill sets across toolsets and frameworks. Further, given the blurring lines between QA and development with testing being integrated across the SDLC, there is a strong need for the partner to have strengths across DevOps, CI/CD in order to make a tangible impact on the delivery cycle.

The ability of the partner to bring to table prebuilt accelerators can go a long way in achieving cost, time and efficiency benefits. The stability or track record of the partner translates to the ability to bring onboard the right team which stays committed throughout the duration of the engagement. The team’s staying power assumes special significance in longer duration engagements wherein shifts in critical talent derails efficiency and timelines on account of challenges involved with newer talent onboarding and effective knowledge transfer.

An often overlooked area is the partner’s integrity. During the evaluation stages, claims pertaining to industry depth as well as technical expertise abound and partners tend to overpromise. Due care needs to be exercised to know if their recommendations are grounded in delivery experience. Closer look at the partner’s references and past engagements not only help to gain insight into their claims but also help to evaluate their ability to deliver in your context.

It’s also worthwhile to explore if the partner is open to differentiated commercial models that are more outcome driven and based on your needs rather than being fixated on the traditional T&M model.

Right Process

With the right partner on board, creating a robust process and governing mechanism assumes tremendous significance. Mapping key touchpoints from the partner side, aligning them to your team, and identifying escalation points serve as a good starting point. With agile and DevOps principles having collaboration across teams as the cornerstone, development, QA, and business stakeholder interactions should form a key component of the process. While cross-functional teams with Dev QA competencies start off each sprint with a planning meeting, formulating cadence calls to assess progress and setting up code drop or hand off criteria between Dev and QA can prevent Agile engagements from degrading into mini waterfall models.

Bringing in automated CI/CD pipelines obviates the need for handoffs substantially. Processes then need to track and manage areas such as quality and release readiness, visibility across all stages of the pipeline through reporting of essential KPIs, documentation for managing version control, resource management, and capacity planning. At times, toolset disparity between various stages and multiple teams driving parallel work streams creates numerous information silos leading to fragmented visibility at the product level. The right process should focus on integration aspects as well to bridge these gaps. Each team needs to be aware and given visibility on ownership at each stage of the pipeline.

Further, a sound process also brings in elements of risk mitigation and impact assessment and ensures adequate controls are built into SOP documents to circumvent any unforeseen event. Security measures is another critical area that needs to be incorporated into the process early on, more often it is an afterthought in the DevOps process. Puppet 2020 State of DevOps report mentions that integrating security fully into the software delivery process can quickly remediate critical vulnerabilities – 45% of organizations with this capability can remediate vulnerabilities within a day.

Right Communication

Clear and effective communication is an integral component of QA, more so when DevOps, Agile, and similar collaboration-heavy initiatives are pursued achieving QA at scale. Effective communication at the beginning of the sprint ensures that cross-functional teams are cognizant of the expectations from each of them and have their eye firmly fixed on the end goal of application release. From then on, a robust feedback loop, one that aims at continuous feedback and response, cutting across all stages of the value chain, plays a vital role in maintaining the health of the DevOps pipeline.

While regular stand-up meetings have their own place in DevOps, effective communication needs to go much beyond to focus on tools, insights across each stage, and collaboration. A wide range of messaging apps like Slack, email, and notification tools accelerate inter-team communication. Many of these toolkits are further integrated with RSS feeds, google drive, and various CI tools like Jenkins, Travis, Bamboo, etc. making build pushes and code change notifications fully automated. Developers need notifications when a build fails, testers need them when a build succeeds and Ops need to be notified at various stages depending on the release workflow.

The toolkits adopted by the partner also need to extend communication to your team. At times, it makes sense for the partner to have customer service and help desk support as an independent channel to accept your concern. The Puppet report further mentions that companies at a high level of DevOps maturity use ticketing systems 16% more than what is used by companies at the lower end of the maturity scale. Communication of the project’s progress and evolution to all concerned stakeholders is integral irrespective of the platforms used. Equally important is the need to categorize communication in terms of priority and based on what is most applicable to classes of users.

Documentation is an important component of communication and from our experiences, commonly underplayed. It is important for sharing work, knowledge transfer, continuous learning and experimentation. Code that is well documented enables faster completion of audit as well. In CI/CD based software release methodology, code documentation plays a strong role in version control across multiple releases. Experts advocate continuous documentation as core communication practice.

Right Outcome

Finally, it goes without saying that setting parameters for measuring the outcome, tracking and monitoring those, determines the success of the partner in scaling your QA initiatives. Metrics like velocity, reliability, reduced application release cycles and ability to ramp up/ramp down are commonly used. Further, there are also a set of metrics aimed at the efficiency of the CI/CD pipeline, like environment provisioning time, features deployment rate, and a series of build, integration, and deployment metrics. However, it is imperative to supplement these with others that are more aligned to customer-centricity – delivering user-ready software faster with minimal errors at scale.

In addition to the metrics that are used to measure and improve various stages of the CI/CD pipeline, we also need to track several non-negotiable improvement measures. Many of these like deployment frequency, error rates at increased load, performance & load balancing, automation coverage of delivery process and recoverability helps to ascertain the efficiency of QA scale up.

Closely following on the heels of an earlier point, an outcome based model which maps financials to your engagement objectives will help to track outcomes to a large extent. While the traditional T&M model is governed by transactional metrics, project overlays abound in cases where engagement scope does not align well to outcome expectations. An outcome based model also pushes the partner to bring in innovation through AI/ML and similar new age technology drivers – providing you access to such skill sets without the need for having them on your rolls.

If you are new to outsourcing, or working with a new partner, it may be good to start with a non-critical aspect of the work (for regular testing or automation), establish the process and then scale the engagement. For those players having maturity in terms of adopting outsourced QA functions in some way or the other, the steps outlined earlier form an all inclusive checklist to ensure maximization of engagement traction and effectiveness with the outsourcing partner.

Partner with us

Trigent’s experienced and versatile Quality Assurance and Testing team is a major contributor to the successful launch, upgrade, and maintenance of quality software used by millions around the globe. Our experienced responsible testing practices put process before convenience to delight stakeholders with an impressive industry rivaled Defect Escape Ratio or DER of 0.2.

Trigent is an early pioneer in IT outsourcing and offshore software development business. We enable organizations to adopt digital processes and customer engagement models to achieve outstanding results and end-user experience. We help clients achieve this through enterprise-wide digital transformation, modernization, and optimization of their IT environment. Our decades of experience, deep domain knowledge, and technology expertise delivers transformational solutions to ISVs, enterprises, and SMBs.

Contact us now.

Accelerate CI/CD Pipeline Blog Series – Part II – Test Automation

In part I of this blog series we spoke about Continuous Testing (CT), CI CD and that Test Automation is a key to its success, how to leverage test automation to enable coverage and speed. Let’s get an in-depth understanding of why it’s essential.

Why Automation of Testing is Essential for CI CD

Code analysis tools, API testing tools, API contract testing tools, Service virtualization tools, performance assessment tools, and end-to-end automation tools are all parts of CT. However, test automation is one of the key enablers of CT as:

  • It allows us to execute tests in parallel, across multiple servers/containers, speeding up the testing process.
  • It frees engineers from repetitive tasks, enabling them to focus on value-adds.
  • It helps validate the impact of even minor changes continuously.

Characteristics of a Good Automation Framework

For test automation to be effective, it must be supported by efficient frameworks. The frameworks provide a structure and help integrate and realize the efforts of a distributed team.

  • Test frameworks must support various languages, browsers, and techniques to make them future-proof. They must possess an Agile testing environment to support the continuous delivery pipeline.
  • The testing platform must be scalable to support as many tests as needed.
  • The platform must support compatibility tests on simulated devices to cut down the cycle.
  • The environment should maximize automation, i.e., trigger tests, analyze results, and share test information across the organization, in a fully automated manner.
  • The testing platform must include security features such as encryption of test data and access control policies

Characteristics of Trigent’s AutoMate Test Automation Framework

Reinforced by globally renowned partners, Trigent’s testing focus is aligned with the current business environment and offers cost benefits, performance, and agility.

As niche test automation experts, we have significant experience in open source and commercial testing tools. Our extensive library of modular, reusable, and resilient frameworks simplifies scenario-based automation. We provide on-demand testing and next-gen scheduling.

Features and benefits
  • Accelerated script development: Script/test cases development effort reduced up to 60-80% in comparison to traditional test automation approaches. Reduces Test Scripting Complexity.
  • Modular and reusable framework components: Reduced dependency on tool-specific resources. Ability to kick-start automation quickly. It supports reusable components. Reduced Test Automation maintenance costs. Allows multi-browser, multi-device testing.
  • Easy test script maintenance: Ease of test execution. Easy to make changes and maintain scripts in the long run. Improved error and exception handling. Supports multiple scripting languages.
  • On-demand Testing: Sanity, Smoke, Integration, Regression, etc. Provides effective test data creation on the go.
  • Hybrid Model: Modular test framework built using JUnit or TestNG. Integrates with multiple test automation tools. Allows a common way of handling multiple test types.
  • Scheduling and customized reporting: Send test results to ALM. Integrates with Test Management tools to track your test plans effectively.
  • Leverages new tech trends: Leverages AI/ML utilities & tools that allow for effective Test Impact analysis and test selection. Integrates with CI/CD tools to enable automated executions. Allows parallel executions to reduce test execution time

Learn more about Trigent software testing services or test automation services

Accelerate CI/CD Pipeline Blog Series – Part 1- Continuous Testing

Given its usefulness in software development, Agile methodologies have come to be embraced across the IT ecosystem to streamline processes, improve feedback, and accelerate innovation.

Organizations now see DevOps as the next wave after Agile that enables Continuous Integration and Continuous Delivery (CI/CD).  While Agile helped streamline and automate the entire software delivery lifecycle, CI/CD goes further. CI checks the code often, and the tested chunks are integrated, sometimes several times in a single day, to create a stream of smaller and frequent releases through CD.

As a principal analyst at Forrester Research puts it succinctly: ”If Agile was the opening act, continuous delivery is the headliner. The link that enables CI/CD, however, is Continuous Testing (CT).

What is Continuous Testing?

Continuous Testing is a process by which feedback on business risks of a software release is acquired as rapidly as possible. It helps in early risk identification & incremental coverage as it’s integrated into the delivery pipeline. Continuous Testing is achieved by making test automation an integral part of the software delivery pipeline. It’s seamlessly interwoven into the software delivery pipeline (not tagged at the end).

Though CI/CD enables speed-to-market, inadequate end-to-end experience testing can turn it into a liability.  A key aspect of CT is to leverage test automation to enable coverage and speed.

Test Automation – Continuous Testing’s Secret Success Factor

Automation of tests is the key to ensure that Quality Assurance is as continuous, agile, and reliable.  CT involves automating the tests and running them early and often.  It leverages service virtualization to increase the test coverage when parts of the business functions are available at different points in time.

Automated Testing binds together all the other processes that comprise the CD pipeline and makes DevOps work. By validating changing scenarios, Smart automation helps in faster software delivery.

In part II of the blog series we will talk more about why test automation is essential for CI/CD Testing, and automation framework.

Learn more about Trigent software testing services or test automation services

Why is Test Automation the Next Big Game-changer in the Insurance Industry?

In a world where technology is creating on-demand and bespoke experiences, the insurance industry is ready for disruption. Characterized until now by lengthy and manual processes and limited offerings, the industry has been quick to burst out of its protective bubble by leveraging new-age technologies including Blockchain, Artificial Intelligence (AI), extended reality, and quantum computing.

Key to accelerating this technology disruption is the InsurTech sector seen collaborating with insurance companies to facilitate nimbly and point solutions across the entire value chain. PwC Australia’s Global InsurTech report projects that the number of insurance providers partnering with InsurTechs will reach 84% between 2020 and 2022.[1]

However, in the race to transform their platforms and products, insurance companies—and their InsurTech partners—are facing several challenges. Primary among them are software and application testing challenges.

Insurers and InsurTechs experience different ‘testing’ difficulties

Digital self-service claims platforms, apps that reward policyholders based on their lifestyle and driving behaviors, and even wearables that monitor diabetes and reward condition management, are some of the trending innovations created by InsurTechs.[2] The companies behind these innovations mostly characterize themselves as digital economies. They evolve faster, are more embracing of Agile & DevOps, and have built products from the ground up are usually less complicated from the product stack that they manage. So when it comes to testing, their primary need is speed. To get to the market first, they need to identify failures, test, and evaluate faster than their competitors.

Furthermore, InsureTech players see themselves primarily as product/platform developers. This requires them to customize and dovetail their product into other complex systems. Testing, therefore, will have to extend beyond their primary product stack and include integration as well.

Insurance companies, on the other hand, have a completely different set of testing needs. Equipped with a substantial customer base, but challenged with changing customer needs, they are transforming to meet the digital, omnichannel, tailored offerings that can enrich the customer journey and experience. This leads them to integrate newer systems into their existing technology stack.

Trigent has successfully worked with several insurance companies and InsurTechs to shrink development cycles and increase the speed-to-market of applications. Contact us to know more about our AI-enabled test automation solution.

Testing needs will then have to be heavily focused on end-to-end validations to ensure that customer experiences are not compromised. Additionally, testing will also have to ensure that integration with complex legacy applications is maintained. At the same time, the performance and security of valuable customer data is not compromised when supporting their large customer bases.

Test automation—an essential tool in accelerating disruption

Test automation solves several of the above challenges. According to the World Quality Report 2019-2020[3], which surveyed 1725 CIOs and senior tech leaders, the main benefits reported were better control and transparency of test activities and detection of defects. The respondents also attributed test automation to gaining reduced test costs, test cycle time, and security risk.

For InsurTechs, like any other product innovator, deploying test automation fulfills an essential requirement—it assures improved coverage of their checks, ensuring faster release schedules. More importantly, test automation serves as a strong foundation during the integration of their product with larger insurance players. The features of a full-stack test automation framework, such as cross-technology and cross-platform readiness, and library customization, ensures that the marriage of services play with product play is effectively implemented.

Insurance companies leveraging test automation have experienced quantifiable benefits. A leading benefits administration and general insurance agency leveraged test automation software and reduced regression testing time by 85%. The process which covered 95% of the test coverage area led to a 45% reduced time-to-market rate.[4]

Test automation also plays a vital role in their transformation journey. For instance, when insurance companies are integrating new technologies, it simultaneously increases the testing workload. Vendors who manage their test outsourcing contracts are more interested in maintaining their SLAs and thus are slower in the uptake of the new changes. Test automation, essential to conduct acceptance tests is then best addressed through outside, neutral parties.

The PwC Insurance 2020 report[5] states that disruption is the new reality in the global insurance industry requiring industry players to keep pace with the sweeping social, technological, environmental, economic, and political developments that are categorizing this change. Implementing early test automation, at this juncture, will then work as the key differentiator for insurance companies and InsurTechs seeking to gain a head start in the digital ecosystems of the future.

Learn more about Trigent software testing services or test automation services

—————————————————————————————
[1] https://www.pwc.com.au/
[2] https://assets.kpmg/
[3] https://content.microfocus.com/
[4] https://www.trigent.com/
[5] https://www.pwc.co.za/en/

Getting Started – Selenium with Python Bindings

Introduction

Selenium Python binding provides a simple API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium WebDriver in an intuitive way. Selenium Python bindings also provide a convenient API to access Selenium WebDrivers such as Firefox, IE, Chrome, etc. The current supported Python versions are 2.7, 3.5, and above. In this blog, I will explain the Selenium 3 WebDriver API and in the next one, I will explain how to install and configure PyDev in Eclipse.

What is Python?

Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. It’s high-level built in data structures, combined with dynamic typing and binding, makes it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Python’s simple, easy-to-learn syntax emphasizes readability, and, therefore reduces the cost of program maintenance. Python supports modules and packages, which encourages program modularity and code reuse. The Python interpreter and the extensive standard library are available in source or binary form, free of charge, for all major platforms, and can be freely distributed.

Related: Define, Implement, Migrate and Script Automated Tests for End-to-end Automation of Applications Spanning Multiple Technologies.

What is Selenium?

Selenium is an open-source automation tool to test your web application. You can do this in various ways. For instance:

  • Selenium supports multiple languages such as Java, C#, Python, Ruby etc..
  • Selenium has components like, IDE, WebDriver.

Downloading Python bindings for Selenium

You can download Python bindings for Selenium from the PyPI page for the Selenium package. However, a better approach would be to use pip to install the selenium package. Python 3.6 has pip available in the standard library. Using pip, you can install selenium like this: `pip install selenium’. You may consider using virtualenv to create isolated Python environments. Python 3.6 has pyvenv which is almost the same as virtualenv.

Detailed instructions for Windows users

Note: You should have an Internet connection to perform this installation.

  1. Install Python 3.6 using the MSI available in python.org download page.
  2. Start a command prompt using the cmd.exe program and run the pip command as given below to install selenium.

C:Python35Scriptspip.exe install selenium.

Advantages of Python in Selenium

  1. Compared to other languages, Python takes less time to run the script and complete the execution.
  2. Python uses indentation, not braces ({}), making it easy to understand the code flow.
  3. Python is simpler and more compact.

Simple Usage

If you have installed Selenium Python bindings, you can start using it from Python in the following way:

from selenium import webdriver
 from selenium.webdriver.common.keys import Keys
 driver = webdriver.Firefox()
 driver.get("http://www.python.org")
 assert "Python" in driver.title
 elem = driver.find_element_by_name("q")
 elem.clear()
 elem.send_keys("pycon")
 elem.send_keys(Keys.RETURN)
 assert "No results found." not in driver.page_source
 driver.close()

If you found this interesting, don’t miss my next blog, ‘Install and Configure PyDev in Eclipse’.

Read Other Selenium related blogs:

Web Application Testing with Selenium WebDriver

Introduction to Selenium with C Sharp

Introduction to Selenium with C Sharp

Selenium WebDriver is a free and open-source library for automated testing web applications. Selenium tests can be written in multiple programming languages such as Java, C#, Python and Ruby and multiple browsers we execute scripts.

Prerequisite:

  • Download and install Visual Studio Frame Work.
  • Your target browser is installed, such as Chrome or Firefox.

Set up Visual Studio Solution:

Create a new project in Visual Studio Select template ‘Templates’ → ‘Visual C#’ → ‘Test’ → ‘Unit Test Project’.

Add Selenium WebDriver package to the project.

Run the following command in ‘Package Manager Console’ (selecting menu ‘Tools’ → ‘NuGet Package Manager’ → ‘Package Manager Console’).

After Selenium WebDriver is installed,  Solution References will appear like the following:

How to Create a test and run it:

  1. Add a new C# (a test) Class. Right mouse click the project name (in Solution Explorer) select ‘Add’ → ‘Unit Test’.
  2. Define the following class code to test and run it.
using System;
 using Microsoft.VisualStudio.TestTools.UnitTesting;
 using OpenQA.Selenium;
 using OpenQA.Selenium.Chrome;
 using OpenQA.Selenium.Firefox;
 using OpenQA.Selenium.Safari;

namespace TestAppDemo
{
[TestClass]
public class UnitTest2
{
static IWebDriver driverGC;
// static IWebDriver driverFF;
[AssemblyInitialize]
public static void setup(TestContext contest)
{
driverGC = new ChromeDriver(@”D:Documents StudySelenium detailschromedriver_win32″);
}
[TestMethod]
public void TestChromeDriver()
{
driverGC.Navigate().GoToUrl(“http://automationpractice.com/index.php”);
driverGC.FindElement(By.Id(“search_query_top”)).SendKeys(“CASUAL DRESSES”);
driverGC.FindElement(By.Id(“search_query_top”)).SendKeys(Keys.Enter);

}
[TestMethod]
public void TestClose()
{
driverGC.Quite();
}

}
}

Right Click on `class’ and select ‘Run Tests’.

After running successful, test case, it will show as below with pass mark:

We have to understand the below elements and their usages:

  • By Id: Finds the Webpage element by its unique control id.
  • By Name: Finds the Webpage element by the control name.
  • By Class: Finds the Webpage element by the Class name.
  • By LinkText: Finds the link element by its exact text displayed on web page
  • By partialLinkText: Finds webpage element that contains the given text
  • By TagName: Finds the Webpage element by the Tag name.
  • By XPath: This is most popular and majorly used for locating webpage element in a easiest way.
  • By CSS selector: Finds Webpage element by the CSS selectorEx.

What are the Advantages of Selenium Test using C#:

  • It is Open source.
  • It can be extended with different technologies.
  • In multiple browsers we execute scripts.
  • It supports multiple OS platforms and compatible with mobile devices.
  • Test executes within the browsers, so focus is not needed while executing the script.
  • Using Selenium Grids, you can execute tests in parallel.
  • Web Driver is faster than Selenium RC because it has simpler architecture.
  • Web Driver directly talks to the web browser whereas Selenium RC needs the help of the RC Server in order to do the same.

Disadvantages of Selenium Test:

  • It supports only web based applications.
  • It does not provide any facility for Object Repository.
  • It does not provide any facility for Recovery Scenario.
  • There is no report generation for test methods by default.
  • Users need to depend on minimum one programming language.

For more details, you can go through this click here.

Read Other Blogs related to Selenium:

Web Application Testing with Selenium WebDriver

Getting Started – Selenium with Python Bindings

Working with Test Driven Development

A technique for using automated unit test scenarios to design and decoupling of dependencies is called Test Driven Development. This technique is heavily used in Agile development methodologies. To use this technique with Visual Studio Team System, you have to understand the following topics:

  • Creating and running automated tests inside the VSTS IDE
  • Abstracting dependencies in an object-oriented world
  • Re-factoring new and old features to remove duplication in the code

The main theme of TDD is “Red, Green, Refactor.”

  • To work with TDD you must have a Test Project solution for creating new test cases in VSTS and it should reference the class library where you are adding new functionality.

Follow these steps to perform TDD:

  1. Understand the requirements of the user story and feature that you are working on.
  2. Red: Create a test and make it fail.
    1. Write the test for already existed code and change the logic of the code to make our test fail.
    2. Run the test case. It should fail. This means your test is calling the correct code and that the code is not working by properly. This is a meaningful failure, and you expect it to fail.
  3. Green: Make the test pass
    1. Initially hard code the expected return value to verify success after that write the correct logic of code to make the test pass.
    2. If you’ve written the code so that the test passes as intended, you are finished. You do not have to write more code speculatively. If new functionality is still needed, then another test is needed. Make this one test pass and continue.
    3. When the test passes, you might want to run all tests up to this point to build confidence that everything else is still working.
  4. Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.
    1. Remove duplication caused by the addition of the new functionality
    2. Make design changes to improve the overall solution.
    3. After each refactoring, rerun all the tests to ensure that they all still pass.

Advantages of Test-Driven Development

  • When the unit test case passes success and logic of the code is refactored to remove duplicate code.
  • The unit tests will not expire until you separate documentation and this will feedback that each component is still working.
  • TDD will help the Developer to understand the complete design and critical analysis of requirements to work with the logic of code and get an accurate result.
  • Any software developed using TDD can be maintained easily, better designed.
  • If a bug is found, the developer should create a test to reveal the bug and then modify the logic of the code until successive test run, all old bugs is verified and Reduced debugging time.

Limitations

  • Due to extensive use of test cases, all code logic is not executed success or failure properly.
  • Some Security Issues for Integration testing and Compliance testing

Read about Behavior Driven Development here.

Trigent’s experienced and versatile Quality Assurance and Testing team is a major contributor to the successful launch, upgrade, and maintenance of quality software used by millions around the globe. Our experienced responsible testing practices put process before convenience to delight stakeholders with an impressive industry rivaled Defect Escape Ratio or DER of 0.2.

Cucumber Test Automation Framework

Introduction:

Cucumber is a testing framework that helps to bridge the gap between software developers and business managers. Tests are written in plain language, based on the Behavior Driven Development (BDD) style of Given, When, Then, which anyone can understand. It is defined by a Language called “Gherkin”

Difference between TDD and BDD:

TDD: TDD (Test Driven Development) is an iterative development process. Each iteration starts with a set of tests written for a new piece of functionality.

BDD: BDD (Behavior Driven Development) is a synthesis and refinement of practices stemming from TDD. Like in TDD, in BDD also, we write tests first and then add application code.

The main objective of this automation framework is to provide the test script development environment, which is BDD based.

The major differences that we get to see here are:

  • Tests are written in plain descriptive English type grammar.
  • Tests are explained as the behavior of the application and are more user-focused.
  • Using examples to clarify requirements.

Prerequisites:

  • Install JDK and Eclipse on your machine
  • Install Cucumber in Eclipse.
  • Add J Unit to “build-path” in Eclipse.
  • Add Web Driver jar files to “build-path” in Eclipse.

What is “Gherkin”?

Gherkin is a simple, lightweight, and structured language that uses regular spoken language to describe requirements and scenarios.

Gherkin Keywords:

  • Feature: Each Gherkin file begins with a ”Feature” keyword. Feature defines the logical test functionality you will test in this feature file. e.g. if you are testing a “payment gateway”, your Feature will become “Payment Gateway”.
  • Scenario: Each Feature will contain some tests to test the feature. Each test is called a Scenario and is described using the Scenario keyword.
  • Given: Defines a “precondition” to the test.
  • When: This keyword defines the test activities that will be executed. By test action, we mean the user input action.
  • Then: This keyword defines the outcome of previous steps.
  • And: This keyword is used to add conditions to your steps.
  • But: This keyword is used to add negative type comments.
  • *: This keyword is very special. It defines the whole purpose of having Given When Then and all the other keywords.

Example:

Feature: Login Action Test
Description: This feature will test a Login and Logout functionality

Scenario 1:

Unsuccessful Login with Invalid Credentials

  1. Given User is on Homepage
  2. When User Navigate to Login Page
  3. And User enters Username and Password
  4. But The user credentials are wrong
  5. Then Message displayed Wrong Username & Password

Scenario 2: 

Successful Login with Valid Credentials

  1. Given User is on Homepage
  2. When User Navigate to Login Page
  3. And User enters Username and Password
  4. Then Message displayed Log in Successfully

Note:

  1. We need to write the “Step definition” in a java class for each and every behavior written in the ‘.feature’ file.
  2. All the web elements are declared in a Class(Objects.java) under “com.cucumber.commonlibs” package.
  3. All the constants are declared in a Class(login.java) under “com.cucumber.stepdefinition” package.
  4. The trigger of execution is declared in a Class(RunCucumberTest.java) under “com.cucumber.stepdefinition” package.

Step Definition:

  • A Step Definition is a small piece of code with a pattern attached to it.
  • In other words, a Step Definition is a java method in a class with an annotation above it. An annotation followed by the pattern is used to link the Step Definition to all the matching Steps.
  • Cucumber finds the “Step Definition” file, with the help of Glue code in Cucumber Options.

Example:

@Then("^Message displayed Login Successfully$")
 public void message_displayed_Login_Successfully() throws Throwable {
 System.out.println("Login Successfully");
 }

Advantages:

  • “Mail notification” enabled, on failure of build.
  • “Scheduled execution” can be performed.
  • Captures the “screen shots” promptly, on failure and embeds with the report.
  • Supports to run the execution, individual/suite wise.
  • OS independent and Browser independent.
  • Easy to understand.
  • Easy to use, highly flexible and above all “Free of Cost”.
  • Provides “micro-reporting”, with the statistics of passed, failed, pending, skipped, undefined and missing steps.

Power your next-gen software products through our automation testing services. Learn from our experts.

Appium Blog Series – Part 2 – Running Appium Automation Script on Android Emulators

About Appium

Appium is an Open Source mobile application UI testing framework, used to automate native, hybrid and web mobile apps. It enables cross-platform mobile app testing by using common API for both Android and iOS platform test scripts, thereby enhancing code-reusability. Appium provides flexibility of writing and running automation tests in any language such as, Java, C#, PHP, Ruby, Python, Perl, Objective-C.

About Android Emulator

Without an Android device, we can still run automation tests of Android Native App’s using an Emulator which mimics the OS and hardware of an Android device on a computer.

In Android SDK folder, there is an interface called as AVD Manager, which creates a virtual mobile device (Emulator) to run on a computer. Running Emulator starts a virtual console in which we can install Android apps.

In the following example, I will demonstrate automation of Android mobile phone’s default installed calculator app on Android Emulator.

Prerequisites for running Appium automation test for a native Android app on Emulator:

  • Java/JDK is installed and path for JAVA_HOME is set on the windows system.
  • Android SDK is installed and required packages of Android API’s are downloaded in SDK Manager.
  • Intel HAXM is installed.
  • Appium is installed.

Start Android Emulator:

In SDK folder on computer, run AVD Manager.exe, AVD Manager Dialog as shown below:

Click Create button, enter below details to create a new AVD.

AVD Manager shows newly created AVD as below, select this AVD and click Start.

In Launch Options dialog click Launch and in 10 to 15 minutes, Android Emulator will be started as below.

In eclipse create a Java project, with package name Android and class name AndroidEmulatorCalculator.

Set Desired Capabilities:

Following code snippet in AndroidEmulatorCalculator class, sets Desired Capabilities for Calculator App.

// Created object of DesiredCapabilities class.
 DesiredCapabilities capabilities = new DesiredCapabilities();

//Set android deviceName desired capability to Android Emulator.
capabilities.setCapability(“deviceName”, “Android Emulator”);

//Set BROWSER_NAME desired capability to Android.
capabilities.setCapability(“browserName”, “Android”);

//Set android VERSION from device desired capability.
capabilities.setCapability(“platformVersion”, “6.0”);

// Set android platformName desired capability to Android.
capabilities.setCapability(“platformName”, “Android”);

// Set android appPackage desired capability.
capabilities.setCapability(“appPackage”, “com.android.calculator2”);

// Set android appActivity desired capability.
capabilities.setCapability(“appActivity”, “com.android.calculator2.Calculator”);

Android Driver:

Following code snippet Creates an object of AndroidDriver class with Desired Capabilities as mentioned above and sets Appium server address with port number, and launches Android calculator app on device.

AndroidDriver driver;
 driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), capabilities);

The code snippet given below shows a simple addition on calculator app by using id tag to locate calculator app elements.

driver.findElement(By.id("com.android.calculator2:id/digit_4")).click();
 driver.findElement(By.id("com.android.calculator2:id/op_add")).click();
 driver.findElement(By.id("com.android.calculator2:id/digit_8")).click();
 driver.findElement(By.id("com.android.calculator2:id/eq")).click();
 String result = driver.findElement(By.className("android.widget.EditText")).getText();
 System.out.println("Result of Addition is : " + result);

Run Automation Test

Start Appium server, and run AndroidEmulatorCalculator class as TestNG test. Automation test will open calculator app on Emulator, clicks on buttons in this sequence -> 4, +, 8 and =.

Gets result from text area of calculator app, and print it in eclipse console.

Learn more why trigent is one of the best automation testing companies.

Read Other Appium related blogs

How to Automate Testing for Android Native Apps Using Appium

Using Appium automation script to Check SQLite DB Content of a Device!

Automated Device Certification Testing On Cloud AWS Device Farm with Appium

Appium – A Good Open Source Mobile App Testing Framework

Appium Blog Series – Part 1 – How to Automate Testing for Android Native Apps Using Appium

Appium is an open source mobile application user interface (UI) testing framework, and is used to automate native, hybrid and web mobile apps. It enables cross-platform mobile app testing by using common API for both Android and iOS platform test scripts, thereby enhancing code-re-usability. Appium provides flexibility of writing and running automation tests in languages such as, Java, C#, PHP, Ruby, Python, Perl, Objective-C.

Appium Architecture In Android:

Appium is a webserver that exposes a REST API. Appium client establishes connection with Appium Server through JSON Wire Protocol by creating a JSON object called as `Desired Capabilities’. Appium Server then creates an automation session by running a session id for the client, based on Key-Value pairs of Desired Capabilities set in Appium client. Appium Server then connects with the UI-Automator provided by Android SDK. UI-Automator communicates with Bootstrap.jar which will run user commands from the test script, on the mobile app in device.

Prerequisites for running Appium automation test for a native Android app:

  • Java/JDK is installed and path for JAVA_HOME is set on the windows system.
  • Android SDK is installed and required packages of Android API’s are downloaded in SDK Manager. Path for ANDROID_HOME, platform-tools is set on the windows system.
  • Eclipse IDE is installed. TestNG is installed.
  • Eclipse ADT Plugin is installed and configured to integrate android development environment in eclipse IDE.
  • Microsoft .Net Framework and Node JS is Installed.
  • Appium is installed.
  • Latest Selenium Webdriver jar files for java client, Appium Client library jar file, Google-gson jar file are downloaded.
  • Android Device is connected to your laptop/computer In USB Debugging mode.

In eclipse create a Java project, with package name Android and class name AndroidCalculator as shown below.

Add selenium webdriver’s jar files, Appium Client library jar file, Google-gson jar file in to java build path, by right clicking on project > Properties > Java Build Path > Libraries > Add External JARs >

Set Desired Capabilities:

We need to provide the following set of Desired Capabilities to provide Appium Server with details of automation session we want to set up:

  • Android Device Name – In command prompt, run command adb devices to show Android Device Name.
  • Android OS Version – In android device, Open settings -> About phone -> Android version
  • Android Software App Package Name, App Activity Name – In command prompt, run command adb logcat, and open Calculator App in Android phone and press ctrl+c. Log show’s App Package Name = com.android.calculator2, App Activity Name = com.android.calculator2.Calculator.

In class AndroidCalculator, following code snippet, creates a object of DesiredCapabilities class.

DesiredCapabilities capabilities = new DesiredCapabilities();

//Set android deviceName desired capability.
capabilities.setCapability(“deviceName”, “Q88HQ4S4SC89S8NB”);

//Set BROWSER_NAME desired capability to Android.
capabilities.setCapability(CapabilityType.BROWSER_NAME, “Android”);

//Set android VERSION from device desired capability.
capabilities.setCapability(CapabilityType.VERSION, “5.0”);

// Set android platformName desired capability to Android.
capabilities.setCapability(“platformName”, “Android”);

// Set android appPackage desired capability.
capabilities.setCapability(“appPackage”, “com.android.calculator2”);

// Set android appActivity desired capability.
capabilities.setCapability(“appActivity”, “com.android.calculator2.Calculator”);

Android Driver:

Following code snippet creates an object of AndroidDriver class with Desired Capabilities as mentioned above and sets Appium server address with port number, and launches android calculator app on device.

AndroidDriver driver;
 driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), capabilities);

Locate elements in Android App:

To locate elements in Android App, we use UI Automator Viewer tool provided by Android SDK.

  1. Run uiautomatorviewer.bat file from SDKtools folder.
  2. In Android device, open calculator app.
  3. In UI Automator Viewer tool, click on Device Screenshot image button.
  4. UI Automator Viewer now shows calculator app’s screenshot on left pane, and right pane show’s calculator app’s UI element’s hierarchy view with node structure that explain’s how elements are arranged. Ex: In calculator snapshot, button 4 is selected, and right pane shows different properties of button 4, which can be used in test script for locating button 4.

Code The snippet below shows  a simple addition on calculator app by using id tag to locate calculator app elements.

driver.findElement(By.id("com.android.calculator2:id/digit_4")).click();
 driver.findElement(By.id("com.android.calculator2:id/op_add")).click();
 driver.findElement(By.id("com.android.calculator2:id/digit_8")).click();
 driver.findElement(By.id("com.android.calculator2:id/eq")).click();
 String result = driver.findElement(By.className("android.widget.EditText")).getText();
 System.out.println("Result of Addition is : " + result);

Run Automation Test:

Start Appium server, and make configuration changes as shown in the below snapshots and click on Run button.

…to be continued.

Read Other Appium related blogs

Don’t miss my next blog on – Running Appium Automation Script on Android Emulators

Using Appium automation script to Check SQLite DB Content of a Device!

Automated Device Certification Testing On Cloud AWS Device Farm with Appium

Appium – A Good Open Source Mobile App Testing Framework

What are the best practices for Automated Regression Testing?

How it all Began ?

Period since the industrial revolution (1820) is referred by scientists as Anthropocene, also referred as Human Age. Significance of this period is automation of manual tasks using machines. Automation has helped mankind become more productive, self-sufficient and innovative.

The onset of Anthropocene era saw machines being used for manufacturing of textiles, paper, cement, chemicals, glass, stream engines, and automobiles and also used for agriculture. Today, you hardly find a household which doesn’t use technology/machines.

Since the invention of computers in the late 20th century, software has empowered humans to eliminate manual processes, introducing automation tools, where ever applicable. And till date, it’s ongoing.

Software development and testing

Software development lifecycle involves four important phases, namely requirement gathering, architecture and design, core development and testing. Software testing validates the functional and non-functional capability of the software. Today, software testing is mostly done manually by one or more person based on the size and complexity of the software.

Automated regression testing

During the initial stages of a software’s life cycle, it is valuable to have humans manually validate software’s capability and usability. However, as a software becomes mature it becomes a challenge in terms of time / cost to validate all aspects of the software manually. For a tester it becomes monotonous and is error prone when executing hundreds of test cases for every major software enhancements. So, it is essential to automate the tasks done by the tester to improve quality and thereby save time / cost. This process is referred as automated regression testing.

During the software development, it is essential that the test team build a regression suite from the very initial stage. This effort will pay high dividend during the later stage of the development and during software maintenance.

4 essential ingredients for an automated regression testing suite 

  1. Test database/environment on which the test scripts have to be executed
  2. Test scripts that translates the regression test case and has an expected result
  3. Test report that compares expected and actual result and
  4. Report that highlights the code covered by the regression suite

Test database has predefined data that are required for the software to run in a stable manner and for the test scripts to be executed successfully. Test team has to continuously update the test database as per the changes to the software.

Test scripts translate the hand written test cases to a re-executable script. There are multiple tools that helps in writing test script, the popular one’s are QTP, Winrunner, Selenium, Watir, SOA test, Testing Anywhere, etc. Selenium, Watir are open source and helps in building your own framework.

Every test script should have an assert statement to validate the expected result and the actual result match. In case they match it is a success else a failure. End of the regression test suite execution the test or project manager should review the report, ascertain the quality of the software and take appropriate action.

Validate test sufficiency:

Regression test suite should be constantly enhanced to reflect the changes to the software. Before enhancing one needs to know if the test scripts or test plans are sufficiently covering all the modules of the software. Code coverage tools helps in assessing the sufficiency of the test scripts. Some of the popular code coverage tools are,

For Java:

a)   EMMA (http://emma.sourceforge.net)

b)   Hansel (http://hansel.sourceforge.net)

c)   jCoverage (http://www.jcoverage.com)

d)   Cobertura (http://cobertura.sourceforge.net)

For .NET:

a)   OpenCover (https://github.com/sawilde/opencover)

b)   Clover.NET (http://www.cenqua.com/clover.net)

c)   NCover (http://ncover.sourceforge.net)

For PHP:

a)   PHP Test Coverage Tool

Our Software testing consulting and automation testing services takes into account all of the above factors. We are amongst a few automation testing companies who provide detailed reporting to validate Testing Sufficiency. Here’s a sample report.

Ensuring maximum test coverage and managing timeline – Software Quality Assurance

What is Software Quality Assurance?

“Quality Assurance” is the calling card for a software testing services provider. No matter how robust a software application is, its failure to perform at a critical instance can prove fatal to clients. Through the lens of history, it has been observed that enterprises spend more on bug fixes as compared to developing software applications.

Though an in-depth analysis of quality is outside the scope of this post, I will give a round-up on “Quality Assurance” by ensuring maximum test coverage as a means of achieving sustained client relationships.

One of the common challenges for a software testing team is to deliver projects within the time-frame yet ensure maximum test coverage. Software testing is not a one size fits all solution for problems as testers have to drill deeper to explore bug fixes that can damage the quality of an application. Though a solution to this is not readily available, however, we may need to arrive at one, based on the project circumstances/needs.

Before I set out for a solution, let us explore some of the possible risks/outcomes of not meeting the time-frame/maximum coverage:

a) Time Frame Overshoot: Time Consuming Delivery, Client Dissatisfaction, Extra Cost, Extended Time to market

b) No Coverage: No confidence in product quality, buggy product, user dissatisfaction, extra cost in fixing issues and re-releasing the product, etc.

“Time Limit” is a management activity, but from a tester’s perspective, we run on a pretty tight schedule and have to focus primarily on ensuring maximum coverage within the prescribed time limit. We have to ensure the application performs its function time after time. And to achieve this, we can develop a few handy utilities/tools like the one I have described below:

1) Develop a complete test suite, which includes:

  1. Test case for all functionality of the application
  2. Critical Functionality of the application being identified and filterable
  3. Prioritize test case as high/medium/low

[Benefits]: Allows to “Pick & Choose” test cases for execution based on:

  1. Critical module/function
  2. High Priority test cases
  3. Regression testing cases based on Change/Bug Fix in the current build

2) Optimize test scenarios:

  1. Eliminate repetitive and non-valuable tests that may yield the same results as other tests in the suite.
  2. Effective Coverage: Focus on combination testing using orthogonal array-based mathematical model tools like All Pairs, PICT, etc., and Decision Tables.

[Benefits]: Provides minimum scenarios covering all possible combinations which are non-repetitive and valuable in predicting the result.

3) Automated  Regression Suite – Automate all possible regression test cases

[Benefits]: Ensures execution and coverage of all mandatory flows without much manual intervention, even during crunch situations

4) Focused Testing: Apart from the above in case we have a very tight deadline, it is always preferable to focus the testing efforts on some critical areas like:

  1. Areas that may be vulnerable to changes and yield more bug fixes
  2. Areas that have yielded more bugs in the past
  3. Areas which are exposed to the End User or User Critical

Summary:

All the above utilities applied together will provide a cohesive framework to guide a tester or a development team to maximize test coverage and achieve greater quality within the stipulated time limit. However, it also depends on the tester’s knack for choosing the “Minimum most test cases” to draw upon insights that can help solve similar issues.

In the process, if the user over emphasizes minimizing the test cases to meet the time limit or if the area of focus is incorrect, he may miss out in terms of coverage and may lead to under testing. Hence the tester should wisely balance in choosing the right focus area and the right test cases for the build and plan to execute them within the given time frame. If the tester feels that the selected/shortlisted test cases cannot be run in the given time frame, it is always advisable to buy more time. And in case that is not possible, it’s prudent to alert the stakeholders about what test cases have been ignored and which modules are under-tested.

Ignore “Quality Assurance” in your services road-map and you do so at your own peril.