In the world of software testing, whenever one talks about compatibility testing there will always be two words which recur `browsers’ & ‘operating systems’!
However, there are times when we come across a scenario where we may have to apply compatibility testing technique to some areas which in turn opens doors of knowledge and provokes the tester to apply different techniques to assure test coverage.
Case study:
We needed to create a simple stand-alone application for secretaries of lawyers & lawyers themselves to create appointments & task lists for their regular activities.Requirement
Challenge
End-users of the application were not fully comfortable with the application for creating appointments & tasks and they wanted the data created in the application to be available /synced with Outlook.
Our initial plan was to test the sync feature with the latest version of Outlook for functional testing and use a previous version of Outlook for compatibility testing.
However, we soon realized that our planned approach will fail when we were informed that the age of end-users varies from 22 to 70 years, i.e. it will be used by young secretaries and lawyers who have little bit knowledge of software usage and other users who have hardly heard of Windows and Outlook updates! What they do have is basic computer skills, which helps them to create appointments & task list and alerts them when required
This real time requirement brought forth the following challenges
- Is the application which syncs with Outlook compatible with all the older features of the OS?
- Does the application sync with all the newer & older versions of Outlook?
- Is Outlook compatible with the OS version which will be used for syncing?
- Will the appointment data be synced accurately when created through Outlook?
- One of the lesser known features of Outlook is that it allows one to create calendars within a calendar or task list. The challenge was ensuring data is retained as it is after sync
- Backward compatibility testing on different Outlook versions by importing the data file (pst) which are created on some newer versions of Outlook and vice versa to ensure correct data retention
- Recurring appointment feature of Outlook has evolved extensively offering multiple options for creating recurring appointments. The challenge was ensuring retention of recurrence pattern when appointments are synced with different versions of Outlook and assurance of the same when the data file is imported into different versions of Outlook.
Approach:
From the challenges imposed by this application it was evident that we needed a thorough approach for compatibility testing.
It was evident that factors like Outlook version, Data files, recursive appointment features, Operating System need to be linked to create a unique platform that can carry out compatibility testing and along with it, we were required to ensure whether the application under test is compatible with different OSs.
Now the scope of the compatibility testing is on the multiple OS versions, multiple Outlook versions and also testing the backward compatibility which includes a lot of platforms to be considered for testing which requires a lot of effort and time.
When we derived the combinations of all the versions we came up with forty five combinations and it was highly impractical to do testing on all the platforms in the available short span of time. The question still remains how to perform effective testing in a limited time frame.
It was `all pairs’ testing technique, which is a combinatorial method of software testing which uses mathematical orthogonal array technique to derive the proper combination. The method helped us to nail down a combination to eight environments on which testing can be carried out and also helped to ensure that testing was done on every supported environment at least once with the stipulated time frame.
Learning:
- We understood the fact that compatibility testing is not always about browsers & OSs.
- We gained knowledge of Outlook which is filled with features and has something for every user to use in his/her daily activity
- We gained expertise in one of the testing techniques which helped us to make our tests stronger with more coverage.
