Compatibility Testing is Not All about Browsers and Operating Systems

In the world of software testing, whenever one talks about compatibility testing there will always be two words which recur `browsers’ & ‘operating systems’!

However, there are times when we come across a scenario where  we may have to apply compatibility testing technique to some areas which in turn opens doors of knowledge and provokes the tester to apply different techniques to assure test coverage.

Case study: 

We needed to create a simple stand-alone application  for secretaries of lawyers & lawyers themselves to create appointments & task lists for their regular activities.Requirement


End-users of the application were not fully comfortable with the application for creating appointments & tasks and they wanted the data created in the application to be available /synced with Outlook.

Our initial plan was to test the sync feature with the latest version of Outlook for functional testing and use a previous version of Outlook for compatibility testing.

However, we soon realized that our planned approach will fail when we were  informed that the age of end-users varies from 22 to 70 years, i.e. it will be used by young secretaries and lawyers who have little bit knowledge of software usage and other users who have hardly heard of Windows and Outlook updates! What they do have is basic computer skills,  which helps them to create appointments & task list and alerts them when required

This real time requirement brought forth the following challenges

  1. Is the application which syncs with Outlook compatible with all the older features of the OS?
  2. Does the application sync with all the newer & older versions of Outlook?
  3. Is Outlook compatible with the OS version which will be used for syncing?
  4. Will the appointment data be synced accurately when  created through Outlook?
  5. One of the lesser known features of Outlook is that it allows one to create calendars within a calendar or task list. The challenge was ensuring data is retained as it is after sync
  6. Backward compatibility testing on different Outlook versions by importing the data file (pst) which are created on some newer versions of Outlook and vice versa to ensure correct data retention
  7. Recurring appointment feature of Outlook has evolved  extensively offering multiple options for creating recurring appointments. The challenge was ensuring retention of recurrence pattern when appointments are synced with different versions of Outlook and assurance of the same when the data file is imported into different versions of Outlook.


From the challenges imposed by this application it was evident that we needed a thorough approach for compatibility testing.

It was evident that factors like Outlook version, Data files, recursive appointment features, Operating System need to be linked to create a unique platform that can carry out compatibility testing and along with it, we were required to ensure whether the application under test is compatible with different OSs.

Now the scope of the compatibility testing is on the multiple OS versions, multiple Outlook versions and also testing the backward compatibility which includes a lot of platforms to be considered for testing which requires a lot of effort and time.

When we derived the combinations of all the versions  we came up with forty five  combinations and it was highly impractical to do testing on all the platforms in the available short span of time. The question still remains how to perform effective testing in a limited time frame.

It was `all pairs’ testing technique, which is a combinatorial method of software testing which uses mathematical orthogonal array technique to derive the proper combination. The method helped us to nail down a combination to eight environments on which testing can be carried out and also helped to ensure that testing was done on every supported environment at least once with the stipulated time frame.


  1. We understood the fact that compatibility testing is not always about browsers & OSs.
  2. We gained knowledge of Outlook which is filled with features and has something for every user to use in his/her daily activity
  3. We gained expertise in one of the testing techniques which helped us to make our tests stronger with more coverage.


  • Ullas D

    He is currently working as Test lead in Trigent with more than 10 years experience in QA. He is involved in test planning, estimation & execution of functional testing & mobile testing efforts. Also, he is involved into project requirement analysis by playing role of Business analyst. He has experience in working on both testing & development services project and understands challenges involved in each phases and planning testing accordingly. He has good knowledge of Selenium webdriver.