One of the primary goals of software testing is to find the maximum number of defects (bugs) using a minimum set of test cases. There is a testing method that helps achieve this goal called the Orthogonal Array Testing Strategy (OATS). This is a statistical method aimed at maximizing error detection while reducing the number of test cases by using pair-wise testing.

What is pair-wise testing?

Let us assume that the software being tested has 5 variables each of which can assume 4 states. The number of test cases that will be required to test all possible combinations is 4 X 4 X 4 X 4 X 4 = 1024. Researchers at Bell Communication Laboratories in the late 1990s found that most defects were caused by incorrect values of a single variable or by values of certain pairs of interacting variables.

Defects caused by specific states or values of a single variable are called single-mode faults. This implies that we could detect all single-mode faults through 4 + 4 + 4 + 4 + 4 = 20 test cases.

Defects that occur when a combination of two variables assume specific states (values) are called double-mode faults. This is an indication of harmful interaction between two variables when they assume certain values. For purposes of illustration, let us assume that we are able to identify 6 variable-pairs with harmful interactions from among the 5 variables. Each variable-pair will require 4 X 4 = 16 test cases to exhaustively test all possible combinations. For 6 variable pairs, a total of 16 X 6 = 96 test cases will be required.

Thus, using the OATS methodology the total number of test cases required to detect all single and double-mode defects is only 20 + 96 = 116 as compared to 1024 test cases required using traditional test methods.

OATS, in effect, provides a statistical method to test pair-wise interactions through the use of orthogonal arrays. An orthogonal array is a rectangular matrix where rows represent test cases (runs) and columns represent variables (factors) being tested. The different states or values that each variable can assume are called levels. The underlying rationale of this technique is based on the hypothesis that almost all defects in software are due to either specific levels of a single factor or due to pair-wise interactions of factors at certain levels. One of the major challenges of this approach is to identify all pair-wise combinations that give rise to harmful interactions.

However the major advantages of OATS are as follows:

- Lower number of test cases.
- All single-mode and double-mode defects are uncovered which account for more than 95% of the bugs.
- Improved productivity due to reduced test cycles and testing times.

Dr. Prasad,

Good post. Not nearly enough software testers know about this powerful approach. Anyone interested in learning more about this topic might be interested in the following articles and tool:

IEEE Computer magazine article describing average benefits delivered by this approach over ten pilot projects:

https://hexawise.com/Combinatorial-Softwar-Testing-Case-Studies-IEEE-Computer-Kuhn-Kacker-Lei-Hunter.pdf

Our easy-to-use Pairwise (and n-wise) test case generator with free trial sign-ups: hexawise.com/users/new

Also, see the articles at combinatorialtesting.com

Thanks.

Justin Hunter

Founder of Hexawise

Posted by: Hexawise | Thursday, August 26, 2010 at 09:16 AM

Hi Dr. Prasad,

I like to learn more about it.Can you please help me inthat regard

Mail ID: rvimalsamsingh@gmail.com

Thanks in Advance

Posted by: vimal | Sunday, August 12, 2012 at 09:59 AM