Black-box and White-box Testing : Week 1 Quiz answers

 Introduction to Test Adequacy

1) Which of the following descriptions of testing terminology is true?

a) A test case is a set of test inputs.

b) A test case specification is a requirement to be satisfied by one or more test cases.

c) Test execution is the activity of executing test cases and evaluating their results.

d) A test suite is a set of test cases.

e) A test obligation is a partial test specification, requiring some property deemed important through testing.

f) A test adequacy criterion is a set of test obligations which predicates whether a <program, test suite> pair is satisfied or not.

ANS - 

b) A test case specification is a requirement to be satisfied by one or more test cases.

c) Test execution is the activity of executing test cases and evaluating their results.

d) A test suite is a set of test cases.

e) A test obligation is a partial test specification, requiring some property deemed important through testing.

f) A test adequacy criterion is a set of test obligations which predicates whether a <program, test suite> pair is satisfied or not.

2) Which of the following description of test adequacy criteria is not true?

a) An unsatisfied test obligation may provide some useful information about improving the test suite.

b) A test suite that satisfies a stringent adequacy criterion can prove the correctness of a program.

c) If test suite satisfies all the obligations of an adequacy criterion, we can gain some evidence of the thoroughness of testing.

ANS - b) A test suite that satisfies a stringent adequacy criterion can prove the correctness of a program.

3) Which of the following descriptions about coverage measurement is not true?

a) A coverage score may be useful in indicating the progress toward a thorough test suite.

b) The coverage score enables a comparison among different test suites in terms of their quality.

c) A coverage score can be used to reveal the trouble spots requiring more attention.

d) Measuring coverage may be harmful since it may lead to a false sense of confidence.

ANS - b) The coverage score enables a comparison among different test suites in terms of their quality.

4) In which situations can test adequacy criteria be useful?

a) To determine when it is safe to stop testing.

b) Automatic test generation.

c) Revealing missing tests.

d) Test selection, or as guidance on manual test generation.

ANS - 

b) Automatic test generation.

c) Revealing missing tests.

d) Test selection, or as guidance on manual test generation.

Factors Influencing Test Effectiveness

1) What is the testing artifact that determines if tests pass?

a) Oracle

b) Model / program

c) Test inputs

d) Specification

ANS - a) Oracle

2) Which of the following are not testing artifacts?

a) Test oracle

b) Coverage criteria

c) Specification

d) Program (model)

e) Test input

ANS - b) Coverage criteria

3) What are the four artifacts that constitute a complete adequacy criteria?

a) Program structure

b) Specification

c) Coverage criteria

d) Number of testers

e) Test suite

f) Oracle

ANS - 

a) Program structure

b) Specification

e) Test suite

f) Oracle

4) Jenny implemented a software component for her hobby IoT project that can cost her $1,000 when it goes wrong. To prevent a possible loss of her asset, she wrote her program very carefully. She even wrote 100 test cases and showed that her test inputs satisfy 80% MC/DC coverage. Despite her efforts, however, her system failed on the field. In this situation, what is the least appropriate thing to do to improve the effectiveness of testing?

a) Write more assertions in the program.

b) Change the structure of the program so that it gets easier to attain a higher MC/DC coverage score.

c) Identify the unsatisfied coverage obligations and create more tests to cover them.

d) Choose a more rigorous test adequacy criterion to be used during the testing process.

ANS - b) Change the structure of the program so that it gets easier to attain a higher MC/DC coverage score.

Program Structure: Reachability and Observability

1) What is true about the reachability and observability?

a) Some parts of the programs are inherently hard to reach.

b) For a test case to find a fault through testing, it is sufficient for the test execution to reach the fault.

c) For a test to find a fault, a fault must propagate to some point where it can be observed.

d) Most coverage criteria help to ensure observability.

ANS -

a) Some parts of the programs are inherently hard to reach.

c) For a test to find a fault, a fault must propagate to some point where it can be observed.

2) Which of the descriptions are true about the reachability and observability of the following code? Suppose that we can only observe the returned value.

int foo(int x, int y) {

  int z;

  if (x >= y) {

    z = x;      // S0

  } else {

    z = y;      // S1

  }

  z = z * 0;    // S2

  if (z < 0) {

    z = -z;     // S3

  }

  return z;

}

a) When S2 is faulty, it can be observed.

b) A fault in S0 can propagate to the return value.

c) Every statement (S0, S1, S2, and S3) in the program is reachable.

d) If S0 and S1 were faulty, they cannot be observed.

ANS -

a) When S2 is faulty, it can be observed.

d) If S0 and S1 were faulty, they cannot be observed.

Mutation Testing Revisited

1) Which of the following are true about mutation testing?

a) With mutation testing, you can know how much of the code structure you covered.

b) The mutation adequacy score shows the quality of tests; the higher the score is, the better the quality of the tests.

c) A mutant is killed when there exists one or more tests that can differentiate the output of the mutant and the original program.

d) Unlike in structural testing, 100% mutation coverage can guarantee the absence of bug in the program.

ANS - 

b) The mutation adequacy score shows the quality of tests; the higher the score is, the better the quality of the tests.

c) A mutant is killed when there exists one or more tests that can differentiate the output of the mutant and the original program.

2) Which of the following are true about mutation?

a) A mutant and the original program are always semantically different 

b) A mutation operator introduces a syntactic change to the program so that the mutant cannot be compiled.

c) A mutant and the original program are always syntactically different.

d) You only create one mutant for mutation testing.

ANS -

c) A mutant and the original program are always syntactically different.

3) Jenny wrote a small program and created 100 test cases.  All tests pass when run on the original program.  She also created a complete set of mutants from the original program to measure the adequacy of her test suite using mutation testing. After running all the test cases on each of the mutants, Jenny found that for each mutant at least one test case failed.  What should she do next?

a) She should create more mutants

b) She should write more test cases

c) She can stop writing tests for this set of mutants because all of the mutants are killed

ANS -

c) She can stop writing tests for this set of mutants because all of the mutants are killed

Program Structure and Fault Finding

1) Mutation can be an effective substitute for actual faults during the testing process.

a) True b) False

ANS - a) True

2) How does reachability impmact the effectiveness of Mutation testing (choose one)?

a) A mutation cannot be stubborn if the statement is easily reachable

b) Mutations in hard-to-reach statements are impossible to kill

c) Mutations in hard-to-reach statements are harder to kill

ANS - c) Mutations in hard-to-reach statements are harder to kill

3) How does observability impact the effectiveness of Mutation testing (choose one)?

a) Observability improves the mutation score by killing reachable, yet stubborn mutants

b) Observability provides better mutation scores than reachability

c) Observable outputs can be used to identify and kill hard-to-reach mutants

ANS - a) Observability improves the mutation score by killing reachable, yet stubborn mutants

4) Branch coverage is particularly effective in improving mutation testing through strong observability

a) False b) True

ANS - a) False

Test Oracles

1) True oracles are previous versions of the program that can always detect faults.

a) True b) False

ANS - b) False

2) Which of the following oracles is mostly used for anomaly detection?

a) Consistency oracle

b) Statistical oracle

c) Heuristic oracle

d) Property-based oracle

ANS - 

3) A heuristic oracle approximates correct program _____.

a) input b) output c) tests

ANS - b) output

4) How are software requirements used in property-based oracles?

a) The requirements are used to create tests for the program.

b) The properties that the oracle checks are expressed in the requirements.

ANS - 

b) The properties that the oracle checks are expressed in the requirements.

5) How are heuristics used in heuristic oracles?

a) Heuristics allow the approximation of correct program behavior.

b) Heuristics are used to create an approximation of the program under test.

ANS - a) Heuristics allow the approximation of correct program behavior.

6) Automated oracles are guaranteed to be correct.

a) True b) False

ANS - b) False

Oracles and Fault Finding

1) A mutant is equivalent to the program under test if...

a) their structure is similar.

b) they behave functionally equivalently for all inputs.

c) the mutant can be used instead of the program without negative consequences.

ANS - b) they behave functionally equivalently for all inputs.

2) A strong oracle may provide more equivalent mutants than a weaker one.

a) True b) False

ANS - b) False

3) A test suite is more effective in exposing faults when using a ____ oracle.

a) stronger b) Weaker

ANS - a) stronger

4) Weaker oracles tend to lead to better mutation scores.

a) True b) False

ANS - b) False

Partition Testing

1) In Partition Testing, the idea is to fragment the ___ space into subsets, in such a way that some of these are ___ in terms of values that expose faults.

a) code; sparse

b) input; dense

c) output; dense

d) test; sparse

ANS - b) input; dense

2) The partitions that are constructed are always domain ______

a) independent b) dependent

ANS - b) dependent

3) In practice, every program fault can lead to a failure-dense partition.

a) False b) True

ANS - a) False

4) In Quasi-partitioning, overlapping partitions are allowed to exist.

a) False b) True

ANS - b) True