A comprehensive study of pseudo-tested methods

@article{VeraPrez2018ACS,
  title={A comprehensive study of pseudo-tested methods},
  author={Oscar Luis Vera-P{\'e}rez and Benjamin Danglot and Monperrus Martin and Beno{\^i}t Baudry},
  journal={Empirical Software Engineering},
  year={2018},
  volume={24},
  pages={1195-1225}
}
Pseudo-tested methods are defined as follows: they are covered by the test suite, yet no test case fails when the method body is removed, i.e., when all the effects of this method are suppressed. This intriguing concept was coined in 2016, by Niedermayr and colleagues, who showed that such methods are systematically present, even in well-tested projects with high statement coverage. This work presents a novel analysis of pseudo-tested methods. First, we run a replication of Niedermayr’s study… 
Rotten Green Tests
TLDR
An approach to identify rotten green tests by combining simple static and dynamic call-site analyses is described, which takes into account test helper methods, inherited helpers, and trait compositions, and has been implemented in a tool called DrTest.
Extreme mutation testing in practice: An industrial case study
  • M. Betka, S. Wagner
  • Computer Science
    2021 IEEE/ACM International Conference on Automation of Software Test (AST)
  • 2021
TLDR
This paper conducts an industrial case study consisting of running traditional and extreme mutation testing in a large software project from the semiconductor industry that is covered by a test suite of more than 11,000 unit tests and proposes that pseudo-tested code should be highlighted in code coverage reports andextreme mutation testing should be performed when writing unit tests rather than in a decoupled session.
Rotten green tests in Java, Pharo and Python
TLDR
An empirical evaluation of the tests in a corpus of projects found in the wild shows that rotten green tests are also present in Java and Python projects, and that they fall into similar categories.
Suggestions on Test Suite Improvements with Automatic Infection and Propagation Analysis
TLDR
Reneri, a tool that observes the program under test and the test suite in order to determine runtime differences between test runs on the original and the transformed method, is developed and processed by Reneri to suggest possible improvements to the developers.
A Method for Finding Missing Unit Tests
  • Daniel Gaston, J. Clause
  • Computer Science
    2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)
  • 2020
TLDR
This work is able to find what code is missing tests by identifying code entities which are not tested in the same way as other similar entities and shows how a code entity with a missing test should be tested by leveraging the tests written for those similar entities.
An approach and benchmark to detect behavioral changes of commits in continuous integration
TLDR
The approach DCI (Detecting behavioral changes in CI) works by generating variations of the existing test cases through assertion amplification and a search-based exploration of the input space and is able to generate test methods that detect behavioral changes.
Automatic Unit Test Amplification For DevOps
TLDR
This thesis aims at addressing the lack of a tool that assists developers in regression testing by using test suite amplification, and proposes a new approach based on both test inputs transformation and assertions generation to amplify the test suite.
The application perspective of mutatoin testing
TLDR
Improved the applicability of mutation by proposing a new speed-up approach and investigating the relationship between testability/observability and mutation testing, which extended the applicable of mutation testing in physical computing and GPU programming domains.
Exposing Test Analysis Results with DrTests
TLDR
DrTests is a new test browser that implements an extensible plugin-based architecture to run and report test results and supports the execution of rotten tests, comments to tests, coverage and profiling tests.
...
...

References

SHOWING 1-10 OF 28 REFERENCES
Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study
TLDR
There are large differences between the tools' effectiveness, ranging from 76% to 88%, with MUJAVA achieving the best results, and no tool is able to subsume the others and practical recommendations on how to strengthen each one of the studied tools are provided.
Programs, tests, and oracles: the foundations of testing revisited
TLDR
This work extends Gourlay's functional description of testing with the notion of a test oracle, an aspect of testing largely overlooked in previous foundational work and only lightly explored in general.
Will My Tests Tell Me If I Break This Code?
TLDR
An extreme mutation testing approach is applied to analyze the tests of open-source projects written in Java and shows that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system).
Assessing and Improving the Mutation Testing Practice of PIT
TLDR
It is shown that in real-world projects the original mutants of PIT are easier to kill and lead to tests that score statistically lower than those of the extended set of mutants for a range of 35% to 70% of the studied classes.
Is mutation an appropriate tool for testing experiments? [software testing]
TLDR
It is concluded that, based on the data available thus far, the use of mutation operators is yielding trustworthy results (generated mutants are similar to real faults); Mutants appear however to be different from hand-seeded faults that seem to be harder to detect than real faults.
State of Mutation Testing at Google
  • Goran Petrović, M. Ivankovic
  • Computer Science
    2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP)
  • 2017
TLDR
This work presents a diff-based probabilistic approach to mutation analysis that drastically reduces the number of mutants by omitting lines of code without statement coverage and lines that are determined to be uninteresting - these arid lines are dubbed.
Are mutants a valid substitute for real faults in software testing?
TLDR
This paper investigates whether mutants are indeed a valid substitute for real faults, i.e., whether a test suite’s ability to detect mutants is correlated with its able to detect real faults that developers have fixed, and shows a statistically significant correlation between mutant detection and real fault detection, independently of code coverage.
Does choice of mutation tool matter?
TLDR
An ensemble of measures for evaluating efficacy of mutants produced by different tools are investigated, including the traditional difficulty of detection, strength of minimal sets, and the diversity of mutants, as well as the information carried by the mutants produced.
Mutations: How Close are they to Real Faults?
TLDR
This paper investigates the characteristics of bug-fixes and other changes in a very large set of randomly selected projects using four different programming languages, and suggests that different languages have different distributions, which suggests that operators optimal in one language may not be optimal for others.
PIT: a practical mutation testing tool for Java (demo)
TLDR
PIT is a practical mutation testing tool for Java, applicable on real-world codebases and robust and well integrated with development tools, as it can be invoked through a command line interface, Ant or Maven.
...
...