Code Defenders: Crowdsourcing Effective Tests and Subtle Mutants with a Mutation Testing Game

@article{Rojas2017CodeDC,
  title={Code Defenders: Crowdsourcing Effective Tests and Subtle Mutants with a Mutation Testing Game},
  author={Jos{\'e} Miguel Rojas and Thomas D. White and Benjamin S. Clegg and Gordon Fraser},
  journal={2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)},
  year={2017},
  pages={677-688}
}
Writing good software tests is difficult and not every developer's favorite occupation. Mutation testing aims to help by seeding artificial faults (mutants) that good tests should identify, and test generation tools help by providing automatically generated tests. However, mutation tools tend to produce huge numbers of mutants, many of which are trivial, redundant, or semantically equivalent to the original program, automated test generation tools tend to produce tests that achieve good code… 

Figures and Tables from this paper

Selection of Software Tests and Mutants with Contextual Bandits
TLDR
The findings identify the ability to learn test selection from a game-play setting, although the effect is parameter-dependent and not equally strong on all program sizes in the authors' experiments.
Teaching Software Testing with the Code Defenders Testing Game: Experiences and Improvements
  • G. Fraser, Alessio Gambi, J. Rojas
  • Computer Science
    2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
  • 2020
TLDR
This paper describes how Code Defenders was integrated into a software testing course, and how the game was improved in order to address the resulting challenges.
Gamifying a Software Testing Course with Code Defenders
TLDR
This paper describes how Code Defenders was integrated as a semester-long activity of an undergraduate and graduate level university course on software testing, addressing challenges such as selecting suitable code to test, managing games, and assessing performance.
The Influence of Test Suite Properties on Automated Grading of Programming Exercises
TLDR
It is shown that different properties of test suites can influence the grades that they produce, with coverage typically making the greatest effect, and mutation score and the potentially redundant repeated coverage of lines also having a significant impact.
Gamification of Software Testing
  • G. Fraser
  • Computer Science
    2017 IEEE/ACM 12th International Workshop on Automation of Software Testing (AST)
  • 2017
TLDR
Gamification is the approach of converting potentially tedious or boring tasks to components of entertaining gameplay, where the competitive nature of humans motivates them to compete and excel and has the potential to fundamentally change software testing in several ways.
A Preliminary Report on Gamifying a Software Testing Course with the Code Defenders Testing Game
TLDR
This paper has integrated Code Defenders as a mandatory component of a software testing course at the University of Passau, which featured ten game sessions of two hours each and involved 120 students.
EMVille: A gamification-based approach to address the equivalent mutant problem
  • Mahdi Houshmand, S. Paydar
  • Computer Science
    2017 7th International Conference on Computer and Knowledge Engineering (ICCKE)
  • 2017
TLDR
This paper proposes a gamification-based approach for improving the process of addressing the equivalent mutant problem by encouraging the experts to get involved in solving instances of this problem.
Collaborative Bug Finding for Android Apps
  • Shin Hwei Tan, Ziqiang Li
  • Computer Science
    2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)
  • 2020
TLDR
Coll collaborative bug finding helps us find 51 new bugs, in which eight have been confirmed and 11 have been fixed by the developers, confirming the intuition that the proposed technique is useful in discovering new bugs for Android apps.
Fostering good coding practices through individual feedback and gamification: an industrial case study
TLDR
This article explores how individual feedback and gamification can motivate developers to pay more attention to good coding practices and implemented these two concepts in a tool that was deployed at two large companies where a case study was conducted.
...
1
2
3
4
...

References

SHOWING 1-10 OF 57 REFERENCES
Code Defenders: A Mutation Testing Game
  • J. Rojas, G. Fraser
  • Computer Science
    2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
  • 2016
TLDR
The first steps towards building Code Defenders are introduced, a mutation testing game where players take the role of an attacker, who aims to create the most subtle non-equivalent mutants, or a defender, who aiming to create strong tests to kill these mutants.
Are mutants a valid substitute for real faults in software testing?
TLDR
This paper investigates whether mutants are indeed a valid substitute for real faults, i.e., whether a test suite’s ability to detect mutants is correlated with its able to detect real faults that developers have fixed, and shows a statistically significant correlation between mutant detection and real fault detection, independently of code coverage.
Covering and Uncovering Equivalent Mutants
TLDR
This paper examines whether changes in coverage can be used to detect non‐equivalent mutants: if a mutant changes the coverage of a run, it is more likely to be non‐Equivalent.
Automated unit test generation during software development: a controlled experiment and think-aloud observations
TLDR
In-depth analysis, using five think-aloud observations with professional programmers, confirms the necessity to increase the usability of automated unit test generation tools, to integrate them better during software development, and to educate software developers on how to best use those tools.
Feedback-Directed Random Test Generation
TLDR
Experimental results indicate that feedback-directed random test generation can outperform systematic and undirectedrandom test generation, in terms of coverage and error detection.
Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)
TLDR
Three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) are applied to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults.
Semi-automatic Search-Based Test Generation
  • Y. Pavlov, G. Fraser
  • Computer Science
    2012 IEEE Fifth International Conference on Software Testing, Verification and Validation
  • 2012
TLDR
A semi-automatic test generation approach based on the search-based EVO SUITE tool is presented, and the usefulness and potential on a set of example classes are evaluated.
Automatically detecting equivalent mutants and infeasible paths
TLDR
This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to detect some equivalent mutants and infeasible paths automatically.
Automatically Detecting Equivalent Mutants and Infeasible Paths
TLDR
This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to automatically detect some equivalent mutants and infeasible paths.
Is mutation an appropriate tool for testing experiments?
TLDR
It is concluded that, based on the data available thus far, the use of mutation operators is yielding trustworthy results (generated mutants are similar to real faults); Mutants appear however to be different from hand-seeded faults that seem to be harder to detect than real faults.
...
1
2
3
4
5
...