An approach and benchmark to detect behavioral changes of commits in continuous integration

@article{Danglot2020AnAA,
  title={An approach and benchmark to detect behavioral changes of commits in continuous integration},
  author={Benjamin Danglot and Monperrus Martin and Walter Rudametkin and Beno{\^i}t Baudry},
  journal={Empirical Software Engineering},
  year={2020},
  volume={25},
  pages={2379-2415}
}
When a developer pushes a change to an application’s codebase, a good practice is to have a test case specifying this behavioral change. Thanks to continuous integration (CI), the test is run on subsequent commits to check that they do no introduce a regression for that behavior. In this paper, we propose an approach that detects behavioral changes in commits. As input, it takes a program, its test suite, and a commit. Its output is a set of test methods that capture the behavioral difference… 

Analysis of the behavioral impact of code modifications

A tool Code Review Helper is implemented that takes a pull request from GitHub, runs it and compares it to the stable version of the project, analysing its impact on the system’s execution, and presents the most relevant information in order to assist the reviewer in the review process.

Production Monitoring to Improve Test Suites

Pankti, an approach which monitors applications as they execute in production and then automatically generates differential unit tests, as well as derived oracles, from the collected data, focuses on one single programming language, Java.

Developer-Centric Test Amplification The Interplay Between Automatic Generation and Human Exploration

This paper conducts 16 semi-structured interviews with software developers supported by the prototypical designs of a developer-centric test amplification approach and a corresponding test exploration tool, and extends the test amplification tool DSpot, generating test cases that are easier to understand.

Developer-centric test amplification

This paper conducts 16 semi-structured interviews with software developers supported by the prototypical designs of a developer-centric test amplification approach and a corresponding test exploration tool, and extends the test amplification tool DSpot, generating test cases that are easier to understand.

Can We Spot Energy Regressions using Developers Tests?

This study investigates if the CI can leverage developers’ tests to perform a new class of test: the energy regression testing, similar to performance regression, but focused on the energy consumption of the program instead of standard performance indicators, like execution time or memory consumption.

Automatic Unit Test Amplification For DevOps

This thesis aims at addressing the lack of a tool that assists developers in regression testing by using test suite amplification, and proposes a new approach based on both test inputs transformation and assertions generation to amplify the test suite.

A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algorithms

A hierarchical approach is proposed to automatically compare the similarity of mapped statements and tokens by different algorithms to determine if each of the compared algorithms generates inaccurate mappings for a statement or its tokens.

AmPyfier: Test Amplification in Python

AmPyfier is presented, a proof-of-concept tool, which brings test amplification to the dynamically typed, interpreted language Python, and it is demonstrated that test amplification is feasible for one of the most popular programming languages in use today.

Amplification Automatique de Tests Unitaires pour DevOps

Au cours des dernieres annees, les tests unitaires sont devenus un element essentiel de tout projetlogiciel serieux afin de verifier son bon fonctionnement.Cependant, les tests sont fastidieux et

References

SHOWING 1-10 OF 46 REFERENCES

Automated Behavioral Regression Testing

  • Wei JinA. OrsoTao Xie
  • Computer Science
    2010 Third International Conference on Software Testing, Verification and Validation
  • 2010
Behavioral Regression Testing, a novel approach to regression testing that focuses on a subset of the code and leveraging differential behavior, can provide developers with more (and more detailed) information than traditional regression testing techniques.

Shadow of a Doubt: Testing for Divergences between Software Versions

A symbolic execution-based technique that is designed to generate test inputs that cover the new program behaviours introduced by a patch and evaluated on the Coreutils patches from the CoREBench suite of regression bugs shows that it is able to generatetest inputs that exercise newly added behaviours and expose some of the regression bugs.

Shadow Symbolic Execution for Testing Software Patches

A symbolic execution-based technique that is designed to generate test inputs that cover the new program behaviours introduced by a patch and evaluated on the Coreutils patches from the CoREBench suite of regression bugs shows that it is able to generatetest inputs that exercise newly added behaviours and expose some of the regression bugs.

BEARS: An Extensible Java Bug Benchmark for Automatic Program Repair Studies

BEARS, a project for collecting and storing bugs into an extensible bug benchmark for automatic repair studies in Java, is presented, and the version 1.0 of BEARS is delivered, which contains 251 reproducible bugs collected from 72 projects that use the Travis CI and Maven build environment.

KATCH: high-coverage testing of software patches

The results show that KATCH can automatically synthesise inputs that significantly increase the patch coverage achieved by the existing manual test suites, and find bugs at the moment they are introduced.

Semantics-assisted code review: An efficient tool chain and a user study

An invariant-mining tool chain, Getty, is created and it is demonstrated that semantically-assisted code review is feasible, effective, and that real programmers can leverage it to improve the quality of their reviews.

DiffGen: Automated Regression Unit-Test Generation

  • Kunal TanejaTao Xie
  • Computer Science
    2008 23rd IEEE/ACM International Conference on Automated Software Engineering
  • 2008
Experimental results show that the approach can effectively expose many behavioral differences that cannot be exposed by state-of-the-art techniques.

Automatic test improvement with DSpot: a study with ten mature open-source projects

This paper presents the concept, design and implementation of a system, that takes developer-written test cases as input (JUnit tests in Java) and synthesizes improved versions of them as output and shows that DSpot is capable of automatically improving unit-tests in real-world, large scale Java software.

How Open Source Projects Use Static Code Analysis Tools in Continuous Integration Pipelines

Study of the usage of static analysis tools in 20 Java open source projects hosted on GitHub and using Travis CI as continuous integration infrastructure reveals that build breakages are quickly fixed by actually solving the problem, rather than by disabling the warning, and are often properly documented.

Differential symbolic execution

A novel extension and application of symbolic execution techniques that computes a precise behavioral characterization of a program change that exploits the fact that program versions are largely similar to reduce cost and improve the quality of analysis results is introduced.