Production Monitoring to Improve Test Suites

  title={Production Monitoring to Improve Test Suites},
  author={Deepika Tiwari and Long Zhang and Monperrus Martin and Beno{\^i}t Baudry},
  journal={IEEE Transactions on Reliability},
In this article, we propose to use production executions to improve the quality of testing for certain methods of interest for developers. These methods can be methods that are not covered by the existing test suite or methods that are poorly tested. We devise an approach called pankti which monitors applications as they execute in production and then automatically generates differential unit tests, as well as derived oracles, from the collected data. pankti’s monitoring and generation focuses… 

Figures and Tables from this paper

Mimicking Production Behavior with Generated Mocks

This paper proposes to monitor an application in production to generate tests that mimic realistic execution scenarios through mocks, and implements an open-source tool called RICK that monitors the invocation of 128 methods in production across the three applications and captures their behavior.

Harvesting Production GraphQL Queries to Detect Schema Faults

A new testing approach based on GraphQL queries that are logged while users interact with an application in production, and it is shown that production queries capture real usages of the application, and are known to trigger behavior that may not be tested by developers.



Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking

Results show that an automatically generated test suite's fault-detection capability can be effectively improved after being augmented by Orstra, and the augmented test suite has an improved capability of guarding against regression faults.

Improving test suites via operational abstraction

The operational difference technique for generating, augmenting, and minimizing test suites is presented, which is analogous to structural code coverage techniques, but it operates in the semantic domain of program properties rather than the syntacticdomain of program text.

Automatic test factoring for java

This work introduces an automatic technique for test factoring, which reduces the running time of a system test suite by up to an order of magnitude, and generates unit tests for T in which E is mocked.

Tests from traces: automated unit test extraction for R

Genthat, a tool developed over the last couple of years to non-invasively record execution traces of R programs and extract unit tests from those traces is presented.

SCARPE: A Technique and Tool for Selective Capture and Replay of Program Executions

A technique for capturing and replaying partial executions of deployed software and can be used for various applications, including generation of test cases from user executions and post-mortem dynamic analysis.

Will My Tests Tell Me If I Break This Code?

An extreme mutation testing approach is applied to analyze the tests of open-source projects written in Java and shows that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system).

Behavioral Execution Comparison: Are Tests Representative of Field Behavior?

Differences between field and test executions—and in particular the finer-grained and more sophisticated ones that are measured using the authors' invariantbased model—can provide insight for developers and suggest a better method for measuring test suite quality.

Automatic test improvement with DSpot: a study with ten mature open-source projects

This paper presents the concept, design and implementation of a system, that takes developer-written test cases as input (JUnit tests in Java) and synthesizes improved versions of them as output and shows that DSpot is capable of automatically improving unit-tests in real-world, large scale Java software.

GenUTest: a unit test and mock aspect generation tool

This paper proposes a technique which automatically generates unit tests for software that does not have such tests, and implemented GenUTest, a prototype tool which captures and logs interobject interactions occurring during the execution of Java programs, using the aspect-oriented language AspectJ.

The Oracle Problem in Software Testing: A Survey

This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.