Compiler validation via equivalence modulo inputs

@article{Le2014CompilerVV,
  title={Compiler validation via equivalence modulo inputs},
  author={Vu Le and Mehrdad Afshari and Zhendong Su},
  journal={Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation},
  year={2014}
}
  • Vu Le, M. Afshari, Z. Su
  • Published 9 June 2014
  • Computer Science
  • Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
We introduce equivalence modulo inputs (EMI), a simple, widely applicable methodology for validating optimizing compilers. [] Key MethodTo create a practical implementation of EMI for validating C compilers, we profile a program's test executions and stochastically prune its unexecuted code. Our extensive testing in eleven months has led to 147 confirmed, unique bug reports for GCC and LLVM alone. The majority of those bugs are miscompilations, and more than 100 have already been fixed. Beyond testing…

Figures and Tables from this paper

Finding compiler bugs via live code mutation
TLDR
A novel EMI technique that allows mutation in the entire program (i.e., both live and dead regions) and significantly increases the EMI variant space by removing the restriction of mutating only the dead regions.
Finding deep compiler bugs via guided stochastic program mutation
TLDR
At Athena, a guided, advanced mutation strategy based on Bayesian optimization, is introduced to generate diverse programs to more thoroughly exercise compilers and helps discover deep bugs that require elaborate mutations.
Finding Typing Compiler Bugs
TLDR
A testing framework for validating static typing procedures in compilers and presents two novel approaches ( type erasure mutation and type overwriting mutation ) that apply targeted transformations to an input program to reveal type inference and soundness compiler bugs respectively.
Haskell Compiler Testing Automation Based on Equivalence-Modulo-Inputs Method
  • Tianchi Li
  • Computer Science
    Proceedings of the 2019 International Conference on Modeling, Simulation and Big Data Analysis (MSBDA 2019)
  • 2019
TLDR
This paper introduces an approach to generate Haskell programs as EMI test input of GHC by solving the two Haskell specified challenges.
SpecTest: Specification-Based Compiler Testing
TLDR
This work presents a novel specification-based testing method named SpecTest, built upon a novel test coverage criterion called semantic coverage which brings together mutation testing and fuzzing to specifically target less tested language features and can discover deep semantic errors in compilers.
High‐coverage metamorphic testing of concurrency support in C compilers
TLDR
It is shown that C4 achieves high strong mutation coverage with respect to a set of concurrency‐related mutants derived from a recent version of LLVM and that it can find historic concurrency-related bugs in GCC.
An Empirical Comparison of Compiler Testing Techniques
  • Junjie Chen, W. Hu, B. Xie
  • Computer Science
    2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)
  • 2016
TLDR
A systematic and comprehensive empirical comparison of three compiler testing techniques, namely, Randomized Differential Testing (RDT), a variant of RDT—Different Optimization Levels (DOL), and Equivalence Modulo Inputs (EMI), indicates that DOL is more effective at detecting bugs related to optimization, whereas RDT is moreeffective at detecting other types of bugs.
Coverage Prediction for Accelerating Compiler Testing
TLDR
The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction), and it is demonstrated that COP significantly accelerates compiler testing, achieving an average of 51.01 percent speedup in test execution time on an existing dataset including three old release versions of the compilers and a new dataset including 12 latest release versions.
Many-core compiler fuzzing
TLDR
This study provides independent validation of claims in prior work related to the effectiveness of random differential testing and EMI testing, proposes novel methods for lifting these techniques to the many-core setting and reveals a significant number of OpenCL compiler bugs in commercial implementations.
Test-case reduction and deduplication almost for free with transformation-based compiler testing
TLDR
This work presents a formulation of transformation-based compiler testing that provides effective test-case reduction almost for free: if transformations are designed to be as small and independent as possible, standard delta debugging can be used to shrink a bug-inducing transformation sequence to a smaller subsequence that still triggers the bug.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 32 REFERENCES
Translation validation for an optimizing compiler
TLDR
A practical translation validation infrastructure, able to check the correctness of many of the transformations performed by a realistic compiler, can be implemented with about the effort typically required to implement one compiler pass.
Finding and understanding bugs in C compilers
TLDR
Csmith, a randomized test-case generation tool, is created and spent three years using it to find compiler bugs, and a collection of qualitative and quantitative results about the bugs it found are presented.
Test-case reduction for C compiler bugs
TLDR
It is concluded that effective program reduction requires more than straightforward delta debugging, so three new, domain-specific test-case reducers are designed and implemented based on a novel framework in which a generic fixpoint computation invokes modular transformations that perform reduction operations.
Random Testing of C Compilers Targeting Arithmetic Optimization
TLDR
This paper presents a method of testing valid- ity of arithmetic optimization of C compilers using ran- dom programs and a method for automatic minimization of error programs which expedites the analysis of detected errors.
Formal verification of SSA-based optimizations for LLVM
TLDR
This paper develops a proof technique for proving SSA-based program invariants and compiler optimizations and uses this technique in the Coq proof assistant to create mechanized correctness proofs of several "micro" transformations that form the building blocks for larger SSA optimizations.
Fuzzing with Code Fragments
TLDR
LangFuzz is an effective tool for security testing: Applied on the Mozilla JavaScript interpreter, it discovered a total of 105 new severe vulnerabilities within three months of operation (and thus became one of the top security bug bounty collectors within this period); applied on the PHP interpreter, It discovered 18 new defects causing crashes.
Automated test program generation for an industrial optimizing compiler
TLDR
The script-driven test program generation process in JTT is shown, and how to produce test programs automatically, based on a temporal-logic model of compiler optimizations, to guarantee the execution of optimizing modules under test during compilation.
Evaluating value-graph translation validation for LLVM
TLDR
The design of a translation validator for LLVM's intra-procedural optimizations is presented, a design that does not require any instrumentation of the optimizer, nor any rewriting of the source code to compile, and needs to run only once to validate a pipeline of optimizations.
Equality saturation: a new approach to optimization
TLDR
The proposed way of structuring optimizers has a variety of benefits over previous approaches: it obviates the need to worry about optimization ordering, enables the use of a global optimization heuristic that selects among fully optimized programs, and can be used to perform translation validation, even on compilers other than the authors' own.
Scaling up Size and Number of Expressions in Random Testing of Arithmetic Optimization of C Compilers
TLDR
Experimental results show that a ran- dom test system based on the enhanced method has higher bug detection capability than existing methods; it has de- tected more bugs than previous method in earlier ver- sions of GCCs and has revealed new bugs in the latest versions of GCC's and LLVMs.
...
1
2
3
4
...