A Static Analyzer for Detecting Tensor Shape Errors in Deep Neural Network Training Code

  title={A Static Analyzer for Detecting Tensor Shape Errors in Deep Neural Network Training Code},
  author={Ho Young Jhoo and Sehoon Kim and Woosung Song and Kyuyeon Park and Dongkwon Lee and Kwangkeun Yi},
  journal={2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)},
  • Ho Young Jhoo, Sehoon Kim, K. Yi
  • Published 16 December 2021
  • Computer Science
  • 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)
We present an automatic static analyzer PyTea that detects tensorshape errors in PyTorch code. The tensor-shape error is critical in the deep neural net code; much of the training cost and intermediate results are to be lost once a tensor shape mismatch occurs in the midst of the training phase. Given the input PyTorch source, PyTea statically traces every possible execution path, collects tensor shape constraints required by the tensor operation sequence of the path, and decides if the… 

Gradual Tensor Shape Checking

A type-based approach to detect tensor shape mismatches that works practically under limitations of undecidability and satisfies the criteria of gradual typing proposed by Siek et al. in 2015.



Static Analysis of Shape in TensorFlow Programs

Pythia, a static analysis that tracks the shapes of tensors across Python library calls and warns of several possible mismatches, is presented, a close modeling of library semantics with respect to tensor shape and an identification of violations and error-prone patterns.

ShapeFlow: Dynamic Shape Interpreter for TensorFlow

ShapeFlow detects shape incompatibility errors highly accurately -- with no false positives and a single false negative -- and highly efficiently -- with an average speed-up of 499X and 24X for the first and second baseline, respectively.

Static Analysis of Python Programs using Abstract Interpretation: An Application to Tensor Shape Analysis

A subset of Python is specified with emphasis on tensor operations, based on The Python Language Reference, and an Abstract Interpreter is defined and its implementation presented, which shows how each part of the Abstract InterPreter was built: the Abstract Domains defined and the abstract semantics.

Ariadne: analysis for machine learning programs

A static framework, WALA, is applied to machine learning code that uses TensorFlow, a type system for tracking tensors—Tensorflow’s core data structures—and a data flow analysis to track their usage are created.

Semi-static type, shape, and symbolic shape inference for dynamic computation graphs

This paper proposes a novel approach for type, shape, and symbolic shape inference of dynamic computation graphs as a mixture of the above two methods and presents results of applying the prototype inference engine for networks written with PyTorch and demonstrates its effectiveness for nontrivial networks.

PyTorch: An Imperative Style, High-Performance Deep Learning Library

This paper details the principles that drove the implementation of PyTorch and how they are reflected in its architecture, and explains how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance.

Deep Networks with Stochastic Depth

Stochastic depth is proposed, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time and reduces training time substantially and improves the test error significantly on almost all data sets that were used for evaluation.

Array programming with NumPy

How a few fundamental array concepts lead to a simple and powerful programming paradigm for organizing, exploring and analysing scientific data is reviewed.

Strictly declarative specification of sophisticated points-to analyses

The DOOP framework for points-to analysis of Java programs is presented, carrying the declarative approach further than past work by describing the full end-to-end analysis in Datalog and optimizing aggressively using a novel technique specifically targeting highly recursive Datalogy programs.

P/Taint: unified points-to and taint analysis

This work proposes a deep unification of information-flow and points-to analysis, and demonstrates the benefits in a realistic implementation that leverages the Doop points- to analysis framework to provide an information-flows analysis with excellent precision and recall for standard Java information- flow benchmarks.