The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software

@article{Butler1993TheIO,
  title={The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software},
  author={Ricky W. Butler and George B. Finelli},
  journal={IEEE Trans. Software Eng.},
  year={1993},
  volume={19},
  pages={3-12}
}
This work affirms that the quantification of life-critical software reliability is infeasible using statistical methods, whether these methods are applied to standard software or fault-tolerant software. The classical methods of estimating reliability are shown to lead to exorbitant amounts of testing when applied to life-critical software. Reliability growth models are examined and also shown to be incapable of overcoming the need for excessive amounts of testing. The key assumption of… 

Figures and Tables from this paper

Software Fault-Freeness and Reliability Predictions
TLDR
This work addresses how to combine evidence concerning probability of failure together with evidence pertaining to likelihood of fault-freeness, in a Bayesian framework, and guarantees reliability predictions that are conservative (err on the side of pessimism), despite the difficulty of stating prior probability distributions for reliability parameters.
Attaining High Confidence in Software Reliability Assessment
TLDR
The transformational approach to software reliability assessment is a novel methodology which combines the strengths of formal verification and statistical testing in a unified and original reliability assessment framework.
Towards assessment of software reliability and its characteristics in safety systems of nuclear reactors
TLDR
Results observed in the case studies indicate that reliability estimates based on number of bugs present in software are likely to be inaccurate for safety-critical software, and the relationship between reliability and errors observed during dynamic analysis indicates that the average warnings and errors decrease exponentially as the reliability increases.
An approach to measuring and assessing dependability for critical software systems
  • D. TangH. Hecht
  • Computer Science
    Proceedings The Eighth International Symposium on Software Reliability Engineering
  • 1997
TLDR
A novel approach, drawing on findings and methods that have been described individually but have never been combined, applied in the late testing phase or early operational phase, to quantify dependability for a category of critical software with such high requirements.
Assessment of a sampling method for measuring safety-critical software reliability
  • F. BastaniA. Pasquini
  • Computer Science
    Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering
  • 1994
TLDR
The paper addresses the issue of using sampling to assess the reliability of safety-critical software and concludes that, in principle, a combination of testing and verification will be successful in measuring ultrahigh software reliability.
Reducing Safety-Critical Software Statistical Testing Cost Based on Importance Sampling Technique
TLDR
The simulated annealing algorithm for calculating optimum transition probabilities of the Markov chain usage model for reducing software statistical testing cost is presented and can still compute the unbiased software reliability from the test results with much less test cases.
A reliability model combining representative and directed testing
  • Brian MitchellS. Zeil
  • Psychology
    Proceedings of IEEE 18th International Conference on Software Engineering
  • 1996
TLDR
A model is presented which permits representative and directed testing to be used in conjunction, using order statistics to combine the observed failure rates of faults no matter how those faults were detected.
Does Software Have to Be Ultra Reliable in Safety Critical Systems?
TLDR
Higher levels of safety performance can be claimed by taking account of: 1) external mitigation to prevent an accident: 2) the fact that software is corrected once failures are detected in operation.
Software Fault Tolerance: A Tutorial
TLDR
How hard-to-detect design faults are likely to be introduced during development and how software faults tend to be state-dependent and activated by particular input sequences are noted.
A software reliability model combining representative and directed testing
TLDR
The results of this work show that the high level of noise present in failure data based on observed failure times makes it very difficult for models that use this type of data to make accurate reliability estimates, support the suggested move to the use of more stable quantities for reliability estimation and prediction.
...
...

References

SHOWING 1-10 OF 33 REFERENCES
Stochastic Reliability-Growth: A Model for Fault-Removal in Computer-Programs and Hardware-Designs
TLDR
The suggested model results in earlier fault-fixes having a greater effect than later ones, the faults which make the greatest contribution to the overall failure rate tend to show themselves earlier, and the DFR property between fault fixes being fixed earlier.
An empirical comparison of software fault tolerance and fault elimination
  • T. ShimeallN. Leveson
  • Computer Science
    [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis
  • 1988
TLDR
A large-scale experiment comparing software fault tolerance and software fault elimination as approaches to improving software reliability is described, and it was found that n-version programming did not tolerate most of the faults detected by the fault elimination techniques.
Fault-Tolerant SoFtware Reliability Modeling
TLDR
The models are used to show that one method of creating fault-tolerant software systems, the Consensus Recovery Block, is more reliable than the other two, and it presents reliability models for each.
On the use and the performance of software reliability growth models
Predicting software reliability
  • B. Littlewood
  • Computer Science
    Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences
  • 1989
TLDR
It turns out that examination of accuracy of past predictions can be used to improve future predictions by a simple recalibration procedure, and sometimes this technique works dramatically well, and results are shown for some real software failure data.
Software reliability: Repetitive run experimentation and modeling
TLDR
Independently generated input data was used to verify that interfailure times are very nearly exponentially distributed and to obtain good estimates of the failure rates of individual errors and demonstrate how widely they vary.
An experimental evaluation of the assumption of independence in multiversion programming
TLDR
N-version programming has been proposed as a method of incorporating fault tolerance into software and it is revealed that the programs were individually extremely reliable but that the number of tests in which more than one program failed was substantially more than expected.
Making statistical inferences about software reliability
Failure times of software undergoing random debugging can be modelled as order statistics of independent but nonidentically distributed exponential random variables. Using this model inferences can
Evaluation of competing software reliability predictions
TLDR
Some techniques are presented which form the basis of a partial solution to the problem of knowing which, if any, of the competing predictions are trustworthy in a reliability growth context.
Software safety: why, what, and how
TLDR
This survey attempts to explain why there is a problem, what the problem is, and what is known about how to solve it.
...
...