The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software

@article{Butler1993TheIO,
  title={The Infeasibility of Quantifying the Reliability of Life-Critical Real-Time Software},
  author={Ricky W. Butler and George B. Finelli},
  journal={IEEE Trans. Software Eng.},
  year={1993},
  volume={19},
  pages={3-12}
}
This work affirms that the quantification of life-critical software reliability is infeasible using statistical methods, whether these methods are applied to standard software or fault-tolerant software. The classical methods of estimating reliability are shown to lead to exorbitant amounts of testing when applied to life-critical software. Reliability growth models are examined and also shown to be incapable of overcoming the need for excessive amounts of testing. The key assumption of… 

Figures and Tables from this paper

POSSIBILITIES AND LIMITATIONS OF APPLYING SOFTWARE RELIABILITY GROWTH MODELS TO SAFETY- CRITICAL SOFTWARE
TLDR
By applying two of the most widely known software reliability growth models to sample software failure data, the possibility of using the software reliability grow models to prove the high reliability of safety-critical software is demonstrated.
Software Fault-Freeness and Reliability Predictions
TLDR
This work addresses how to combine evidence concerning probability of failure together with evidence pertaining to likelihood of fault-freeness, in a Bayesian framework, and guarantees reliability predictions that are conservative (err on the side of pessimism), despite the difficulty of stating prior probability distributions for reliability parameters.
Attaining High Confidence in Software Reliability Assessment
TLDR
The transformational approach to software reliability assessment is a novel methodology which combines the strengths of formal verification and statistical testing in a unified and original reliability assessment framework.
Toward dependable safety-critical software
TLDR
A rigorous method for assessing ultra high reliability requirements and then develops mechanisms for incorporating one of more backup versions is presented, which provides a confidence estimate for the correctness of the software for a given operational situation and serves as a trigger for switching to a backup version.
Importance Sampling Based Safety-Critical Software Statistical Testing Acceleration
TLDR
This paper discusses software statistical testing acceleration based on importance sampling technique, which shows that when both the critical operations and the entire software are adequately tested, the method can still get the unbiased software reliability from the test results with much less test cases.
Towards assessment of software reliability and its characteristics in safety systems of nuclear reactors
TLDR
Results observed in the case studies indicate that reliability estimates based on number of bugs present in software are likely to be inaccurate for safety-critical software, and the relationship between reliability and errors observed during dynamic analysis indicates that the average warnings and errors decrease exponentially as the reliability increases.
An approach to measuring and assessing dependability for critical software systems
  • D. Tang, H. Hecht
  • Computer Science
    Proceedings The Eighth International Symposium on Software Reliability Engineering
  • 1997
TLDR
A novel approach, drawing on findings and methods that have been described individually but have never been combined, applied in the late testing phase or early operational phase, to quantify dependability for a category of critical software with such high requirements.
Impact of program transformation on software reliability assessment
  • F. Bastani, B. Cukic
  • Computer Science
    Proceedings. IEEE High-Assurance Systems Engineering Workshop (Cat. No.96TB100076)
  • 1996
TLDR
This work identifies program transformations and partial proofs which have a measurable impact on the reliability assessment procedure and reduces the effective size of the input space which can facilitate sampling without replacement, thereby increasing the confidence in the reliability estimate.
Assessment of a sampling method for measuring safety-critical software reliability
  • F. Bastani, A. Pasquini
  • Computer Science
    Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering
  • 1994
TLDR
The paper addresses the issue of using sampling to assess the reliability of safety-critical software and concludes that, in principle, a combination of testing and verification will be successful in measuring ultrahigh software reliability.
Reducing Safety-Critical Software Statistical Testing Cost Based on Importance Sampling Technique
TLDR
The simulated annealing algorithm for calculating optimum transition probabilities of the Markov chain usage model for reducing software statistical testing cost is presented and can still compute the unbiased software reliability from the test results with much less test cases.
...
...

References

SHOWING 1-10 OF 33 REFERENCES
Stochastic Reliability-Growth: A Model for Fault-Removal in Computer-Programs and Hardware-Designs
TLDR
The suggested model results in earlier fault-fixes having a greater effect than later ones, the faults which make the greatest contribution to the overall failure rate tend to show themselves earlier, and the DFR property between fault fixes being fixed earlier.
An empirical comparison of software fault tolerance and fault elimination
  • T. Shimeall, N. Leveson
  • Computer Science
    [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis
  • 1988
TLDR
A large-scale experiment comparing software fault tolerance and software fault elimination as approaches to improving software reliability is described, and it was found that n-version programming did not tolerate most of the faults detected by the fault elimination techniques.
Fault-Tolerant SoFtware Reliability Modeling
TLDR
The models are used to show that one method of creating fault-tolerant software systems, the Consensus Recovery Block, is more reliable than the other two, and it presents reliability models for each.
On the use and the performance of software reliability growth models
Predicting software reliability
  • B. Littlewood
  • Computer Science
    Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences
  • 1989
TLDR
It turns out that examination of accuracy of past predictions can be used to improve future predictions by a simple recalibration procedure, and sometimes this technique works dramatically well, and results are shown for some real software failure data.
Software reliability: Repetitive run experimentation and modeling
TLDR
Independently generated input data was used to verify that interfailure times are very nearly exponentially distributed and to obtain good estimates of the failure rates of individual errors and demonstrate how widely they vary.
An experimental evaluation of the assumption of independence in multiversion programming
TLDR
N-version programming has been proposed as a method of incorporating fault tolerance into software and it is revealed that the programs were individually extremely reliable but that the number of tests in which more than one program failed was substantially more than expected.
Making statistical inferences about software reliability
Failure times of software undergoing random debugging can be modelled as order statistics of independent but nonidentically distributed exponential random variables. Using this model inferences can
Evaluation of competing software reliability predictions
TLDR
Some techniques are presented which form the basis of a partial solution to the problem of knowing which, if any, of the competing predictions are trustworthy in a reliability growth context.
Software safety: why, what, and how
TLDR
This survey attempts to explain why there is a problem, what the problem is, and what is known about how to solve it.
...
...