Problems with scoring methods and ordinal scales in risk assessment

  title={Problems with scoring methods and ordinal scales in risk assessment},
  author={Douglas Hubbard and Dylan Evans},
  journal={IBM J. Res. Dev.},
Risk assessment methods based on scoring methods that rate the severity of each risk factor on an ordinal scale are widely used and frequently perceived by users to have value. [...] Key Method We then review the literature on the use of ordinal scales in risk analysis, the use of "verbal scales" for eliciting estimates of risks and probabilities, and the extensive research about peculiar human errors when assessing risks. We also supplement this overview with some data of our own. When these diverse kinds of…Expand

Figures and Topics from this paper

Can Public Health Risk Assessment Using Risk Matrices Be Misleading?
This work evaluated a practical public health example because it provided experiential risk data that allowed it to assess the practical implications of the published concern that risk matrices would make predictions that are worse than random. Expand
Recommendations on the use and design of risk matrices
Risk matrices are widely used in risk management. They are a regular feature in various risk management standards and guidelines and are also used as formal corporate risk acceptance criteria. It isExpand
On the assessment of uncertainty in risk diagrams
Abstract Risk matrices and risk diagrams are widely used tools for analyzing, assessing and visualizing risk in many industries, and are used extensively for risk management purposes. Despite theirExpand
Rules for a coherent real estate risk scoring
The purpose of this paper is to supplement the literature with general rules under which scoring systems should be designed and validated, so that they can become reliable risk instruments. Expand
Do experts agree when assessing risks? An empirical study
This paper presents two studies conducted in a large European airline and urges organizations to comprehend the distinction between experience and expertise, and focus on training their staff in published expert judgment methods. Expand
A Review of Risk Matrices Used in Acute Hospitals in England.
Findings show that hospitals may provide insufficient guidance on how to use risk matrices as well as what to do in response to the existing criticisms ofrisk matrices, which is likely to lead to variation in the quality of risk assessment and in the subsequent deployment of resources. Expand
Designing and exploring risk matrices with MACBETH
This paper studies ways in which multiple criteria and portfolio decision analyses can improve the design and deployment of risk matrices, using MACBETH, and sketches a new modeling approach for Improving Risk Matrices (IRIS). Expand
Occupational health and safety: Designing and building with MACBETH a value risk-matrix for evaluating health and safety risks
Risk matrices (RMs) are commonly used to evaluate health and safety risks. Nonetheless, they violate some theoretical principles that compromise their feasibility and use. This study describes howExpand
Quantifying Uncertainty in Expert Judgment: Initial Results
The domain-specific training was accompanied by notable improvements in the relative accuracy of the participants' answers when more contextual information to the questions was given along with reference points about similar software systems. Expand
Risk assessment in safety of machinery: Impact of construction flaws in risk estimation parameters
The experimental results show that when the users perceive a certain challenge in the utilization of a risk estimation parameter, they are usually able to associate it with the presence of the flaw affecting the parameter, and demonstrate quite clearly that the impact of the construction flaws in the parameters is not uniform. Expand


What's wrong with risk matrices?
  • L. A. Cox
  • Engineering, Medicine
  • Risk analysis : an official publication of the Society for Risk Analysis
  • 2008
Risk matrices-tables mapping "frequency" and "severity" ratings to corresponding risk priority levels-are popular in applications as diverse as terrorism risk analysis, highway construction projectExpand
Subjective Probability: A Judgment of Representativeness
This paper explores a heuristic — representativeness — according to which the subjective probability of an event, or a sample, is determined by the degree to which it: (i) is similar in essentialExpand
An Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models
Several studies this past decade have examined differences between holistic and decomposed approaches to determining weights in additive utility models. Some have argued that it matters little whichExpand
A clarification of “remarks on the analytic hierarchy process”
The replies by Saaty (1990) and by Harker and Vargas (1990) to my remarks on the Analytic Hierarchy Process (Dyer 1990) are useful in focusing attention on areas of disagreement regarding the AHP,Expand
Calibration of probabilities: the state of the art to 1980
From the subjectivist point of view (de Finetti, 1937/1964), a probability is a degree of belief in a proposition. It expresses a purely internal state; there is no “right,” “correct,” or “objective”Expand
On the psychology of prediction
In this paper, we explore the rules that determine intuitive predictions and judgments of confidence and contrast these rules to the normative principles of statistical prediction. Two classes ofExpand
Remarks on the analytic hierarchy process
The analytic hierarchy process AHP is flawed as a procedure for ranking alternatives in that the rankings produced by this procedure are arbitrary. This paper provides a brief review of several areasExpand
The Failure of Risk Management: Why It's Broken and How to Fix It
Preface. Acknowledgments. PART ONE AN INTRODUCTION TO THE CRISIS. CHAPTER 1 Healthy Skepticism for Risk Management. Common Mode Failure. What Counts as Risk Management. Anecdote: The Risk ofExpand
Determinants of linear judgment: a meta-analysis of lens model studies.
A detailed analysis of learning studies revealed that the most effective form of feedback was information about the task, and empirically under what conditions the application of bootstrapping--or replacing judges by their linear models--is advantageous. Expand
Interaction with Others Increases Decision Confidence but Not Decision Quality: Evidence against Information Collection Views of Interactive Decision Making
Abstract We present three studies of interactive decision making, where decision makers interact with others before making a final decision alone. Because the theories of lay observers and socialExpand