Ratings and rankings: Voodoo or science?

  title={Ratings and rankings: Voodoo or science?},
  author={Paolo Paruolo and Andrea Saltelli and Michaela Saisana},
  journal={Quality Engineering},
Summary.  Composite indicators aggregate a set of variables by using weights which are understood to reflect the variables’ importance in the index. We propose to measure the importance of a given variable within existing composite indicators via Karl Pearson's ‘correlation ratio’; we call this measure the ‘main effect’. Because socio-economic variables are heteroscedastic and correlated, relative nominal weights are hardly ever found to match relative main effects; we propose to summarize… 

Figures and Tables from this paper

Weights and Importance in Composite Indicators: Mind the Gap
Multidimensional measures (often termed composite indicators) are popular tools in the public discourse for assessing the performance of countries on human development, perceived corruption,
The Importance of Variables in Composite Indices: A Contribution to the Methodology and Application to Development Indices
AbstractThe paper examines the issue of weights and importance in composite indices of development. Building a composite index involves several steps, one of them being the weighting of variables.
A critical comparative analysis of five world university rankings
  • H. Moed
  • Computer Science
  • 2016
A comparative analysis is conducted of five ranking systems: ARWU, Leiden, THE, QS and U-Multirank, arguing that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a dataset and tools to observe patterns in multi-faceted data.
A non-simplistic approach to composite indicators and rankings: an illustration by comparing the sustainability of the EU Countries
Composite indicators are very popular, despite being affected by several problems that often result in lack of robustness of the rankings involved. The aim of this paper is to show that composite
Measuring the importance of variables in composite indicators
Composite indicators are aggregations of measurable variables (indicators) that aim to quantify underlying concepts that are not directly observable, such as competitiveness, freedom of press or
Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change
Quantitative measures of research output, especially bibliometric measures, have not only been introduced within research funding systems in many countries, but they are also increasingly used in the
Methodological challenges in building composite indexes: Linking theory to practice
Composite indicators are emerging in several fields and disciplines as appealing method to synthesize a multitude of information, in a compact, single, and unique way. The process of aggregating
Quantitative Storytelling in the Making of a Composite Indicator
The reasons for and against composite indicators are briefly reviewed, as well as the available theories for their construction. After noting the strong normative dimension of these measures—which
A framework based on statistical analysis and stakeholders' preferences to inform weighting in composite indicators
It is demonstrated that the strength of the correlations between the indicators is directly linked with their capacity to transfer information to the CI, and analysts can choose a weighting scheme and possibly modify the index while achieving a more consistent index.
Assessing the gender gap in labour market index: volatility of results and reliability
Purpose - – The construction and use of composite indicators has recently increased considerably because such indicators permit the comparison and ranking of countries with respect to complex


Rickety numbers: Volatility of university rankings and policy implications
The most popular world university rankings are routinely taken at face value by media and social actors. While these rankings are politically influential, they are sensitive to both the conceptual
Noncompensatory/nonlinear composite indicators for ranking countries: a defensible setting
Composite indicators (or indexes) are very common in economic and business statistics for benchmarking the mutual and relative progress of countries in a variety of policy domains such as industrial
Differential Weighting: A Review of Methods and Empirical Studies1
When measures are to be combined to form a composite measure or to predict a criterion, the question of differential weighting of the component measures arises. Can differential weighting improve the
Composite Indicators between Analysis and Advocacy
We explore to what extent composite indicators, capable of aggregating multi-dimensional processes into simplified, stylised concepts, are up to the task of underpinning the development of data-based
Consequences of Data Error in Aggregate Indicators: Evidence from the Human Development Index
This paper examines the consequences of data error in data series used to construct aggregate indicators. Using the most popular indicator of country level economic development, the Human Development
Should you believe in the Shanghai ranking?
The view is that the Shanghai ranking, in spite of the media coverage it receives, does not qualify as a useful and pertinent tool to discuss the “quality” of academic institutions, let alone to guide the choice of students and family or to promote reforms of higher education systems.
Making best use of model evaluations to compute sensitivity indices
This paper deals with computations of sensitivity indices in sensitivity analysis. Given a mathematical or computational model y=f(x1,x2,…,xk), where the input factors xi's are uncorrelated with one
World University rankings
The use of national and international rankings of universities is increasing rapidly. Dill and Soo (2005), who take a generally positive view of university league tables, review the performance of
Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators
To what extent the use of uncertainty and sensitivity analysis may increase transparency or make policy inference more defensible by applying the methodology to a known composite indicator: the United Nations's technology achievement index is discussed.
Majority Judgment: Measuring, Ranking, and Electing
In Majority Judgment, Michel Balinski and Rida Laraki argue that the traditional theory of social choice offers no acceptable solution to the problems of how to elect, to judge, or to rank. They find