• Corpus ID: 7670402

Problems with the Calculation of Novelty Metrics

@inproceedings{Brown2014ProblemsWT,
  title={Problems with the Calculation of Novelty Metrics},
  author={David C. Brown},
  year={2014}
}
Shah, Vargas-Hernandez & Smith [2003] (SVS) have proposed a widely adopted and very influential set of metrics, including one for novelty. They attempted to make them specifically appropriate for Engineering Design, while keeping principles from Cognitive Psychology in mind. In this position paper we view the SVS proposal for possible use in Computational Design Creativity systems. Unfortunately we conclude that, while it appears to be useful for human assessment of design novelty it poses… 
web.cs.wpi.edu
20 Citations
Highly Influential Citations
4
Background Citations
13
Methods Citations
5
Results Citations
1

20 Citations

ISSUES RELATED TO MISSING ATTRIBUTES IN A-POSTERIORI NOVELTY ASSESSMENTS
TLDR
This paper argues about the recalled issue, providing detailed suggestions for a correct “a-posteriori” novelty assessment of ideas generated from tasks with multiple required functions.
Refined metric for a-posteriori novelty assessments
TLDR
This paper aims at identifying the roots of these problems and at proposing a refined assessment approach, capable to extend the applicability of SNM to any set of ideas.
A-POSTERIORI NOVELTY ASSESSMENTS FOR SEQUENTIAL DESIGN SESSIONS
TLDR
This paper argues about the problem of when in presence of sequential idea generation sessions of the same task, the application of the recalled metric could lead to misleading results, and provides useful suggestions for a correct “a-posteriori” assessment for sequential ideageneration sessions.
A Brief Review of Approaches to Design Novelty Assessment
This paper questions the assumption in design creativity research that we know how to calculate a meaningful score for the novelty of a design. This paper looks at the basics of novelty calculation,…
Impact of missing attributes on the novelty metric of Shah et al.
TLDR
The investigation has been performed by relying on a high number of randomly generated virtual sets of ideas and revealed that the problem of “missing attributes”, if not opportunely faced, leads the considered metric to misleading novelty scores.
Three-Dimensional Approach for Assessing Uncommonness of Ideas
TLDR
This paper proposes an integrated procedure where the advantages offered by the three families of metrics can be applied concurrently, and results show that a more comprehensive set of information about a-posteriori novelty can be extracted.
A-Posteriori Novelty Metrics Based on Idea Decomposition
TLDR
The problems affecting the original SNM version have not been comprehensively resolved, and the different variants proposed in literature can even present some additional flaws, thus providing fundamental information to support accurate metric selection.
Novelty metrics in engineering design
TLDR
A map of novelty metrics has been obtained, which shows that most of them belong to the uncommonness type, while others need further investigations, which is intended to support scholars in browsing the different metrics, when selecting the most suited for their specific research objectives.
Orienting through the Variants of the Shah’s A-Posteriori Novelty Metric
TLDR
This paper argues about the variants proposed for the a-posteriori metric of Shah et al., and proposes a selection framework to support researchers in selecting the most suited for their experimental needs, which highlights the presence of commonness of key attributes.
Unpacking Subjective Creativity Ratings: Using Embeddings to Explain and Measure Idea Novelty
TLDR
A way to uncover human judgment of design idea similarity using two dimensional idea maps is demonstrated, which provide a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational.
...
...

References

SHOWING 1-10 OF 19 REFERENCES
USING AI TO EVALUATE CREATIVE DESIGNS
Many have offered criteria for judging a design as creative. Among these criteria have been novelty, value, and surprise. We offer a unique perspective and synthesis of these three criteria with the…
Creativity, Surprise & Design: An Introduction and Investigation
This paper weaves together some of the basic ideas about surprise, and creativity, in the context of design. Situations in which design surprise might occur are identified, and a few are discussed in…
Measuring Functional Creativity: Non‐Expert Raters and the Creative Solution Diagnosis Scale
The Creative Solution Diagnosis Scale (CSDS) is a 30-item scale based on a core of four criteria: Relevance & Effectiveness, Novelty, Elegance, and Genesis. The CSDS offers potential for the…
Developing computational design creativity systems
The underlying hypothesis of this paper is that the rigor needed to build computational design creativity systems will produce insights that will increase our understanding of creativity. The paper…
Identifying Quality, Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation
TLDR
A method for evaluating ideas with regard to four dimensions—novelty, workability, relevance, and specificity—and has identified two measurable sub-dimensions for each of the four main dimensions is described.
Metrics for measuring ideation effectiveness
Assessing design creativity
Originality and Novelty: A Different Universe
In order to validate and steer the development of ideation methods, in recent years, a number of quantitative metrics were developed aimed at measuring the effectiveness thereof, e.g. novelty and…
Let’s not get too creative!
This short position paper urges researchers to assume that existing research on creativity evaluation should form the basis for establishing computable metrics: what already exists, even if it is not…
Refined metrics for measuring ideation effectiveness
...
...