The (black) art of runtime evaluation: Are we comparing algorithms or implementations?
@article{Kriegel2016TheA, title={The (black) art of runtime evaluation: Are we comparing algorithms or implementations?}, author={H. Kriegel and Erich Schubert and A. Zimek}, journal={Knowledge and Information Systems}, year={2016}, volume={52}, pages={341-378} }
Any paper proposing a new algorithm should come with an evaluation of efficiency and scalability (particularly when we are designing methods for “big data”). However, there are several (more or less serious) pitfalls in such evaluations. We would like to point the attention of the community to these pitfalls. We substantiate our points with extensive experiments, using clustering and outlier detection methods with and without index acceleration. We discuss what we can learn from evaluations… CONTINUE READING
Figures, Tables, and Topics from this paper
103 Citations
DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN
- Computer Science
- TODS
- 2017
- 352
- PDF
Realization of Random Forest for Real-Time Evaluation through Tree Framing
- Computer Science
- 2018 IEEE International Conference on Data Mining (ICDM)
- 2018
- 4
- PDF
The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search
- Computer Science
- SISAP
- 2019
- 3
- PDF
Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework
- Computer Science, Mathematics
- ArXiv
- 2020
- 5
- PDF
Redundancies in Data and their Effect on the Evaluation of Recommendation Systems: A Case Study on the Amazon Reviews Datasets
- Computer Science
- SDM
- 2017
- 9
- PDF
Benchmarking Nearest Neighbor Search: Influence of Local Intrinsic Dimensionality and Result Diversity in Real-World Datasets
- Computer Science
- EDML@SDM
- 2019
- 4
- PDF
Statistically Rigorous Testing of Clustering Implementations
- Computer Science
- 2019 IEEE International Conference On Artificial Intelligence Testing (AITest)
- 2019
- 1
- PDF
Similarity Search and Applications
- Political Science, Computer Science
- Lecture Notes in Computer Science
- 2017
- Highly Influenced
References
SHOWING 1-10 OF 114 REFERENCES
Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection
- Computer Science
- Data Mining and Knowledge Discovery
- 2012
- 210
An Experimental Analysis of Iterated Spatial Joins in Main Memory
- Computer Science
- Proc. VLDB Endow.
- 2013
- 43
- PDF
A Quantitative Comparison of the Subgraph Miners MoFa, gSpan, FFSM, and Gaston
- Computer Science
- PKDD
- 2005
- 146
- PDF
DBSCAN Revisited: Mis-Claim, Un-Fixability, and Approximation
- Computer Science
- SIGMOD Conference
- 2015
- 126
- Highly Influential
- PDF
STR: a simple and efficient algorithm for R-tree packing
- Computer Science
- Proceedings 13th International Conference on Data Engineering
- 1997
- 461
- PDF