The (black) art of runtime evaluation: Are we comparing algorithms or implementations?

@article{Kriegel2016TheA,
  title={The (black) art of runtime evaluation: Are we comparing algorithms or implementations?},
  author={H. Kriegel and Erich Schubert and A. Zimek},
  journal={Knowledge and Information Systems},
  year={2016},
  volume={52},
  pages={341-378}
}
  • H. Kriegel, Erich Schubert, A. Zimek
  • Published 2016
  • Computer Science
  • Knowledge and Information Systems
  • Any paper proposing a new algorithm should come with an evaluation of efficiency and scalability (particularly when we are designing methods for “big data”). However, there are several (more or less serious) pitfalls in such evaluations. We would like to point the attention of the community to these pitfalls. We substantiate our points with extensive experiments, using clustering and outlier detection methods with and without index acceleration. We discuss what we can learn from evaluations… CONTINUE READING
    103 Citations
    DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN
    • 352
    • PDF
    Realization of Random Forest for Real-Time Evaluation through Tree Framing
    • 4
    • PDF
    Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework
    • 5
    • PDF
    Numerically stable parallel computation of (co-)variance
    • 9
    • PDF
    Statistically Rigorous Testing of Clustering Implementations
    • 1
    • PDF
    Similarity Search and Applications
    • Highly Influenced
    Anytime parallel density-based clustering
    • 7

    References

    SHOWING 1-10 OF 114 REFERENCES
    Frequent Subgraph Miners : Runtimes Don ’ t Say Everything
    • Georges-Köhler-Allee Albert-Ludwidgs-Universität, Gebäude
    • 2006
    • 15
    • PDF
    Making k-means Even Faster
    • 143
    • PDF
    Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection
    • 210
    An Experimental Analysis of Iterated Spatial Joins in Main Memory
    • 43
    • PDF
    A Quantitative Comparison of the Subgraph Miners MoFa, gSpan, FFSM, and Gaston
    • 146
    • PDF
    DBSCAN Revisited: Mis-Claim, Un-Fixability, and Approximation
    • 126
    • Highly Influential
    • PDF
    A fast APRIORI implementation
    • 271
    • PDF
    STR: a simple and efficient algorithm for R-tree packing
    • 461
    • PDF