The (black) art of runtime evaluation: Are we comparing algorithms or implementations?

@article{Kriegel2016TheA,
  title={The (black) art of runtime evaluation: Are we comparing algorithms or implementations?},
  author={Hans-Peter Kriegel and Erich Schubert and Arthur Zimek},
  journal={Knowledge and Information Systems},
  year={2016},
  volume={52},
  pages={341-378}
}
Any paper proposing a new algorithm should come with an evaluation of efficiency and scalability (particularly when we are designing methods for “big data”). However, there are several (more or less serious) pitfalls in such evaluations. We would like to point the attention of the community to these pitfalls. We substantiate our points with extensive experiments, using clustering and outlier detection methods with and without index acceleration. We discuss what we can learn from evaluations… Expand
View on Springer
www-ai.cs.tu-dortmund.de
128 Citations
Highly Influential Citations
7
Background Citations
39
Methods Citations
29
Results Citations
4

128 Citations

Citation Type

Citation Type

DBSCAN Revisited, Revisited: Why and How You Should (Still) Use DBSCAN
TLDR
In new experiments, it is shown that the new SIGMOD 2015 methods do not appear to offer practical benefits if the DBSCAN parameters are well chosen and thus they are primarily of theoretical interest. Expand
Realization of Random Forest for Real-Time Evaluation through Tree Framing
TLDR
This paper introduces a method that optimizes the execution of Decision Trees (DT), a probabilistic view of decision tree execution, and presents a theoretically well-founded memory layout which maximizes locality during execution in both cases. Expand
The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search
TLDR
Different visualization concepts are introduced that allow to get a more fine-grained overview of the inner workings of nearest neighbor search principles: results on a single dataset predict results on all other datasets well. Expand
Numerically stable parallel computation of (co-)variance
TLDR
This paper studies a popular incremental technique originally proposed by Welford, which is extended to weighted covariance and correlation and showcases applications from the classic computation of variance as well as advanced applications such as stock market analysis with exponentially weighted moving models and Gaussian mixture modeling for cluster analysis that all benefit from this approach. Expand
Redundancies in Data and their Effect on the Evaluation of Recommendation Systems: A Case Study on the Amazon Reviews Datasets
TLDR
A collection of datasets crawled from Amazon, “Amazon reviews”, is popular in the evaluation of recommendation systems, however, it is observed that these datasets contain redundancies and their impact depends on the complexity of the methods. Expand
Benchmarking Nearest Neighbor Search: Influence of Local Intrinsic Dimensionality and Result Diversity in Real-World Datasets
TLDR
Different visualization concepts are introduced that allow to get a more fine-grained overview of the inner workings of nearest neighbor search principles: results on a single dataset predict results on all other datasets well. Expand
Statistically Rigorous Testing of Clustering Implementations
TLDR
This work conducts statistical hypothesis testing on the outcome of differential clustering to reveal problematic outcomes and indicates that there are statistically significant differences in clustering outcomes in a variety of scenarios where users might not expect clustering outcome variation. Expand
Similarity Search and Applications
TLDR
This paper analyzes the problem of understanding how the strategy for searching through an index tree, also called scheduling policy, can influence costs, and characterize the policies’ behavior through an analytical cost model, in which a major role is played by parameterized local distance distributions. Expand
Anytime parallel density-based clustering
TLDR
This paper proposes a novel anytime approach, called AnyDBC, that compresses the data into smaller density-connected subsets called primitive clusters and labels objects based on connected components of these primitive clusters to reduce the label propagation time of DBSCAN. Expand
Memory-Efficient RkNN Retrieval by Nonlinear k-Distance Approximation
TLDR
This work investigates that the assumption that the k-distances in real-world datasets often follow the power-law distribution is violated and uncover that it is violated in regions of changing density, which are typical for real-life datasets. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 114 REFERENCES
The (black) art of runtime evaluation
Any paper proposing a new algorithm should come with an evaluation of efficiency and scalability (particularly when we are designing methods for "big data"). However, there are several (more or les...
Frequent Subgraph Miners : Runtimes Don ’ t Say Everything
In recent years several frequent subgraph miners were proposed. The authors of these new algorithms typically compared the runtimes of their implementations with those of previous implementations to… Expand
Frequent subgraph miners: runtimes don't say everything
TLDR
This paper presents results of an additional experimental comparison of several graph miners, which differs in the following aspects from this previous study: (1) they compare original implementations; (2) these implementations are compared on a larger set of measures than runtimes, thus providing further insight in the benefits of the algorithms. Expand
Making k-means Even Faster
TLDR
This paper proposes a new acceleration for exact k-means that gives the same answer, but is much faster in practice, and uses one novel lower bound for point-center distances. Expand
Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection
TLDR
A formalized method of analysis is provided to allow for a theoretical comparison and generalization of many existing methods and improves understanding of the shared properties and of the differences of outlier detection models. Expand
An Experimental Analysis of Iterated Spatial Joins in Main Memory
TLDR
Surprisingly, it is found that when queries and updates can be batched, repeatedly re-computing the join result from scratch outperforms using a moving object index in all but the most extreme cases. Expand
A Quantitative Comparison of the Subgraph Miners MoFa, gSpan, FFSM, and Gaston
TLDR
This paper has re-implemented the subgraph miners MoFa, gSpan, FFSM, and Gaston within a common code base and with the same level of programming expertise and optimization effort. Expand
DBSCAN Revisited: Mis-Claim, Un-Fixability, and Approximation
TLDR
It is proved that for d ≥ 3, the DBSCAN problem requires Ω(n4/3) time to solve, unless very significant breakthroughs---ones widely believed to be impossible---could be made in theoretical computer science, and the running time can be dramatically brought down to O(n) in expectation regardless of the dimensionality d. Expand
A fast APRIORI implementation
TLDR
It is shown that the effect of implementation can be more important than the selection of the algorithm, and an implementation of APRIORI is described that outperforms all implementations known to us. Expand
STR: a simple and efficient algorithm for R-tree packing
Presents the results from an extensive comparison study of three R-tree packing algorithms: the Hilbert and nearest-X packing algorithms, and an algorithm which is very simple to implement, called… Expand
...
1
2
3
4
5
...