Report on the Cloud-Based Evaluation Approaches Workshop 2015

  title={Report on the Cloud-Based Evaluation Approaches Workshop 2015},
  author={Henning M{\"u}ller and Jayashree Kalpathy-Cramer and Allan Hanbury and Keyvan Farahani and Rinat A. Sergeev and Jin H. Paik and Arno Klein and Antonio Criminisi and Andrew D. Trister and Thea C. Norman and David N. Kennedy and Ganapati Srinivasa and Artem Mamonov and Nina Preuss},
  journal={SIGIR Forum},
Data analysis requires new approaches in many domains for evaluating tools and techniques, particularly when the data sets grow large and more complex. Evaluation-as-service (EaaS) was coined as a term to represent evaluation approaches based on APIs, virtual machines or source code submission, different from the common paradigm of evaluating techniques on a distributed test collection, tasks and submitted results files. Such new approaches become necessary when data sets become extremely large… 

Figures from this paper

EaaS: Evaluation-as-a-Service and Experiences from the VISCERAL Project

The next steps for research infrastructures are imagined and the impact that EaaS can have in this context to make research in data science more efficient and effective is summarized.

Evaluation-as-a-Service for the Computational Sciences

The article summarizes several existing approaches to EaaS and analyzes their usage scenarios and also the advantages and disadvantages, and compares and compares the current approaches and consolidate the experiences to outline the next steps of EAAS, particularly toward sustainable research infrastructures.

VISCERAL: Evaluation-as-a-Service for Medical Imaging

The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of the problems of static data collections as the data change over time and data preparation often takes much time, and has been applied in the VISCERAL project.

A Survey on Online Judge Systems and Their Applications

The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.

An Assessment of Imaging Informatics for Precision Medicine in Cancer.

A survey of the role and priorities for imaging informatics to help advance quantitative imaging in the era of precision medicine is provided.

1 A Survey on Online Judge Systems and Their Applications

This research presents a meta-modelling framework for estimating the energy consumption and energy efficiency of energy-efficient lightbulbs through a variety of approaches, including “solution-based” and “smart” approaches.

Evaluation as a Service architecture and crowdsourced problems solving implemented in platform

TLDR, a cloud computing architecture that tries to make assessment process more reliable by providing online tools and test instances dedicated to the evaluation of algorithms, is presented together with four challenges that were organized with its support.



Bringing the Algorithms to the Data: Cloud-Based Benchmarking for Medical Image Analysis

This text presents reflections and ideas of a concrete project on using cloud---based benchmarking paradigms for medical image analysis and retrieval using cloud computing technology to run two evaluation campaigns in 2013 and 2014 using the proposed technology.

Evaluation-as-a-Service: Overview and Outlook

The objective of this white paper are to summarize and compare the current approaches and consolidate the experiences of these approaches to outline the next steps of EaaS, particularly towards sustainable research infrastructures.

Report on the Evaluation-as-a-Service (EaaS) Expert Workshop

The objective of the meeting was to bring together initiatives that use cloud infrastructures, virtual machines, APIs (Application Programming Interface) and related projects that provide evaluation of information retrieval or machine learning tools as a service.

VISCERAL: Towards Large Data in Medical Imaging - Challenges and Directions

The project Visceral will provide the means to jump---start this process by providing access to unprecedented amounts of real world imaging data annotated through experts and by using a community effort to generate a large corpus of automatically generated standard annotations.

Marginality and Problem-Solving Effectiveness in Broadcast Search

Female solvers---known to be in the “outer circle” of the scientific establishment---performed significantly better than men in developing successful solutions, and the value of openness is demonstrated in removing barriers to entry to nonobvious individuals.

Perspective: Sustaining the big-data ecosystem

Organizing and accessing biomedical big data will require quite different business models, say Philip E. Bourne, Jon R. Lorsch and Eric D. Green.

Incentives and Problem Uncertainty in Innovation Contests: An Empirical Analysis

It is shown that the effort-reducing effect of greater rivalry dominates for less uncertain problems, whereas the effect on the extreme value prevails for more uncertain problems and higher uncertainty reduces the negative effect of added competitors on incentives.


  • ACM SIGIR Forum
  • 2016

Sustaining the big–data ecosystem

  • Nature
  • 2015

Green . Sustaining the big – data ecosystem

  • Nature
  • 2015