iSENSE: Completion-Aware Crowdtesting Management

  title={iSENSE: Completion-Aware Crowdtesting Management},
  author={Junjie Wang and Ye Yang and Rahul Krishna and Tim Menzies and Qing Wang},
  journal={2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)},
  • Junjie Wang, Ye Yang, Qing Wang
  • Published 25 May 2019
  • Computer Science
  • 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)
Crowdtesting has become an effective alternative to traditional testing, especially for mobile applications. However, crowdtesting is hard to manage in nature. Given the complexity of mobile applications and unpredictability of distributed crowdtesting processes, it is difficult to estimate (a) remaining number of bugs yet to be detected or (b) required cost to find those bugs. Experience-based decisions may result in ineffective crowdtesting processes, e.g., there is an average of 32% wasteful… 

Figures and Tables from this paper

Quest for the Golden Approach: An Experimental Evaluation of Duplicate Crowdtesting Reports Detection
This paper conducts the first experimental evaluation of the commonly-used and state-of-the-art approaches for duplicate detection in crowdtesting reports, and explores which is the golden approach.
Context-aware In-process Crowdworker Recommendation
A context-aware in-process crowdworker recommendation approach, iRec, to detect more bugs earlier and potentially shorten the non-yielding windows and results show the potential of iRec in improving the cost-effectiveness of crowdtesting by saving the cost and shortening the testing process.
Artificial Intelligence-Powered Worker Engagement in Software Crowdsourcing
Crowdsourced Software Engineering (CSE) evolved from outsourcing and open source development. It has created a fundamental shift; there are now many open-call software minitasks that are advertised
Guided Bug Crush: Assist Manual GUI Testing of Android Apps via Hint Moves
This work proposes an approach named NaviDroid for navigating testers via highlighted next operations for more effective and efficient testing, and constructs an enriched state transition graph with the triggering actions as the edges for two involved states.
Owl Eyes: Spotting UI Display Issues via Visual Understanding
This work proposes a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot, which can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug.
OwlEyes-online: a fully automated platform for detecting and localizing UI display issues
An online UI display issue detection tool OwlEyes-Online is implemented, which provides a simple and easy-to-use platform for users to realize the automatic detection and localization of UI display issues.
This article investigates the necessity and feasibility of close prediction of crowdtesting tasks based on an industrial dataset, and proposes a close prediction approach named iSENSE2.0, which applies incremental sampling technique to process crowdtesting reports arriving in chronological order and organizes them into fixed-sized groups as dynamic inputs.


Multi-Objective Crowd Worker Selection in Crowdsourced Testing
A Multi-Objective crowd wOrker SElection approach (MOOSE), which includes three objectives: maximizing the coverage of test requirement, minimizing the cost, and maximizing bug-detection experience of the selected crowd workers.
Who Should Be Selected to Perform a Task in Crowdsourced Testing?
ExReDiv is introduced, a novel hybrid approach to select a set of workers for a test task that consists of three key strategies: the experience strategy selects experienced workers, the relevance strategy selects workers with expertise relevant to the given test task, and the diversity strategy selects diverse workers to avoid detecting duplicated bugs.
Test report prioritization to assist crowdsourced testing
The first technique of its kind, to the best of the knowledge, to prioritize test reports for manual inspection is created, which shows that DivRisk can significantly outperform random prioritization and can approximate the best theoretical result for a real-world industrial mobile application.
COCOON: Crowdsourced Testing Quality Maximization Under Context Coverage Constraint
It is proved that the Cocoon problem is NP-Complete and then two greedy approaches are introduced, which can be potentially used as online services in practice.
Successes, challenges, and rethinking – an industrial investigation on crowdsourced mobile application testing
This paper will present a large-scale and real-life industrial study based on a CMAT intermediary on testing five real- life Android applications using their CMAT platform and give an insightful analysis to investigate the successes and challenges of applying CMAT.
Fuzzy Clustering of Crowdsourced Test Reports for Apps
This study proposes a new framework named Test Report Fuzzy Clustering Framework (TERFUR) by aggregating redundant and multi-bug test reports into clusters to reduce the number of inspected test reports and proves the effectiveness of TERFUR in prioritizing test reports for manual inspection.
Local-based active classification of test report to assist crowdsourced testing
This work proposes LOcal-based Active ClassiFication (LOAF) to classify true fault from crowdsourced test reports, and proposes a small portion of instances which are most informative within local neighborhood, and asks user their labels, then learns classifiers based on local neighborhood.
Towards Effectively Test Report Classification to Assist Crowdsourced Testing
The first work to address the test report classification problem in real industrial crowdsourced testing practice by proposing a cluster-based classification approach, which first clusters similar reports together and then builds classifiers based on most similar clusters with ensemble method.
Multi-objective test report prioritization using image understanding
By taking the similarity of screenshots into consideration, this paper presents a multi-objective optimization-based prioritization technique to assist inspections of crowdsourced test reports and shows that image-understanding techniques can provide benefit to test-report prioritization for most applications.
Reproducing Context-Sensitive Crashes of Mobile Apps Using Crowdsourced Monitoring
MoTiF, a crowdsourced approach to support app developers in automatically reproducing context-sensitive crashes faced by end-users in the wild, is introduced by analyzing recurrent patterns in crash data, and the shortest sequence of events reproducing a crash is derived, and turned into a test suite.