Towards Machine Learning-Based Auto-tuning of MapReduce

@article{Yigitbasi2013TowardsML,
  title={Towards Machine Learning-Based Auto-tuning of MapReduce},
  author={Nezih Yigitbasi and Theodore L. Willke and Guangdeng Liao and Dick H. J. Epema},
  journal={2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems},
  year={2013},
  pages={11-20}
}
MapReduce, which is the de facto programming model for large-scale distributed data processing, and its most popular implementation Hadoop have enjoyed widespread adoption in industry during the past few years. Unfortunately, from a performance point of view getting the most out of Hadoop is still a big challenge due to the large number of configuration parameters. Currently these parameters are tuned manually by trial and error, which is ineffective due to the large parameter space and the… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 46 CITATIONS

Evaluation of Novel Approaches to Software Engineering

  • Communications in Computer and Information Science
  • 2018
VIEW 5 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

PTC: Performance Tuning Component for Auto-Tuning of MapReduce's Configuration

VIEW 5 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Co-locating and concurrent fine-tuning MapReduce applications on microservers for energy efficiency

  • 2017 IEEE International Symposium on Workload Characterization (IISWC)
  • 2017
VIEW 5 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Intersection of HPC and Machine Learning

Kadupitiya Kadupitige
  • 2017
VIEW 4 EXCERPTS
CITES BACKGROUND, METHODS & RESULTS
HIGHLY INFLUENCED

An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems

  • 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS)
  • 2016
VIEW 4 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

Real Time Autotuning for MapReduce on Hadoop/YARN

VIEW 4 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

References

Publications referenced by this paper.
SHOWING 1-10 OF 16 REFERENCES

Application performance modeling in a virtualized environment

  • HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture
  • 2010
VIEW 1 EXCERPT

The HiBench benchmark suite: Characterization of the MapReduce-based data analysis

  • 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010)
  • 2010
VIEW 1 EXCERPT

Machine learning-based prefetch optimization for data center applications

  • Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
  • 2009
VIEW 2 EXCERPTS

CARVE: A Cognitive Agent for Resource Value Estimation

  • 2008 International Conference on Autonomic Computing
  • 2008
VIEW 1 EXCERPT