Integrating apache spark into PBS-Based HPC environments

  title={Integrating apache spark into PBS-Based HPC environments},
  author={Troy Baer and Paul Peltz and Junqi Yin and Edmon Begoli},
This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NICS) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC… CONTINUE READING
Recent Discussions
This paper has been referenced on Twitter 2 times over the past 90 days. VIEW TWEETS

From This Paper

Figures, tables, and topics from this paper.
7 Citations
5 References
Similar Papers


Publications citing this paper.


Publications referenced by this paper.
Showing 1-5 of 5 references

How YARN changed Hadoop job scheduling

  • Adam Diaz
  • Linux Journal,
  • 2014
Highly Influential
5 Excerpts

Similar Papers

Loading similar papers…