Scheduling Storms and Streams in the Cloud

@article{Ghaderi2016SchedulingSA,
  title={Scheduling Storms and Streams in the Cloud},
  author={Javad Ghaderi and Sanjay Shakkottai and Rayadurgam Srikant},
  journal={ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)},
  year={2016},
  volume={1},
  pages={1 - 28}
}
  • J. Ghaderi, S. Shakkottai, R. Srikant
  • Published 2016
  • Computer Science, Mathematics
  • ACM Transactions on Modeling and Performance Evaluation of Computing Systems (TOMPECS)
Motivated by emerging big streaming data processing paradigms (e.g., Twitter Storm, Streaming MapReduce), we investigate the problem of scheduling graphs over a large cluster of servers. Each graph is a job, where nodes represent compute tasks and edges indicate data flows between these compute tasks. Jobs (graphs) arrive randomly over time and, upon completion, leave the system. When a job arrives, the scheduler needs to partition the graph and distribute it over the servers to satisfy load… Expand
A Traffic and Resource Aware Online Storm Scheduler
TLDR
The proposed approach uses a graph partitioning algorithm and real time traffic data monitored from a deployed application to derive a near-optimal operator placement plan that minimizes inter-node communication and balances the overall communication load distribution. Expand
Simple high-performance algorithms for scheduling jobs in the cloud
  • J. Ghaderi
  • Computer Science
  • 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)
  • 2015
TLDR
A class of randomized algorithms for placing VMs in the servers that can achieve maximum throughput without preemptions are presented, which are naturally distributed, have low complexity, and each queue needs to perform limited operations. Expand
Randomized algorithms for scheduling VMs in the cloud
  • J. Ghaderi
  • Computer Science
  • IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications
  • 2016
TLDR
A class of randomized algorithms for placing VMs in the servers that can achieve maximum throughput without preemptions are presented, which are naturally distributed, have low complexity, and each queue needs to perform limited operations. Expand
On Non-Preemptive VM Scheduling in the Cloud
TLDR
A non-preemptive scheduling algorithm is proposed that can provide rβ fraction of the throughput region for β < r and can be tuned to provide a trade-off between achievable throughput, delay, and computational complexity of the scheduling algorithm. Expand
On Non-Preemptive VM Scheduling in the Cloud
TLDR
A non-preemptive scheduling algorithm is proposed that can provide rβ fraction of the throughput region for β < r and can be tuned to provide a tradeoff between achievable throughput, delay, and computational complexity of the scheduling algorithm. Expand
Task Scheduling in Data Stream Processing Systems
TLDR
Three proposed schedulers are presented, each having a different heuristic partitioning approach to minimise inter-node communication for either homogeneous or heterogeneous clusters, and it is demonstrated how each scheduler can efficiently assign groups of highly communicating tasks to compute nodes. Expand
I-Scheduler: Iterative scheduling for distributed stream processing systems
TLDR
This paper proposes a heuristic scheduling algorithm which reliably and efficiently finds highly communicating tasks by exploiting graph partitioning algorithms and a mathematical optimisation software package and evaluates the communication cost of this method using three micro-benchmarks. Expand
The Real-Time Scheduling Strategy Based on Traffic and Load Balancing in Storm
  • J. Zhang, Chunlin Li, Liye Zhu, Yanpei Liu
  • Computer Science
  • 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS)
  • 2016
TLDR
Experiments demonstrate that this real-time scheduling algorithm compared to the default scheduling algorithm, performance of average latency and average inter-node traffic in the system improved above 50%, and compared the traffic-based scheduling algorithms, improved about 10%. Expand
Power-Aware Allocation of Graph Jobs in Geo-Distributed Cloud Networks
TLDR
A framework for efficient allocation of graph jobs in geo-distributed cloud networks (GDCNs), explicitly considering the power consumption of the datacenters (DCs) is developed, and a novel low-complexity (decentralized) sub-graph extraction method is provided. Expand
Allocation of Computation-Intensive Graph Jobs Over Vehicular Clouds in IoV
TLDR
This article presents a novel framework for VCs that maps components of graph jobs to service providers via opportunistic vehicle-to-vehicle communication and proposes a novel low complexity randomized graph job allocation mechanism by considering hierarchical tree-based subgraph isomorphism extraction. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 48 REFERENCES
Adaptive online scheduling in storm
TLDR
Two advanced generic schedulers for Storm are proposed that provide improved performance for a wide range of application topologies and can produce schedules that achieve significantly better performances compared to those produced by Storm's default scheduler. Expand
Scheduling Decisions in Stream Processing on Heterogeneous Clusters
  • M. Rychlý, P. Škoda, P. Smrz
  • Computer Science
  • 2014 Eighth International Conference on Complex, Intelligent and Software Intensive Systems
  • 2014
TLDR
A proposal of a novel scheduler for stream processing frameworks on heterogeneous clusters is presented, which employs design-time knowledge as well as benchmarking techniques to achieve optimal resource-aware deployment of applications over the clusters and eventually better overall utilization of the cluster. Expand
Randomized Scheduling Algorithm for Queueing Networks
TLDR
A stable scheduling algorithm is presented that is myopic, distributed and performs few logical operations at each node per unit time for the packet-level wireless network model and a buffered circuit switched network model for an optical core of future Internet. Expand
TimeStream: reliable stream computation in the cloud
TLDR
This work advocates a powerful new abstraction called resilient substitution that caters to the specific needs in this new computation model to handle failure recovery and dynamic reconfiguration in response to load changes. Expand
Joint VM placement and routing for data center traffic engineering
TLDR
This work proposes an efficient online algorithm that employs the real data center traffic traces under a spectrum of elephant and mice flows and demonstrates a consistent and significant improvement over the benchmark achieved by common heuristics. Expand
Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters
TLDR
D-Streams support a new recovery mechanism that improves efficiency over the traditional replication and upstream backup solutions in streaming databases: parallel recovery of lost state across the cluster. Expand
MapReduce Online
TLDR
A modified version of the Hadoop MapReduce framework that supports online aggregation, which allows users to see "early returns" from a job as it is being computed, and can reduce completion times and improve system utilization for batch jobs as well. Expand
Asymptotic optimality of a greedy randomized algorithm in a large-scale service system with general packing constraints
TLDR
This work proposes an extremely simple and easily implementable customer placement algorithm, called Greedy-Random (GRAND), and proves that GRAND with 0 is asymptotically optimal, as the customer arrival rates grow to infinity and aarrow 0→0, in the sense of minimizing the total number of occupied servers in steady state. Expand
An Infinite Server System with General Packing Constraints
  • A. Stolyar
  • Mathematics, Computer Science
  • Oper. Res.
  • 2013
We consider a service system model primarily motivated by the problem of efficient assignment of virtual machines to physical host machines in a network cloud, so that the number of occupied hosts isExpand
S4: Distributed Stream Computing Platform
TLDR
The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. Expand
...
1
2
3
4
5
...