Storm@twitter

Abstract

This paper describes the use of Storm at Twitter. Storm is a real-time fault-tolerant and distributed stream data processing system. Storm is currently being used to run various critical computations in Twitter at scale, and in real-time. This paper describes the architecture of Storm and its methods for distributed scale-out and fault-tolerance. This paper also describes how queries (aka. topologies) are executed in Storm, and presents some operational stories based on running Storm at Twitter. We also present results from an empirical evaluation demonstrating the resilience of Storm in dealing with machine failures. Storm is under active development at Twitter and we also present some potential directions for future work.

DOI: 10.1145/2588555.2595641

10 Figures and Tables

0204060802014201520162017
Citations per Year

169 Citations

Semantic Scholar estimates that this publication has 169 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Toshniwal2014Stormtwitter, title={Storm@twitter}, author={Ankit Toshniwal and Siddarth Taneja and Amit Shukla and Karthikeyan Ramasamy and Jignesh M. Patel and Sanjeev Kulkarni and Jason Jackson and Krishna Gade and Maosong Fu and Jake Donham and Nikunj Bhagat and Sailesh Mittal and Dmitriy V. Ryaboy}, booktitle={SIGMOD Conference}, year={2014} }