CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System

  title={CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System},
  author={Huiling Qin and Xianyuan Zhan and Yu Zheng},
Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often leads to limited or no anomaly labels in the dataset. Second, anomalies in large systems usually occur in a collective manner due to the underlying dependency structure among devices or sensors. Lastly, real-time anomaly detection for high-dimensional data… 

Figures and Tables from this paper

Network-Wide Traffic States Imputation Using Self-interested Coalitional Learning

This study focuses on developing a robust and interpretable network-wide traffic state imputation framework using partially observed traffic information, and introduces a new learning strategy, called SCL, which forges cooperation between a main self-interested semi-supervised learning task and a discriminator as a critic to facilitate main task training while providing interpretability on the results.



Deep Anomaly Detection with Deviation Networks

A novel anomaly detection framework and its instantiation that can be trained substantially more data-efficiently and achieves significantly better anomaly scoring than state-of-the-art competing methods is introduced.

Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network

The core idea is to capture the normal patterns of multivariate time series by learning their robust representations with key techniques such as stochastic variable connection and planar normalizing flow, reconstruct input data by the representations, and use the reconstruction probabilities to determine anomalies.

Group Anomaly Detection using Deep Generative Models

This paper proposes deep generative models: Adversarial autoencoder (AAE) and variational autoen coder (VAE) for group anomaly detection and shows that both AAE and VAE detect group anomalies using point-wise input data where group memberships are known a priori.

Collective Anomaly Detection Based on Long Short-Term Memory Recurrent Neural Networks

This paper proposes a real time collective anomaly detection model based on neural network learning that is built on a time series version of the KDD 1999 dataset and demonstrates that it is possible to offer reliable and efficient collective anomalies detection.

Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications

Donut is proposed, an unsupervised anomaly detection algorithm based on VAE that greatly outperforms a state-of-arts supervised ensemble approach and a baseline VAE approach, and its best F-scores range from 0.75 to 0.9 for the studied KPIs from a top global Internet company.

Deep Learning for Anomaly Detection: A Survey

This survey presents a structured and comprehensive overview of research methods in deep learning-based anomaly detection, grouping state-of-the-art deep anomaly detection research techniques into different categories based on the underlying assumptions and approach adopted.

Deep Structured Energy Based Models for Anomaly Detection

This paper proposes deep structured energy based models (DSEBMs), where the energy function is the output of a deterministic deep neural network with structure, and develops novel model architectures to integrate EBMs with different types of data such as static data, sequential data, and spatial data.

Learning Competitive and Discriminative Reconstructions for Anomaly Detection

This paper proposes a new method for anomaly detection that can learn the labels of unlabelled data directly and works in a discriminative manner without suffering from overfitting, thus it is efficient and scalable even for large-scale datasets.

Detecting collective anomalies from multiple spatio-temporal datasets across different domains

This work evaluates the method using five datasets related to New York City: 311 complaints, taxicab data, bike rental data, points of interest, and road network data, finding the anomalies that cannot be identified by a single dataset.

Time-Series Anomaly Detection Service at Microsoft

This work is the first attempt to borrow the SR model from visual saliency detection domain to time-series anomaly detection, and innovatively combine SR and CNN together to improve the performance of SR model.