AIOPS 学习之路

异常检测

Anomaly detection techniques can also be classified into three modes, supervised, semi-supervised and unsupervised

unsupervised

VAE:
https://github.com/NetManAIOps/donut

Conditional VAE
https://github.com/NetManAIOps/Bagel

supervised

参考

https://github.com/tencent/metis
https://github.com/etsy/skyline
https://github.com/earthgecko/skyline
https://github.com/jx-zhong-for-academic-purpose/GCN-Anomaly-Detection

LSTM for time series anomaly detection

• 1. Train a LSTM prediction model
• 2. Compute the errors, and fit a multivariate Gaussion distribution
• 3. Detect the anomaly score based on the Mahalanobis distance

Variational AutoEncoder for time series anomaly detection

surpervised classification

• Ignore the outliers, and train an auto-encoder model only on normal data
• Feed the model with new data, if the reconstruction error is too high, then label them as the anomaly

unsurpervised clustering

Cluster-based Algorithm for Anomaly Detection in Time Series

density-based (density estimation) Unsupervised

Detects

Anomaly score

Reconstruction error

if you want to detect whether this example is an anomaly or not, predict it use a good trained model. If the model can’t predict it very well, then the problem is from this example instead of the model.

根因分析

• Granger Causality

• Transfer Entropy

Further reading

• https://github.com/h-cel/ClimateVegetationDynamics_GrangerCausality
• https://github.com/USC-Melady/Granger-causality
• https://github.com/fangsfdavid/causality
• https://github.com/DarkEyes/VLTimeSeriesCausality
• https://github.com/M-Nauta/TCDF
• https://github.com/akelleh/causality

Reference

Lstm-Variational-Auto-encoder
Extreme Rare Event Classification using AutoEncoders in Keras
RNN-Time-series-Anomaly-Detection
LSTM for Anomaly Detection in Time Series Data
cs.kuleuven.be
Time-Series Anomaly Detection Service at Microsoft

reference and further reading

https://github.com/donglee-afar/logdeep

https://github.com/jixinpu/aiopstools

https://github.com/chris-chris/mlops-example

https://github.com/chenryn/aiops-handbook

https://github.com/ChaojunWang1994/anomaly-detection-for-high-dimensional-sparse-data

https://github.com/Marcnuth/AnomalyDetection

https://github.com/visenger/awesome-mlops

https://github.com/yzhao062/pyod

https://github.com/johannfaouzi/pyts

https://github.com/NetManAIOps

https://tech.meituan.com/2020/10/15/mt-aiops-horae.html

Causality:
Hotspot: https://github.com/junkfei/Multidimensional_root_cause_analysis
Attributor,
IDice,
Squeeze,
Apriori
time adversial

Synthetic Control Method: https://github.com/jlhourENSAE/PenSynthPy/blob/master/functions/pensynth_functions.py

sequential bayesian inference
inverse problem

https://zhuanlan.zhihu.com/p/114918107

Anomaly:
Unsupervised first
Classification when cumulate enough data