seldonio/alibi-detectAlgorithms for outlier, adversarial and drift detection
Alibi Detect is a source-available Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline detectors for tabular data, text, images and time series. Both TensorFlow and PyTorch backends are supported for drift detection.
For more background on the importance of monitoring outliers and distributions in a production setting, check out this talk from the Challenges in Deploying and Monitoring Machine Learning Systems ICML 2020 workshop, based on the paper Monitoring and explainability of models in production and referencing Alibi Detect.
For a thorough introduction to drift detection, check out Protecting Your Machine Learning Against Drift: An Introduction. The talk covers what drift is and why it pays to detect it, the different types of drift, how it can be detected in a principled manner and also describes the anatomy of a drift detector.
The package, alibi-detect can be installed from:
pip)conda/mamba)alibi-detect can be installed from PyPI:
pip install alibi-detect
Alternatively, the development version can be installed:
pip install git+https://github.com/SeldonIO/alibi-detect.git
To install with the TensorFlow backend:
pip install alibi-detect[tensorflow]
To install with the PyTorch backend:
pip install alibi-detect[torch]
To install with the KeOps backend:
pip install alibi-detect[keops]
To use the Prophet time series outlier detector:
pip install alibi-detect[prophet]
To install from conda-forge it is recommended to use mamba, which can be installed to the base conda enviroment with:
conda install mamba -n base -c conda-forge
To install alibi-detect:
mamba install -c conda-forge alibi-detect
We will use the VAE outlier detector to illustrate the API.
from alibi_detect.od import OutlierVAE
from alibi_detect.saving import save_detector, load_detector
# initialize and fit detector
od = OutlierVAE(threshold=0.1, encoder_net=encoder_net, decoder_net=decoder_net, latent_dim=1024)
od.fit(x_train)
# make predictions
preds = od.predict(x_test)
# save and load detectors
filepath = './my_detector/'
save_detector(od, filepath)
od = load_detector(filepath)
The predictions are returned in a dictionary with as keys meta and data. meta contains the detector's metadata while data is in itself a dictionary with the actual predictions. It contains the outlier, adversarial or drift scores and thresholds as well as the predictions whether instances are e.g. outliers or not. The exact details can vary slightly from method to method, so we encourage the reader to become familiar with the types of algorithms supported.
The following tables show the advised use cases for each algorithm. The column Feature Level indicates whether the detection can be done at the feature level, e.g. per pixel for an image. Check the algorithm reference list for more information with links to the documentation and original papers as well as examples for each of the detectors.