Model drift represents the concept of a model degrading in performance over time due to changes in the underlying dataset or concepts. Monitoring model drift involves the ability to surface this performance information to the end-user and configure alerts so that they can take actions (pre-emptive preferably) to reduce the business impact of this drift. For more details, see Monitoring Alerts.
You can monitor three types of drift using ModelOps: feature, prediction, and performance.
Feature Drift
Also referred to as data drift, feature drift is based on understanding and monitoring changes in the dataset statistics the model was trained on versus the dataset statistics the model is currently predicting. You can monitor this data in an offline process or in real time, as data is fed into the model, to analyze and capture dataset statistics when it has evolved past a certain “divergence” threshold or if it has changed completely. Dataset statistics include only distribution histogram and frequencies to focus column data and importance.
See Feature Drift.
Prediction Drift
Prediction drift uses feature importance to monitor dataset statistics of the model output to look for deviations. Prior to training a model, features and target must be identified. In the case of an imported model that was trained externally, the data for that model must be identified.
After training a model, ModelOps computes and shows the distribution and statistics for each of the model features, and provides a visual overlay of training versus evaluation/scoring statistics.
See Prediction Drift.
Performance Drift
Performance drift involves using a metric to compare predicted and actual data results. If actual data is not available, you can use drift in prediction and feature distributions to identify changes in the model.
See Performance Drift.