Drift Detection for Multiclass Classification Model Outputs
Why Perform Drift Detection for Model Outputs
The distribution of model outputs tells us how likely it is that our population will do what the model predicts. If the model’s population changes, then our populations’ actions will be different. The difference in actions is very important to know as soon as possible because they directly affect the business results from operating a machine learning model.
Just The Code
>>> import nannyml as nml
>>> import pandas as pd
>>> from IPython.display import display
>>>
>>> reference_df = nml.load_synthetic_multiclass_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_multiclass_classification_dataset()[1]
>>>
>>> display(reference_df.head())
>>>
>>> calc = nml.StatisticalOutputDriftCalculator(
... y_pred='y_pred',
... y_pred_proba={
... 'prepaid_card': 'y_pred_proba_prepaid_card',
... 'upmarket_card': 'y_pred_proba_upmarket_card',
... 'highstreet_card': 'y_pred_proba_highstreet_card'
... },
... timestamp_column_name='timestamp')
>>>
>>> calc.fit(reference_df)
>>>
>>> results = calc.calculate(analysis_df)
>>>
>>> display(results.data)
>>>
>>> figure = results.plot(kind='predicted_labels_drift', plot_reference=True)
>>> figure.show()
>>>
>>> figure = results.plot(kind='predicted_labels_distribution', plot_reference=True)
>>> figure.show()
>>>
>>> for label in calc.y_pred_proba.keys():
... figure = results.plot(kind='prediction_drift', class_label=label, plot_reference=True)
... figure.show()
>>>
>>> for label in calc.y_pred_proba.keys():
... figure = results.plot(kind='prediction_distribution', class_label=label, plot_reference=True)
... figure.show()
Walkthrough
NannyML detects data drift for Model Outputs using the Univariate Drift Detection methodology.
In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods
Let’s start by loading some synthetic data provided by the NannyML package, and setting it up as our reference and analysis dataframes. This synthetic data is for a binary classification model, but multi-class classification can be handled in the same way.
>>> import nannyml as nml
>>> import pandas as pd
>>> from IPython.display import display
>>>
>>> reference_df = nml.load_synthetic_multiclass_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_multiclass_classification_dataset()[1]
>>>
>>> display(reference_df.head())
acq_channel |
app_behavioral_score |
requested_credit_limit |
app_channel |
credit_bureau_score |
stated_income |
is_customer |
period |
identifier |
timestamp |
y_pred_proba_prepaid_card |
y_pred_proba_highstreet_card |
y_pred_proba_upmarket_card |
y_pred |
y_true |
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
Partner3 |
1.80823 |
350 |
web |
309 |
15000 |
True |
reference |
60000 |
2020-05-02 02:01:30 |
0.97 |
0.03 |
0 |
prepaid_card |
prepaid_card |
1 |
Partner2 |
4.38257 |
500 |
mobile |
418 |
23000 |
True |
reference |
60001 |
2020-05-02 02:03:33 |
0.87 |
0.13 |
0 |
prepaid_card |
prepaid_card |
2 |
Partner2 |
-0.787575 |
400 |
web |
507 |
24000 |
False |
reference |
60002 |
2020-05-02 02:04:49 |
0.47 |
0.35 |
0.18 |
prepaid_card |
upmarket_card |
3 |
Partner3 |
-2.13177 |
300 |
mobile |
324 |
38000 |
False |
reference |
60003 |
2020-05-02 02:07:59 |
0.26 |
0.5 |
0.24 |
highstreet_card |
upmarket_card |
4 |
Partner3 |
-1.36294 |
450 |
mobile |
736 |
38000 |
True |
reference |
60004 |
2020-05-02 02:20:19 |
0.03 |
0.04 |
0.93 |
upmarket_card |
upmarket_card |
The StatisticalOutputDriftCalculator
class implements the functionality needed for drift detection in model outputs. First, the class is instantiated with appropriate parameters.
To check the model outputs for data drift, we only need to pass in the column header of the outputs as y_pred and y_pred_proba.
Then the fit()
method
is called on the reference data, so that the data baseline can be established.
Then the calculate()
method
calculates the drift results on the data provided. An example using it can be seen below.
>>> calc = nml.StatisticalOutputDriftCalculator(
... y_pred='y_pred',
... y_pred_proba={
... 'prepaid_card': 'y_pred_proba_prepaid_card',
... 'upmarket_card': 'y_pred_proba_upmarket_card',
... 'highstreet_card': 'y_pred_proba_highstreet_card'
... },
... timestamp_column_name='timestamp')
>>>
>>> calc.fit(reference_df)
>>>
>>> results = calc.calculate(analysis_df)
We can then display the results in a table, or as plots.
>>> display(results.data)
key |
start_index |
end_index |
start_date |
end_date |
period |
y_pred_chi2 |
y_pred_p_value |
y_pred_alert |
y_pred_threshold |
y_pred_proba_prepaid_card_dstat |
y_pred_proba_prepaid_card_p_value |
y_pred_proba_prepaid_card_alert |
y_pred_proba_prepaid_card_threshold |
y_pred_proba_upmarket_card_dstat |
y_pred_proba_upmarket_card_p_value |
y_pred_proba_upmarket_card_alert |
y_pred_proba_upmarket_card_threshold |
y_pred_proba_highstreet_card_dstat |
y_pred_proba_highstreet_card_p_value |
y_pred_proba_highstreet_card_alert |
y_pred_proba_highstreet_card_threshold |
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
[0:5999] |
0 |
5999 |
2020-09-01 03:10:01 |
2020-09-13 16:15:10 |
2.41991 |
0.298 |
False |
0.05 |
0.0133667 |
0.281 |
False |
0.05 |
0.0122833 |
0.38 |
False |
0.05 |
0.0057 |
0.994 |
False |
0.05 |
|
1 |
[6000:11999] |
6000 |
11999 |
2020-09-13 16:15:32 |
2020-09-25 19:48:42 |
1.26339 |
0.532 |
False |
0.05 |
0.0220333 |
0.01 |
True |
0.05 |
0.00845 |
0.828 |
False |
0.05 |
0.0135667 |
0.265 |
False |
0.05 |
|
2 |
[12000:17999] |
12000 |
17999 |
2020-09-25 19:50:04 |
2020-10-08 02:53:47 |
0.211705 |
0.9 |
False |
0.05 |
0.00931667 |
0.727 |
False |
0.05 |
0.00786667 |
0.886 |
False |
0.05 |
0.00845 |
0.828 |
False |
0.05 |
|
3 |
[18000:23999] |
18000 |
23999 |
2020-10-08 02:57:34 |
2020-10-20 15:48:19 |
1.04594 |
0.593 |
False |
0.05 |
0.0068 |
0.961 |
False |
0.05 |
0.0126167 |
0.347 |
False |
0.05 |
0.02025 |
0.022 |
True |
0.05 |
|
4 |
[24000:29999] |
24000 |
29999 |
2020-10-20 15:49:06 |
2020-11-01 22:04:40 |
2.89101 |
0.236 |
False |
0.05 |
0.0161333 |
0.116 |
False |
0.05 |
0.0126167 |
0.347 |
False |
0.05 |
0.01025 |
0.612 |
False |
0.05 |
|
5 |
[30000:35999] |
30000 |
35999 |
2020-11-01 22:04:59 |
2020-11-14 03:55:33 |
131.238 |
0 |
True |
0.05 |
0.174467 |
0 |
True |
0.05 |
0.1468 |
0 |
True |
0.05 |
0.2077 |
0 |
True |
0.05 |
|
6 |
[36000:41999] |
36000 |
41999 |
2020-11-14 03:55:49 |
2020-11-26 09:19:06 |
155.593 |
0 |
True |
0.05 |
0.1713 |
0 |
True |
0.05 |
0.144717 |
0 |
True |
0.05 |
0.210867 |
0 |
True |
0.05 |
|
7 |
[42000:47999] |
42000 |
47999 |
2020-11-26 09:19:22 |
2020-12-08 14:33:56 |
182.001 |
0 |
True |
0.05 |
0.170533 |
0 |
True |
0.05 |
0.140967 |
0 |
True |
0.05 |
0.2153 |
0 |
True |
0.05 |
|
8 |
[48000:53999] |
48000 |
53999 |
2020-12-08 14:34:25 |
2020-12-20 18:30:30 |
137.685 |
0 |
True |
0.05 |
0.173467 |
0 |
True |
0.05 |
0.14205 |
0 |
True |
0.05 |
0.209533 |
0 |
True |
0.05 |
|
9 |
[54000:59999] |
54000 |
59999 |
2020-12-20 18:31:09 |
2021-01-01 22:57:55 |
164.407 |
0 |
True |
0.05 |
0.1673 |
0 |
True |
0.05 |
0.14755 |
0 |
True |
0.05 |
0.20505 |
0 |
True |
0.05 |
NannyML can show the statistical properties of the drift in model outputs as a plot.
>>> for label in calc.y_pred_proba.keys():
... figure = results.plot(kind='prediction_drift', class_label=label, plot_reference=True)
... figure.show()
NannyML can also visualise how the distributions of the model predictions evolved over time.
>>> for label in calc.y_pred_proba.keys():
... figure = results.plot(kind='prediction_distribution', class_label=label, plot_reference=True)
... figure.show()
NannyML can show the statistical properties of the drift in the predicted labels as a plot.
>>> figure = results.plot(kind='predicted_labels_drift', plot_reference=True)
>>> figure.show()
NannyML can also visualise how the distributions of the predicted labels evolved over time.
>>> figure = results.plot(kind='predicted_labels_distribution', plot_reference=True)
>>> figure.show()
What Next
If required, the Performance Estimation functionality of NannyML can help provide estimates of the impact of the observed changes to Model Outputs.