Calculating Standard Performance Metrics for Multiclass Classification
Note
The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.
Just The Code
>>> import nannyml as nml
>>> from IPython.display import display
>>> reference_df, analysis_df, analysis_targets_df = nml.load_synthetic_multiclass_classification_dataset()
>>> analysis_df = analysis_df.merge(analysis_targets_df, left_index=True, right_index=True)
>>> display(reference_df.head(3))
>>> calc = nml.PerformanceCalculator(
... y_pred_proba={
... 'prepaid_card': 'y_pred_proba_prepaid_card',
... 'highstreet_card': 'y_pred_proba_highstreet_card',
... 'upmarket_card': 'y_pred_proba_upmarket_card'
... },
... y_pred='y_pred',
... y_true='y_true',
... timestamp_column_name='timestamp',
... problem_type='classification_multiclass',
... metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
... chunk_size=6000)
>>> calc.fit(reference_df)
>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())
>>> display(results.filter(period='reference').to_df())
>>> figure = results.plot()
>>> figure.show()
Advanced configuration
To learn how
Chunk
works and to set up custom chunkings check out the chunking tutorialTo learn how
ConstantThreshold
works and to set up custom threshold check out the thresholds tutorial
Walkthrough
For simplicity this guide is based on a synthetic dataset where the monitored model predicts which type of credit card product new customers should be assigned to. Check out Credit Card Dataset to learn more about this dataset.
In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods
The analysis_targets_df
dataframe contains the target results of the analysis period. This is kept separate in the
synthetic data because it is not used during performance estimation. But it is required
to calculate performance, so the first thing we need to in this case is set up the right data in the right dataframes.
The analysis target values are joined on the analysis frame by the identifier column.
>>> import nannyml as nml
>>> from IPython.display import display
>>> reference_df, analysis_df, analysis_targets_df = nml.load_synthetic_multiclass_classification_dataset()
>>> analysis_df = analysis_df.merge(analysis_targets_df, left_index=True, right_index=True)
>>> display(reference_df.head(3))
id |
acq_channel |
app_behavioral_score |
requested_credit_limit |
app_channel |
credit_bureau_score |
stated_income |
is_customer |
timestamp |
y_pred_proba_prepaid_card |
y_pred_proba_highstreet_card |
y_pred_proba_upmarket_card |
y_pred |
y_true |
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
0 |
Partner3 |
1.80823 |
350 |
web |
309 |
15000 |
True |
2020-05-02 02:01:30 |
0.97 |
0.03 |
0 |
prepaid_card |
prepaid_card |
1 |
1 |
Partner2 |
4.38257 |
500 |
mobile |
418 |
23000 |
True |
2020-05-02 02:03:33 |
0.87 |
0.13 |
0 |
prepaid_card |
prepaid_card |
2 |
2 |
Partner2 |
-0.787575 |
400 |
web |
507 |
24000 |
False |
2020-05-02 02:04:49 |
0.47 |
0.35 |
0.18 |
prepaid_card |
upmarket_card |
Next a PerformanceCalculator
is created using a list of metrics to calculate
(or just one metric), the data columns required for these metrics, and an optional chunking specification.
The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:
roc_auc
- one-vs-the-rest, macro-averagedf1
- macro-averagedprecision
- macro-averagedrecall
- macro-averagedspecificity
- macro-averagedaccuracy
average_precision
- macro-averaged
For more information on metrics, check the metrics
module.
>>> calc = nml.PerformanceCalculator(
... y_pred_proba={
... 'prepaid_card': 'y_pred_proba_prepaid_card',
... 'highstreet_card': 'y_pred_proba_highstreet_card',
... 'upmarket_card': 'y_pred_proba_upmarket_card'
... },
... y_pred='y_pred',
... y_true='y_true',
... timestamp_column_name='timestamp',
... problem_type='classification_multiclass',
... metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
... chunk_size=6000)
>>> calc.fit(reference_df)
The new PerformanceCalculator
is fitted using the
fit()
method on the reference data.
The fitted PerformanceCalculator
can then be used to calculate
realized performance metrics on all data which has target values available with the
calculate()
method.
NannyML can output a dataframe that contains all the results of the analysis data.
>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())
chunk
key
|
chunk_index
|
start_index
|
end_index
|
start_date
|
end_date
|
period
|
targets_missing_rate
|
roc_auc
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
f1
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
precision
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
recall
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
specificity
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
accuracy
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
[0:5999] |
0 |
0 |
5999 |
2020-09-01 03:10:01 |
2020-09-13 16:15:10 |
analysis |
0 |
0.00214318 |
0.907595 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.751103 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.75127 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.751033 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.87555 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.751167 |
0.765102 |
0.741231 |
False |
1 |
[6000:11999] |
1 |
6000 |
11999 |
2020-09-13 16:15:32 |
2020-09-25 19:48:42 |
analysis |
0 |
0.00214318 |
0.910534 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.763045 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.763125 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.763148 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.881508 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.763 |
0.765102 |
0.741231 |
False |
2 |
[12000:17999] |
2 |
12000 |
17999 |
2020-09-25 19:50:04 |
2020-10-08 02:53:47 |
analysis |
0 |
0.00214318 |
0.909414 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.758487 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.758503 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.758484 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.879367 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.758667 |
0.765102 |
0.741231 |
False |
3 |
[18000:23999] |
3 |
18000 |
23999 |
2020-10-08 02:57:34 |
2020-10-20 15:48:19 |
analysis |
0 |
0.00214318 |
0.911577 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.758944 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.758973 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.758986 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.87963 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.759167 |
0.765102 |
0.741231 |
False |
4 |
[24000:29999] |
4 |
24000 |
29999 |
2020-10-20 15:49:06 |
2020-11-01 22:04:40 |
analysis |
0 |
0.00214318 |
0.907533 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.757964 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.75795 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.757979 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.878986 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.758 |
0.765102 |
0.741231 |
False |
5 |
[30000:35999] |
5 |
30000 |
35999 |
2020-11-01 22:04:59 |
2020-11-14 03:55:33 |
analysis |
0 |
0.00214318 |
0.748588 |
0.913516 |
0.900902 |
True |
0.00565227 |
0.557112 |
0.764944 |
0.741254 |
True |
0.00556588 |
0.559696 |
0.764978 |
0.741261 |
True |
0.00556473 |
0.557829 |
0.765026 |
0.741261 |
True |
0.0030024 |
0.779905 |
0.882586 |
0.870566 |
True |
0.00556637 |
0.560833 |
0.765102 |
0.741231 |
True |
6 |
[36000:41999] |
6 |
36000 |
41999 |
2020-11-14 03:55:49 |
2020-11-26 09:19:06 |
analysis |
0 |
0.00214318 |
0.751137 |
0.913516 |
0.900902 |
True |
0.00565227 |
0.559148 |
0.764944 |
0.741254 |
True |
0.00556588 |
0.562915 |
0.764978 |
0.741261 |
True |
0.00556473 |
0.56017 |
0.765026 |
0.741261 |
True |
0.0030024 |
0.780676 |
0.882586 |
0.870566 |
True |
0.00556637 |
0.562333 |
0.765102 |
0.741231 |
True |
7 |
[42000:47999] |
7 |
42000 |
47999 |
2020-11-26 09:19:22 |
2020-12-08 14:33:56 |
analysis |
0 |
0.00214318 |
0.756399 |
0.913516 |
0.900902 |
True |
0.00565227 |
0.565055 |
0.764944 |
0.741254 |
True |
0.00556588 |
0.569069 |
0.764978 |
0.741261 |
True |
0.00556473 |
0.565943 |
0.765026 |
0.741261 |
True |
0.0030024 |
0.784223 |
0.882586 |
0.870566 |
True |
0.00556637 |
0.569833 |
0.765102 |
0.741231 |
True |
8 |
[48000:53999] |
8 |
48000 |
53999 |
2020-12-08 14:34:25 |
2020-12-20 18:30:30 |
analysis |
0 |
0.00214318 |
0.758561 |
0.913516 |
0.900902 |
True |
0.00565227 |
0.563897 |
0.764944 |
0.741254 |
True |
0.00556588 |
0.566673 |
0.764978 |
0.741261 |
True |
0.00556473 |
0.564723 |
0.765026 |
0.741261 |
True |
0.0030024 |
0.783422 |
0.882586 |
0.870566 |
True |
0.00556637 |
0.567833 |
0.765102 |
0.741231 |
True |
9 |
[54000:59999] |
9 |
54000 |
59999 |
2020-12-20 18:31:09 |
2021-01-01 22:57:55 |
analysis |
0 |
0.00214318 |
0.753937 |
0.913516 |
0.900902 |
True |
0.00565227 |
0.561644 |
0.764944 |
0.741254 |
True |
0.00556588 |
0.565129 |
0.764978 |
0.741261 |
True |
0.00556473 |
0.562772 |
0.765026 |
0.741261 |
True |
0.0030024 |
0.782429 |
0.882586 |
0.870566 |
True |
0.00556637 |
0.566 |
0.765102 |
0.741231 |
True |
The results from the reference data are also available.
>>> display(results.filter(period='reference').to_df())
chunk
key
|
chunk_index
|
start_index
|
end_index
|
start_date
|
end_date
|
period
|
targets_missing_rate
|
roc_auc
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
f1
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
precision
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
recall
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
specificity
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
accuracy
sampling_error
|
value
|
upper_threshold
|
lower_threshold
|
alert
|
|
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
[0:5999] |
0 |
0 |
5999 |
2020-05-02 02:01:30 |
2020-05-14 12:25:35 |
reference |
0 |
0.00214318 |
0.90476 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.750532 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.7505 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.750576 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.875226 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.7505 |
0.765102 |
0.741231 |
False |
1 |
[6000:11999] |
1 |
6000 |
11999 |
2020-05-14 12:29:25 |
2020-05-26 18:27:42 |
reference |
0 |
0.00214318 |
0.905917 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.751148 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.751142 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.751157 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.875424 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.751 |
0.765102 |
0.741231 |
False |
2 |
[12000:17999] |
2 |
12000 |
17999 |
2020-05-26 18:31:06 |
2020-06-07 19:55:45 |
reference |
0 |
0.00214318 |
0.909329 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.75714 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.75728 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.757174 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.878628 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.757167 |
0.765102 |
0.741231 |
False |
3 |
[18000:23999] |
3 |
18000 |
23999 |
2020-06-07 19:58:39 |
2020-06-19 19:42:20 |
reference |
0 |
0.00214318 |
0.906731 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.750274 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.750415 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.750241 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.875163 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.750333 |
0.765102 |
0.741231 |
False |
4 |
[24000:29999] |
4 |
24000 |
29999 |
2020-06-19 19:44:14 |
2020-07-02 01:58:05 |
reference |
0 |
0.00214318 |
0.910577 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.759144 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.759175 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.759197 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.879686 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.759333 |
0.765102 |
0.741231 |
False |
5 |
[30000:35999] |
5 |
30000 |
35999 |
2020-07-02 02:06:56 |
2020-07-14 08:14:04 |
reference |
0 |
0.00214318 |
0.904577 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.74863 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.74866 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.748657 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.874329 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.748667 |
0.765102 |
0.741231 |
False |
6 |
[36000:41999] |
6 |
36000 |
41999 |
2020-07-14 08:14:08 |
2020-07-26 12:55:42 |
reference |
0 |
0.00214318 |
0.906673 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.752763 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.752684 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.752944 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.876407 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.752833 |
0.765102 |
0.741231 |
False |
7 |
[42000:47999] |
7 |
42000 |
47999 |
2020-07-26 12:57:37 |
2020-08-07 16:32:15 |
reference |
0 |
0.00214318 |
0.908703 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.755883 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.75582 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.756006 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.877891 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.755833 |
0.765102 |
0.741231 |
False |
8 |
[48000:53999] |
8 |
48000 |
53999 |
2020-08-07 16:33:44 |
2020-08-20 00:06:08 |
reference |
0 |
0.00214318 |
0.905072 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.74742 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.747441 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.747436 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.873832 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.747667 |
0.765102 |
0.741231 |
False |
9 |
[54000:59999] |
9 |
54000 |
59999 |
2020-08-20 00:07:58 |
2020-09-01 03:03:23 |
reference |
0 |
0.00214318 |
0.909749 |
0.913516 |
0.900902 |
False |
0.00565227 |
0.758055 |
0.764944 |
0.741254 |
False |
0.00556588 |
0.758077 |
0.764978 |
0.741261 |
False |
0.00556473 |
0.758052 |
0.765026 |
0.741261 |
False |
0.0030024 |
0.879177 |
0.882586 |
0.870566 |
False |
0.00556637 |
0.758333 |
0.765102 |
0.741231 |
False |
Apart from chunk-related data, the results data have a set of columns for each calculated metric.
targets_missing_rate - The fraction of missing target data.
value - the realized metric value for a specific chunk.
sampling_error - the estimate of the Sampling Error.
upper_threshold and lower_threshold - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during
fit
phase. You can also set up custom thresholds using constant or standard deviations thresholds, to learn more about it check out our tutorial on thresholds.alert - flag indicating potentially significant performance change.
True
if estimated performance crosses upper or lower threshold.
The results can be plotted for visual inspection. Our plot contains several key elements.
The blue step plot shows the performance in each chunk of the provided data. Thick squared point markers indicate the middle of these chunks.
The gray vertical line splits the reference and analysis data periods.
The red horizontal dashed lines show upper and lower thresholds that indicate the range of expected performance values.
The red diamond-shaped point markers in the middle of a chunk indicate that an alert has been raised. Alerts are caused by the performance crossing the upper or lower threshold.
>>> figure = results.plot()
>>> figure.show()
Insights
After reviewing the performance calculation results, we should be able to clearly see how the model is performing against the targets, according to whatever metrics we wish to track.
What Next
If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes. We can also plot the realized performance and compare it with the estimated results.
It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.