Estimating Performance for Binary Classification

This tutorial explains how to use NannyML to estimate the performance of binary classification models in the absence of target data. To find out how CBPE estimates performance, read the explanation of Confidence-based Performance Estimation.

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_loan_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_loan_dataset()[1]

>>> display(reference_df.head(3))

>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='repaid',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'confusion_matrix',],
...     chunk_size=5000,
...     problem_type='classification_binary',
...     normalize_confusion_matrix='pred',
>>> )

>>> estimator.fit(reference_df)

>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> metric_fig = results.plot()
>>> metric_fig.show()

Advanced configuration

• Set up custom chunking [tutorial] [API reference]

• Set up custom thresholds [tutorial] [API reference]

Walkthrough

For simplicity this guide is based on a synthetic dataset included in the library, where the monitored model predicts whether a customer will repay a loan to buy a car. You can read more about this synthetic dataset here.

In order to monitor a model, NannyML needs to learn about it and set expectations from a reference dataset. Then NannyML can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

We start by loading the dataset we’ll be using:

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_loan_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_loan_dataset()[1]

>>> display(reference_df.head(3))

	car_value	salary_range	debt_to_income_ratio	loan_length	repaid_loan_on_prev_car	size_of_downpayment	driver_tenure	repaid	timestamp	y_pred_proba	y_pred
0	39811	40K - 60K €	0.63295	19	False	40%	0.212653	1	2018-01-01 00:00:00.000	0.99	1
1	12679	40K - 60K €	0.718627	7	True	10%	4.92755	0	2018-01-01 00:08:43.152	0.07	0
2	19847	40K - 60K €	0.721724	17	False	0%	0.520817	1	2018-01-01 00:17:26.304	1	1

Next we create the Confidence-based Performance Estimation (CBPE) estimator with a list of metrics, and an optional chunking specification. For more information about chunking you can check the setting up page and advanced guide.

Note

The list of metrics specifies which performance metrics of the monitored model will be estimated. The following metrics are currently supported:

• roc_auc

• f1

• precision

• recall

• specificity

• accuracy

• confusion_matrix

  • Normalization options:

    • None: returns counts

    • true: normalize by true class of observations

    • pred: normalize by predicted class of observations

    • all: normalize by all observations

>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='repaid',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'confusion_matrix',],
...     chunk_size=5000,
...     problem_type='classification_binary',
...     normalize_confusion_matrix='pred',
>>> )

>>> estimator.fit(reference_df)

The CBPE estimator is then fitted using the fit() method on the reference data.

The fitted estimator can be used to estimate performance on other data, for which performance cannot be calculated. Typically, this would be used on the latest production data where target is missing. In our example this is the analysis_df data.

NannyML can then output a dataframe that contains all the results. Let’s have a look at the results for analysis period only.

>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

	chunk key	chunk_index	start_index	end_index	start_date	end_date	period	roc_auc value	sampling_error	realized	upper_confidence_boundary	lower_confidence_boundary	upper_threshold	lower_threshold	alert	true_positive value	sampling_error	realized	upper_confidence_boundary	lower_confidence_boundary	upper_threshold	lower_threshold	alert	true_negative value	sampling_error	realized	upper_confidence_boundary	lower_confidence_boundary	upper_threshold	lower_threshold	alert	false_positive value	sampling_error	realized	upper_confidence_boundary	lower_confidence_boundary	upper_threshold	lower_threshold	alert	false_negative value	sampling_error	realized	upper_confidence_boundary	lower_confidence_boundary	upper_threshold	lower_threshold	alert
0	[0:4999]	0	0	4999	2018-10-30 18:00:00	2018-11-30 00:27:16.848000	analysis	0.968631	0.00181072	nan	0.974063	0.963198	0.97866	0.963317	False	0.957786	0.00412503	nan	0.970161	0.945411	0.967054	0.946681	False	0.925605	0.00498909	nan	0.940572	0.910637	0.943281	0.919069	False	0.042214	0.00412503	nan	0.0545891	0.0298389	0.0533188	0.0329461	False	0.0743953	0.00498909	nan	0.0893626	0.059428	0.0809314	0.0567187	False
1	[5000:9999]	1	5000	9999	2018-11-30 00:36:00	2018-12-30 07:03:16.848000	analysis	0.969044	0.00181072	nan	0.974476	0.963612	0.97866	0.963317	False	0.957956	0.00412503	nan	0.970331	0.94558	0.967054	0.946681	False	0.930102	0.00498909	nan	0.945069	0.915135	0.943281	0.919069	False	0.0420444	0.00412503	nan	0.0544195	0.0296694	0.0533188	0.0329461	False	0.0698982	0.00498909	nan	0.0848654	0.0549309	0.0809314	0.0567187	False
2	[10000:14999]	2	10000	14999	2018-12-30 07:12:00	2019-01-29 13:39:16.848000	analysis	0.969444	0.00181072	nan	0.974876	0.964012	0.97866	0.963317	False	0.958275	0.00412503	nan	0.970651	0.9459	0.967054	0.946681	False	0.933897	0.00498909	nan	0.948865	0.91893	0.943281	0.919069	False	0.0417245	0.00412503	nan	0.0540996	0.0293494	0.0533188	0.0329461	False	0.0661027	0.00498909	nan	0.08107	0.0511354	0.0809314	0.0567187	False
3	[15000:19999]	3	15000	19999	2019-01-29 13:48:00	2019-02-28 20:15:16.848000	analysis	0.969047	0.00181072	nan	0.974479	0.963615	0.97866	0.963317	False	0.956998	0.00412503	nan	0.969374	0.944623	0.967054	0.946681	False	0.933643	0.00498909	nan	0.94861	0.918675	0.943281	0.919069	False	0.0430015	0.00412503	nan	0.0553766	0.0306264	0.0533188	0.0329461	False	0.0663575	0.00498909	nan	0.0813248	0.0513902	0.0809314	0.0567187	False
4	[20000:24999]	4	20000	24999	2019-02-28 20:24:00	2019-03-31 02:51:16.848000	analysis	0.968873	0.00181072	nan	0.974305	0.963441	0.97866	0.963317	False	0.958208	0.00412503	nan	0.970583	0.945833	0.967054	0.946681	False	0.931676	0.00498909	nan	0.946644	0.916709	0.943281	0.919069	False	0.0417921	0.00412503	nan	0.0541671	0.029417	0.0533188	0.0329461	False	0.0683236	0.00498909	nan	0.0832909	0.0533564	0.0809314	0.0567187	False
5	[25000:29999]	5	25000	29999	2019-03-31 03:00:00	2019-04-30 09:27:16.848000	analysis	0.960478	0.00181072	nan	0.96591	0.955046	0.97866	0.963317	True	0.941318	0.00412503	nan	0.953693	0.928942	0.967054	0.946681	True	0.881311	0.00498909	nan	0.896278	0.866344	0.943281	0.919069	True	0.0586825	0.00412503	nan	0.0710575	0.0463074	0.0533188	0.0329461	True	0.118689	0.00498909	nan	0.133656	0.103722	0.0809314	0.0567187	True
6	[30000:34999]	6	30000	34999	2019-04-30 09:36:00	2019-05-30 16:03:16.848000	analysis	0.961134	0.00181072	nan	0.966566	0.955701	0.97866	0.963317	True	0.941868	0.00412503	nan	0.954243	0.929493	0.967054	0.946681	True	0.881941	0.00498909	nan	0.896908	0.866974	0.943281	0.919069	True	0.0581319	0.00412503	nan	0.070507	0.0457568	0.0533188	0.0329461	True	0.118059	0.00498909	nan	0.133026	0.103092	0.0809314	0.0567187	True
7	[35000:39999]	7	35000	39999	2019-05-30 16:12:00	2019-06-29 22:39:16.848000	analysis	0.960536	0.00181072	nan	0.965968	0.955104	0.97866	0.963317	True	0.939413	0.00412503	nan	0.951788	0.927038	0.967054	0.946681	True	0.884317	0.00498909	nan	0.899284	0.869349	0.943281	0.919069	True	0.0605867	0.00412503	nan	0.0729618	0.0482117	0.0533188	0.0329461	True	0.115683	0.00498909	nan	0.130651	0.100716	0.0809314	0.0567187	True
8	[40000:44999]	8	40000	44999	2019-06-29 22:48:00	2019-07-30 05:15:16.848000	analysis	0.961869	0.00181072	nan	0.967301	0.956437	0.97866	0.963317	True	0.943347	0.00412503	nan	0.955722	0.930972	0.967054	0.946681	True	0.887574	0.00498909	nan	0.902541	0.872606	0.943281	0.919069	True	0.0566531	0.00412503	nan	0.0690282	0.044278	0.0533188	0.0329461	True	0.112426	0.00498909	nan	0.127394	0.0974592	0.0809314	0.0567187	True
9	[45000:49999]	9	45000	49999	2019-07-30 05:24:00	2019-08-29 11:51:16.848000	analysis	0.960537	0.00181072	nan	0.965969	0.955104	0.97866	0.963317	True	0.941133	0.00412503	nan	0.953508	0.928758	0.967054	0.946681	True	0.881951	0.00498909	nan	0.896918	0.866984	0.943281	0.919069	True	0.0588671	0.00412503	nan	0.0712422	0.046492	0.0533188	0.0329461	True	0.118049	0.00498909	nan	0.133016	0.103082	0.0809314	0.0567187	True

Apart from chunk-related data, the results data have the following columns for each metric that was estimated:

• value - the estimate of a metric for a specific chunk.

• sampling_error - the estimate of the Sampling Error.

• realized - when target values are available for a chunk, the realized performance metric will also be calculated and included within the results.

• upper_confidence_boundary and lower_confidence_boundary - These values show the Confidence Band of the relevant metric and are equal to estimated value +/- 3 times the estimated Sampling Error.

• upper_threshold and lower_threshold - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during fit phase.

• alert - flag indicating potentially significant performance change. True if estimated performance crosses upper or lower threshold.

These results can be also plotted. Our plot contains several key elements.

• The purple dashed step plot shows the estimated performance in each chunk of the analysis period. Thick squared point markers indicate the middle of these chunks.

• The blue dashed step plot shows the estimated performance in each chunk of the reference period. Thick squared point markers indicate the middle of these chunks.

• The black vertical line splits the reference and analysis periods.

• The low-saturated colored area around the estimated performance indicates the sampling error.

• The red horizontal dashed lines show upper and lower thresholds for alerting purposes.

• If the estimated performance crosses the upper or lower threshold an alert is raised which is indicated with a red diamond-shaped point marker in the middle of the chunk.

Description of tabular results above explains how the confidence bands and thresholds are calculated. Additional information is shown in the hover (these are interactive plots, though only static views are included here).

>>> metric_fig = results.plot()
>>> metric_fig.show()

Insights

After reviewing the performance estimation results, we should be able to see any indications of performance change that NannyML has detected based upon the model’s inputs and outputs alone.

What’s next

The Data Drift functionality can help us to understand whether data drift is causing the performance problem. When the target values become available they can be compared with the estimated results.

You can learn more about the Confidence Based Performance Estimation and its limitations in the How it Works page.