Monitoring Realized Performance for Binary Classification

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df, analysis_df, analysis_target_df = nml.load_synthetic_car_loan_dataset()

>>> analysis_df = analysis_df.merge(analysis_target_df, left_index=True, right_index=True)

>>> display(reference_df.head(3))

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='repaid',
...     timestamp_column_name='timestamp',
...     problem_type='classification_binary',
...     metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
...     chunk_size=5000)

>>> calc.fit(reference_df)

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> display(results.filter(period='reference').to_df())

>>> figure = results.plot()
>>> figure.show()

Advanced configuration

• Set up custom chunking [tutorial] [API reference]

• Set up custom thresholds [tutorial] [API reference]

Walkthrough

For simplicity this guide is based on a synthetic dataset included in the library, where the monitored model predicts whether a customer will repay a loan to buy a car. You can read more about this synthetic dataset here.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

The analysis_targets dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation.. But it is required to calculate performance, so the first thing we need to in this case is set up the right data in the right dataframes.

The analysis target values are joined on the analysis frame by their index. Your dataset may already contain the target column, so you may skip this join.

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df, analysis_df, analysis_target_df = nml.load_synthetic_car_loan_dataset()

>>> analysis_df = analysis_df.merge(analysis_target_df, left_index=True, right_index=True)

>>> display(reference_df.head(3))

	car_value	salary_range	debt_to_income_ratio	loan_length	repaid_loan_on_prev_car	size_of_downpayment	driver_tenure	repaid	timestamp	y_pred_proba	y_pred
0	39811	40K - 60K €	0.63295	19	False	40%	0.212653	1	2018-01-01 00:00:00.000	0.99	1
1	12679	40K - 60K €	0.718627	7	True	10%	4.92755	0	2018-01-01 00:08:43.152	0.07	0
2	19847	40K - 60K €	0.721724	17	False	0%	0.520817	1	2018-01-01 00:17:26.304	1	1

Next a PerformanceCalculator is created using a list of metrics to calculate (or just one metric), the data columns required for these metrics, and an optional chunking specification.

Note

The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:

• roc_auc

• f1

• precision

• recall

• specificity

• accuracy

For more information on metrics, check the metrics module.

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='repaid',
...     timestamp_column_name='timestamp',
...     problem_type='classification_binary',
...     metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
...     chunk_size=5000)

>>> calc.fit(reference_df)

The new PerformanceCalculator is fitted using the fit() method on the reference data.

The fitted PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the calculate() method. NannyML can output a dataframe that contains all the results of the analysis data.

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

	chunk key	chunk_index	start_index	end_index	start_date	end_date	period	targets_missing_rate	roc_auc sampling_error	value	upper_threshold	lower_threshold	alert	f1 sampling_error	value	upper_threshold	lower_threshold	alert	precision sampling_error	value	upper_threshold	lower_threshold	alert	recall sampling_error	value	upper_threshold	lower_threshold	alert	specificity sampling_error	value	upper_threshold	lower_threshold	alert	accuracy sampling_error	value	upper_threshold	lower_threshold	alert
0	[0:4999]	0	0	4999	2018-10-30 18:00:00	2018-11-30 00:27:16.848000	analysis	0	0.00181072	0.970962	0.97866	0.963317	False	0.00643972	0.943344	0.95085	0.93466	False	0.00412503	0.95666	0.967054	0.946681	False	0.00513664	0.930394	0.941033	0.9171	False	0.00400118	0.954847	0.967647	0.948756	False	0.0032614	0.9422	0.951421	0.935859	False
1	[5000:9999]	1	5000	9999	2018-11-30 00:36:00	2018-12-30 07:03:16.848000	analysis	0	0.00181072	0.970248	0.97866	0.963317	False	0.00643972	0.940178	0.95085	0.93466	False	0.00412503	0.957016	0.967054	0.946681	False	0.00513664	0.923922	0.941033	0.9171	False	0.00400118	0.959874	0.967647	0.948756	False	0.0032614	0.9422	0.951421	0.935859	False
2	[10000:14999]	2	10000	14999	2018-12-30 07:12:00	2019-01-29 13:39:16.848000	analysis	0	0.00181072	0.976282	0.97866	0.963317	False	0.00643972	0.948201	0.95085	0.93466	False	0.00412503	0.958368	0.967054	0.946681	False	0.00513664	0.938246	0.941033	0.9171	False	0.00400118	0.961494	0.967647	0.948756	False	0.0032614	0.9502	0.951421	0.935859	False
3	[15000:19999]	3	15000	19999	2019-01-29 13:48:00	2019-02-28 20:15:16.848000	analysis	0	0.00181072	0.967721	0.97866	0.963317	False	0.00643972	0.942142	0.95085	0.93466	False	0.00412503	0.959866	0.967054	0.946681	False	0.00513664	0.92506	0.941033	0.9171	False	0.00400118	0.961875	0.967647	0.948756	False	0.0032614	0.9436	0.951421	0.935859	False
4	[20000:24999]	4	20000	24999	2019-02-28 20:24:00	2019-03-31 02:51:16.848000	analysis	0	0.00181072	0.969886	0.97866	0.963317	False	0.00643972	0.940488	0.95085	0.93466	False	0.00412503	0.953764	0.967054	0.946681	False	0.00513664	0.927577	0.941033	0.9171	False	0.00400118	0.954564	0.967647	0.948756	False	0.0032614	0.941	0.951421	0.935859	False
5	[25000:29999]	5	25000	29999	2019-03-31 03:00:00	2019-04-30 09:27:16.848000	analysis	0	0.00181072	0.96005	0.97866	0.963317	True	0.00643972	0.913685	0.95085	0.93466	True	0.00412503	0.922449	0.967054	0.946681	True	0.00513664	0.905086	0.941033	0.9171	True	0.00400118	0.924091	0.967647	0.948756	True	0.0032614	0.9146	0.951421	0.935859	True
6	[30000:34999]	6	30000	34999	2019-04-30 09:36:00	2019-05-30 16:03:16.848000	analysis	0	0.00181072	0.95853	0.97866	0.963317	True	0.00643972	0.914954	0.95085	0.93466	True	0.00412503	0.931473	0.967054	0.946681	True	0.00513664	0.89901	0.941033	0.9171	True	0.00400118	0.932525	0.967647	0.948756	True	0.0032614	0.9156	0.951421	0.935859	True
7	[35000:39999]	7	35000	39999	2019-05-30 16:12:00	2019-06-29 22:39:16.848000	analysis	0	0.00181072	0.959041	0.97866	0.963317	True	0.00643972	0.913395	0.95085	0.93466	True	0.00412503	0.925379	0.967054	0.946681	True	0.00513664	0.901718	0.941033	0.9171	True	0.00400118	0.927113	0.967647	0.948756	True	0.0032614	0.9144	0.951421	0.935859	True
8	[40000:44999]	8	40000	44999	2019-06-29 22:48:00	2019-07-30 05:15:16.848000	analysis	0	0.00181072	0.963094	0.97866	0.963317	True	0.00643972	0.920589	0.95085	0.93466	True	0.00412503	0.935524	0.967054	0.946681	True	0.00513664	0.906124	0.941033	0.9171	True	0.00400118	0.938443	0.967647	0.948756	True	0.0032614	0.9224	0.951421	0.935859	True
9	[45000:49999]	9	45000	49999	2019-07-30 05:24:00	2019-08-29 11:51:16.848000	analysis	0	0.00181072	0.957556	0.97866	0.963317	True	0.00643972	0.913498	0.95085	0.93466	True	0.00412503	0.921304	0.967054	0.946681	True	0.00513664	0.905823	0.941033	0.9171	True	0.00400118	0.918886	0.967647	0.948756	True	0.0032614	0.9122	0.951421	0.935859	True

There results from the reference data are also available.

>>> display(results.filter(period='reference').to_df())

	chunk key	chunk_index	start_index	end_index	start_date	end_date	period	targets_missing_rate	roc_auc sampling_error	value	upper_threshold	lower_threshold	alert	f1 sampling_error	value	upper_threshold	lower_threshold	alert	precision sampling_error	value	upper_threshold	lower_threshold	alert	recall sampling_error	value	upper_threshold	lower_threshold	alert	specificity sampling_error	value	upper_threshold	lower_threshold	alert	accuracy sampling_error	value	upper_threshold	lower_threshold	alert
0	[0:4999]	0	0	4999	2018-01-01 00:00:00	2018-01-31 06:27:16.848000	reference	0	0.00181072	0.976253	0.97866	0.963317	False	0.00643972	0.944707	0.95085	0.93466	False	0.00412503	0.960301	0.967054	0.946681	False	0.00513664	0.929612	0.941033	0.9171	False	0.00400118	0.962421	0.967647	0.948756	False	0.0032614	0.9462	0.951421	0.935859	False
1	[5000:9999]	1	5000	9999	2018-01-31 06:36:00	2018-03-02 13:03:16.848000	reference	0	0.00181072	0.969045	0.97866	0.963317	False	0.00643972	0.937564	0.95085	0.93466	False	0.00412503	0.95268	0.967054	0.946681	False	0.00513664	0.922921	0.941033	0.9171	False	0.00400118	0.955424	0.967647	0.948756	False	0.0032614	0.9394	0.951421	0.935859	False
2	[10000:14999]	2	10000	14999	2018-03-02 13:12:00	2018-04-01 19:39:16.848000	reference	0	0.00181072	0.971742	0.97866	0.963317	False	0.00643972	0.945973	0.95085	0.93466	False	0.00412503	0.959658	0.967054	0.946681	False	0.00513664	0.932673	0.941033	0.9171	False	0.00400118	0.96	0.967647	0.948756	False	0.0032614	0.9462	0.951421	0.935859	False
3	[15000:19999]	3	15000	19999	2018-04-01 19:48:00	2018-05-02 02:15:16.848000	reference	0	0.00181072	0.971642	0.97866	0.963317	False	0.00643972	0.943212	0.95085	0.93466	False	0.00412503	0.95942	0.967054	0.946681	False	0.00513664	0.927542	0.941033	0.9171	False	0.00400118	0.960831	0.967647	0.948756	False	0.0032614	0.9442	0.951421	0.935859	False
4	[20000:24999]	4	20000	24999	2018-05-02 02:24:00	2018-06-01 08:51:16.848000	reference	0	0.00181072	0.969085	0.97866	0.963317	False	0.00643972	0.937989	0.95085	0.93466	False	0.00412503	0.955047	0.967054	0.946681	False	0.00513664	0.92153	0.941033	0.9171	False	0.00400118	0.955357	0.967647	0.948756	False	0.0032614	0.9382	0.951421	0.935859	False
5	[25000:29999]	5	25000	29999	2018-06-01 09:00:00	2018-07-01 15:27:16.848000	reference	0	0.00181072	0.967364	0.97866	0.963317	False	0.00643972	0.942581	0.95085	0.93466	False	0.00412503	0.952975	0.967054	0.946681	False	0.00513664	0.93241	0.941033	0.9171	False	0.00400118	0.955582	0.967647	0.948756	False	0.0032614	0.9442	0.951421	0.935859	False
6	[30000:34999]	6	30000	34999	2018-07-01 15:36:00	2018-07-31 22:03:16.848000	reference	0	0.00181072	0.968692	0.97866	0.963317	False	0.00643972	0.94414	0.95085	0.93466	False	0.00412503	0.957561	0.967054	0.946681	False	0.00513664	0.93109	0.941033	0.9171	False	0.00400118	0.958866	0.967647	0.948756	False	0.0032614	0.945	0.951421	0.935859	False
7	[35000:39999]	7	35000	39999	2018-07-31 22:12:00	2018-08-31 04:39:16.848000	reference	0	0.00181072	0.970205	0.97866	0.963317	False	0.00643972	0.944612	0.95085	0.93466	False	0.00412503	0.955972	0.967054	0.946681	False	0.00513664	0.933519	0.941033	0.9171	False	0.00400118	0.956592	0.967647	0.948756	False	0.0032614	0.945	0.951421	0.935859	False
8	[40000:44999]	8	40000	44999	2018-08-31 04:48:00	2018-09-30 11:15:16.848000	reference	0	0.00181072	0.974096	0.97866	0.963317	False	0.00643972	0.944523	0.95085	0.93466	False	0.00412503	0.962582	0.967054	0.946681	False	0.00513664	0.927129	0.941033	0.9171	False	0.00400118	0.963232	0.967647	0.948756	False	0.0032614	0.945	0.951421	0.935859	False
9	[45000:49999]	9	45000	49999	2018-09-30 11:24:00	2018-10-30 17:51:16.848000	reference	0	0.00181072	0.971757	0.97866	0.963317	False	0.00643972	0.942249	0.95085	0.93466	False	0.00412503	0.952478	0.967054	0.946681	False	0.00513664	0.932237	0.941033	0.9171	False	0.00400118	0.953711	0.967647	0.948756	False	0.0032614	0.943	0.951421	0.935859	False

Apart from chunking and chunk and period-related columns, the results data have a set of columns for each calculated metric.

• targets_missing_rate - The fraction of missing target data.

• value - the realized metric value for a specific chunk.

• sampling_error - the estimate of the Sampling Error.

• upper_threshold and lower_threshold - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during fit phase.

• alert - flag indicating potentially significant performance change. True if estimated performance crosses upper or lower threshold.

The results can be plotted for visual inspection.

>>> figure = results.plot()
>>> figure.show()

Insights

After reviewing the performance calculation results, we should be able to clearly see how the model is performing against the targets, according to whatever metrics we wish to track.

What Next

If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes.

It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.