Estimating Performance for Binary Classification

This tutorial explains how to use NannyML to estimate the performance of binary classification models in the absence of target data. To find out how CBPE estimates performance, read the explanation of Confidence-based Performance Estimation.

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_binary_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_binary_classification_dataset()[1]

>>> display(reference_df.head(3))

>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='work_home_actual',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=5000,
...     problem_type='classification_binary',
>>> )

>>> estimator.fit(reference_df)

>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> metric_fig = results.filter(period='analysis').plot()
>>> metric_fig.show()

>>> metric_fig = results.plot()
>>> metric_fig.show()

Walkthrough

For simplicity this guide is based on a synthetic dataset included in the library, where the monitored model predicts whether an employee will work from home. You can read more about this synthetic dataset here.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

We start by loading the dataset we’ll be using:

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_binary_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_binary_classification_dataset()[1]

>>> display(reference_df.head(3))

distance_from_office

salary_range

gas_price_per_litre

public_transportation_cost

wfh_prev_workday

workday

tenure

identifier

work_home_actual

timestamp

y_pred_proba

period

y_pred

0

5.96225

40K - 60K €

2.11948

8.56806

False

Friday

0.212653

0

1

2014-05-09 22:27:20

0.99

reference

1

1

0.535872

40K - 60K €

2.3572

5.42538

True

Tuesday

4.92755

1

0

2014-05-09 22:59:32

0.07

reference

0

2

1.96952

40K - 60K €

2.36685

8.24716

False

Monday

0.520817

2

1

2014-05-09 23:48:25

1

reference

1

Next we create the Confidence-based Performance Estimation (CBPE) estimator with a list of metrics, and an optional chunking specification. For more information about chunking you can check the setting up page and advanced guide.

The list of metrics specifies which performance metrics of the monitored model will be estimated. The following metrics are currently supported:

  • roc_auc - one-vs-the-rest, macro-averaged

  • f1 - macro-averaged

  • precision - macro-averaged

  • recall - macro-averaged

  • specificity - macro-averaged

  • accuracy

>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='work_home_actual',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=5000,
...     problem_type='classification_binary',
>>> )

>>> estimator.fit(reference_df)

The CBPE estimator is then fitted using the fit() method on the reference data.

The fitted estimator can be used to estimate performance on other data, for which performance cannot be calculated. Typically, this would be used on the latest production data where target is missing. In our example this is the analysis_df data.

NannyML can then output a dataframe that contains all the results. Let’s have a look at the results for analysis period only.

>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

(‘chunk’, ‘key’)

(‘chunk’, ‘chunk_index’)

(‘chunk’, ‘start_index’)

(‘chunk’, ‘end_index’)

(‘chunk’, ‘start_date’)

(‘chunk’, ‘end_date’)

(‘chunk’, ‘period’)

(‘roc_auc’, ‘sampling_error’)

(‘roc_auc’, ‘realized’)

(‘roc_auc’, ‘value’)

(‘roc_auc’, ‘upper_confidence_boundary’)

(‘roc_auc’, ‘lower_confidence_boundary’)

(‘roc_auc’, ‘upper_threshold’)

(‘roc_auc’, ‘lower_threshold’)

(‘roc_auc’, ‘alert’)

(‘f1’, ‘sampling_error’)

(‘f1’, ‘realized’)

(‘f1’, ‘value’)

(‘f1’, ‘upper_confidence_boundary’)

(‘f1’, ‘lower_confidence_boundary’)

(‘f1’, ‘upper_threshold’)

(‘f1’, ‘lower_threshold’)

(‘f1’, ‘alert’)

0

[0:4999]

0

0

4999

2017-08-31 04:20:00

2018-01-02 00:45:44

analysis

0.00181072

nan

0.968631

0.974063

0.963198

0.97866

0.963317

False

0.00610429

nan

0.948555

0.966868

0.930242

0.961094

0.935047

False

1

[5000:9999]

1

5000

9999

2018-01-02 01:13:11

2018-05-01 13:10:10

analysis

0.00181072

nan

0.969044

0.974476

0.963612

0.97866

0.963317

False

0.00610429

nan

0.946578

0.964891

0.928265

0.961094

0.935047

False

2

[10000:14999]

2

10000

14999

2018-05-01 14:25:25

2018-09-01 15:40:40

analysis

0.00181072

nan

0.969444

0.974876

0.964012

0.97866

0.963317

False

0.00610429

nan

0.948807

0.96712

0.930494

0.961094

0.935047

False

3

[15000:19999]

3

15000

19999

2018-09-01 16:19:07

2018-12-31 10:11:21

analysis

0.00181072

nan

0.969047

0.974479

0.963615

0.97866

0.963317

False

0.00610429

nan

0.947655

0.965968

0.929342

0.961094

0.935047

False

4

[20000:24999]

4

20000

24999

2018-12-31 10:38:45

2019-04-30 11:01:30

analysis

0.00181072

nan

0.968873

0.974305

0.963441

0.97866

0.963317

False

0.00610429

nan

0.948834

0.967147

0.930521

0.961094

0.935047

False

5

[25000:29999]

5

25000

29999

2019-04-30 11:02:00

2019-09-01 00:24:27

analysis

0.00181072

nan

0.960478

0.96591

0.955046

0.97866

0.963317

True

0.00610429

nan

0.923233

0.941545

0.90492

0.961094

0.935047

True

6

[30000:34999]

6

30000

34999

2019-09-01 00:28:54

2019-12-31 09:09:12

analysis

0.00181072

nan

0.961134

0.966566

0.955701

0.97866

0.963317

True

0.00610429

nan

0.923596

0.941909

0.905283

0.961094

0.935047

True

7

[35000:39999]

7

35000

39999

2019-12-31 10:07:15

2020-04-30 11:46:53

analysis

0.00181072

nan

0.960536

0.965968

0.955104

0.97866

0.963317

True

0.00610429

nan

0.922902

0.941215

0.904589

0.961094

0.935047

True

8

[40000:44999]

8

40000

44999

2020-04-30 12:04:32

2020-09-01 02:46:02

analysis

0.00181072

nan

0.961869

0.967301

0.956437

0.97866

0.963317

True

0.00610429

nan

0.924172

0.942485

0.905859

0.961094

0.935047

True

9

[45000:49999]

9

45000

49999

2020-09-01 02:46:13

2021-01-01 04:29:32

analysis

0.00181072

nan

0.960537

0.965969

0.955104

0.97866

0.963317

True

0.00610429

nan

0.924915

0.943228

0.906602

0.961094

0.935047

True

Apart from chunk-related data, the results data have the following columns for each metric that was estimated:

  • realized_<metric> - when target values are available for a chunk, the realized performance metric will also be calculated and included within the results.

  • estimated_<metric> - the estimate of a metric for a specific chunk,

  • upper_confidence_<metric> and lower_confidence_<metric> - These values show the Confidence Band of the relevant metric and are equal to estimated value +/- 3 times the estimated Sampling Error.

  • upper_threshold_<metric> and lower_threshold_<metric> - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during fit phase.

  • alert_<metric> - flag indicating potentially significant performance change. True if estimated performance crosses upper or lower threshold.

These results can be also plotted. Our plot contains several key elements.

  • The purple dashed step plot shows the estimated performance in each chunk of the analysis period. Thick squared point markers indicate the middle of these chunks.

  • The low-saturated purple area around the estimated performance indicates the sampling error.

  • The red horizontal dashed lines show upper and lower thresholds for alerting purposes.

  • If the estimated performance crosses the upper or lower threshold an alert is raised which is indicated with a red, low-saturated background in the whole width of the relevant chunk. This is additionally indicated by a red, diamond-shaped point marker in the middle of the chunk.

Description of tabular results above explains how the confidence bands and thresholds are calculated. Additional information is shown in the hover (these are interactive plots, though only static views are included here).

>>> metric_fig = results.filter(period='analysis').plot()
>>> metric_fig.show()
../../_images/tutorial-perf-est-guide-analysis.svg

To get a better context let’s also plot estimation of performance on analysis data together with calculated performance on the reference period (where the target was available).

  • The right-hand side of the plot shows the estimated performance for the analysis period, as before.

  • The purple dashed vertical line splits the reference and analysis periods.

  • On the left-hand side of the line, the actual model performance (not estimation!) is plotted with a solid light blue line. This facilitates comparison of the estimation against the reference period, and sets expectations on the variability of the performance.

>>> metric_fig = results.plot()
>>> metric_fig.show()
../../_images/tutorial-perf-est-guide-with-ref.svg

Insights

After reviewing the performance estimation results, we should be able to see any indications of performance change that NannyML has detected based upon the model’s inputs and outputs alone.

What’s next

The Data Drift functionality can help us to understand whether data drift is causing the performance problem. When the target values become available they can be compared with the estimated results.

You can learn more about the Confidence Based Performance Estimation and its limitations in the How it Works page.