# Estimating Performance for Multiclass Classification

This tutorial explains how to use NannyML to estimate the performance of multiclass classification models in the absence of target data. To find out how CBPE estimates performance, read the explanation of Confidence-based Performance Estimation.

## Just The Code

```>>> import pandas as pd
>>> import nannyml as nml
>>> from IPython.display import display

>>> estimator = nml.CBPE(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'},
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=6000,
>>> )
>>> estimator.fit(reference_df)

>>> results = estimator.estimate(analysis_df)
>>> display(results.data)

>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', metric=metric)
...     metric_fig.show()

>>> for metric in estimator.metrics:
...     fig2 = results.plot(kind='performance', plot_reference=True, metric=metric)
...     fig2.show()
```

## Walkthrough

For simplicity the guide is based on a synthetic dataset where the monitored model predicts which type of credit card product new customers should be assigned to. You can learn more about this dataset here.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

```>>> import pandas as pd
>>> import nannyml as nml
>>> from IPython.display import display

```

acq_channel

app_behavioral_score

requested_credit_limit

app_channel

credit_bureau_score

stated_income

is_customer

period

identifier

timestamp

y_pred_proba_prepaid_card

y_pred_proba_highstreet_card

y_pred_proba_upmarket_card

y_pred

y_true

0

Partner3

1.80823

350

web

309

15000

True

reference

60000

2020-05-02 02:01:30

0.97

0.03

0

prepaid_card

prepaid_card

1

Partner2

4.38257

500

mobile

418

23000

True

reference

60001

2020-05-02 02:03:33

0.87

0.13

0

prepaid_card

prepaid_card

2

Partner2

-0.787575

400

web

507

24000

False

reference

60002

2020-05-02 02:04:49

0.47

0.35

0.18

prepaid_card

upmarket_card

Next we create the Confidence-based Performance Estimation (`CBPE`) estimator with a list of metrics, and an optional chunking specification. For more information about chunking you can check the setting up page and advanced guide.

The list of metrics specifies which performance metrics of the monitored model will be estimated. The following metrics are currently supported:

• `roc_auc` - one-vs-the-rest, macro-averaged

• `f1` - macro-aveaged

• `precision` - macro-averaged

• `recall` - macro-averaged

• `specificity` - macro-averaged

• `accuracy`

```>>> estimator = nml.CBPE(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'},
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=6000,
>>> )
>>> estimator.fit(reference_df)
```

The `CBPE` estimator is then fitted using the `fit()` method on the `reference` data.

The fitted `estimator` can be used to estimate performance on other data, for which performance cannot be calculated. Typically, this would be used on the latest production data where target is missing. In our example this is the `analysis_df` data.

NannyML can then output a dataframe that contains all the results. Let’s have a look at the results for analysis period only.

```>>> results = estimator.estimate(analysis_df)
>>> display(results.data)
```

key

start_index

end_index

start_date

end_date

realized_roc_auc

estimated_roc_auc

sampling_error_roc_auc

upper_confidence_roc_auc

lower_confidence_roc_auc

upper_threshold_roc_auc

lower_threshold_roc_auc

period

estimated

realized_f1

estimated_f1

sampling_error_f1

upper_confidence_f1

lower_confidence_f1

upper_threshold_f1

lower_threshold_f1

0

[0:5999]

0

5999

2020-09-01 03:10:01

2020-09-13 16:15:10

nan

0.907037

0.00214318

0.913467

0.900607

0.913516

0.900902

False

analysis

True

nan

0.753301

0.00565227

0.770258

0.736345

0.764944

0.741254

False

1

[6000:11999]

6000

11999

2020-09-13 16:15:32

2020-09-25 19:48:42

nan

0.909948

0.00214318

0.916378

0.903519

0.913516

0.900902

False

analysis

True

nan

0.756422

0.00565227

0.773378

0.739465

0.764944

0.741254

False

2

[12000:17999]

12000

17999

2020-09-25 19:50:04

2020-10-08 02:53:47

nan

0.909958

0.00214318

0.916388

0.903529

0.913516

0.900902

False

analysis

True

nan

0.758166

0.00565227

0.775122

0.741209

0.764944

0.741254

False

3

[18000:23999]

18000

23999

2020-10-08 02:57:34

2020-10-20 15:48:19

nan

0.909105

0.00214318

0.915535

0.902676

0.913516

0.900902

False

analysis

True

nan

0.756557

0.00565227

0.773514

0.739601

0.764944

0.741254

False

4

[24000:29999]

24000

29999

2020-10-20 15:49:06

2020-11-01 22:04:40

nan

0.907189

0.00214318

0.913618

0.900759

0.913516

0.900902

False

analysis

True

nan

0.753618

0.00565227

0.770575

0.736661

0.764944

0.741254

False

5

[30000:35999]

30000

35999

2020-11-01 22:04:59

2020-11-14 03:55:33

nan

0.819515

0.00214318

0.825945

0.813086

0.913516

0.900902

True

analysis

True

nan

0.630985

0.00565227

0.647941

0.614028

0.764944

0.741254

True

6

[36000:41999]

36000

41999

2020-11-14 03:55:49

2020-11-26 09:19:06

nan

0.820257

0.00214318

0.826687

0.813828

0.913516

0.900902

True

analysis

True

nan

0.631482

0.00565227

0.648439

0.614525

0.764944

0.741254

True

7

[42000:47999]

42000

47999

2020-11-26 09:19:22

2020-12-08 14:33:56

nan

0.819127

0.00214318

0.825557

0.812698

0.913516

0.900902

True

analysis

True

nan

0.630552

0.00565227

0.647509

0.613595

0.764944

0.741254

True

8

[48000:53999]

48000

53999

2020-12-08 14:34:25

2020-12-20 18:30:30

nan

0.819406

0.00214318

0.825836

0.812977

0.913516

0.900902

True

analysis

True

nan

0.631736

0.00565227

0.648692

0.614779

0.764944

0.741254

True

9

[54000:59999]

54000

59999

2020-12-20 18:31:09

2021-01-01 22:57:55

nan

0.821584

0.00214318

0.828014

0.815154

0.913516

0.900902

True

analysis

True

nan

0.633443

0.00565227

0.6504

0.616487

0.764944

0.741254

True

Apart from chunk-related data, the results data have the following columns for each metric that was estimated:

• `realized_<metric>` - when `target` values are available for a chunk, the realized performance metric will also be calculated and included within the results.

• `estimated_<metric>` - the estimate of a metric for a specific chunk,

• `upper_confidence_<metric>` and `lower_confidence_<metric>` - These values show the Confidence Band of the relevant metric and are equal to estimated value +/- 3 times the estimated Sampling Error.

• `upper_threshold_<metric>` and `lower_threshold_<metric>` - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the `reference` partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during `fit` phase.

• `alert_<metric>` - flag indicating potentially significant performance change. `True` if estimated performance crosses upper or lower threshold.

These results can be also plotted. Our plot contains several key elements.

• The purple dashed step plot shows the estimated performance in each chunk of the analysis period. Thick squared point markers indicate the middle of these chunks.

• The low-saturated purple area around the estimated performance indicates the sampling error.

• The red horizontal dashed lines show upper and lower thresholds for alerting purposes.

• If the estimated performance crosses the upper or lower threshold an alert is raised which is indicated with a red, low-saturated background in the whole width of the relevant chunk. This is additionally indicated by a red, diamond-shaped point marker in the middle of the chunk.

Description of tabular results above explains how the confidence bands and thresholds are calculated. Additional information is shown in the hover (these are interactive plots, though only static views are included here).

```>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', metric=metric)
...     metric_fig.show()
```

To get a better context let’s additionally plot estimation of performance on analysis data together with calculated performance on reference period (where the target was available).

• The right-hand side of the plot shows the estimated performance for the analysis period as before.

• The purple dashed vertical line splits the reference and analysis periods.

• On the left-hand side of the line, the actual model performance (not estimation!) is plotted with a solid light blue line. This facilitates interpretation of the estimation, as it helps to set expectations on the variability of the realised performance.

```>>> for metric in estimator.metrics:
...     fig2 = results.plot(kind='performance', plot_reference=True, metric=metric)
...     fig2.show()
```

## Insights

After reviewing the performance estimation results, we should be able to see any indications of performance change that NannyML has detected based upon the model’s inputs and outputs alone.

## What’s next

The Data Drift functionality can help us to understand whether data drift is causing the performance problem. When the target results become available they can be compared with the estimated results.

You can learn more about the Confidence Based Performance Estimation and its limitations in the How it Works page.