Calculating Confusion Matrix Elements for Multiclass Classification

This tutorial explains how to use NannyML to calculate the confusion matrix for multiclass classification models.

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df, analysis_df, analysis_target_df = nml.load_synthetic_multiclass_classification_dataset()

>>> analysis_df = analysis_df.merge(analysis_target_df, left_index=True, right_index=True)

>>> display(reference_df.head(3))

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'
...     },
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['confusion_matrix'],
...     normalize_confusion_matrix='all',
...     chunk_size=6000)

>>> calc.fit(reference_df)

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> display(results.filter(period='reference').to_df())

>>> figure = results.plot()
>>> figure.show()

Walkthrough

For simplicity this guide is based on a synthetic dataset where the monitored model predicts which type of credit card product new customers should be assigned to. Check out Credit Card Dataset to learn more about this dataset.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

The analysis_targets dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation. But it is required to calculate performance, so the first thing we need to in this case is set up the right data in the right dataframes.

The analysis target values are joined on the analysis frame by their index. Your dataset may already contain the target column, so you may skip this join.

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df, analysis_df, analysis_target_df = nml.load_synthetic_multiclass_classification_dataset()

>>> analysis_df = analysis_df.merge(analysis_target_df, left_index=True, right_index=True)

>>> display(reference_df.head(3))

id

acq_channel

app_behavioral_score

requested_credit_limit

app_channel

credit_bureau_score

stated_income

is_customer

timestamp

y_pred_proba_prepaid_card

y_pred_proba_highstreet_card

y_pred_proba_upmarket_card

y_pred

y_true

0

0

Partner3

1.80823

350

web

309

15000

True

2020-05-02 02:01:30

0.97

0.03

0

prepaid_card

prepaid_card

1

1

Partner2

4.38257

500

mobile

418

23000

True

2020-05-02 02:03:33

0.87

0.13

0

prepaid_card

prepaid_card

2

2

Partner2

-0.787575

400

web

507

24000

False

2020-05-02 02:04:49

0.47

0.35

0.18

prepaid_card

upmarket_card

Next a PerformanceCalculator is created using the following:

  • y_pred_proba: the name of the column in the reference data that contains the predicted probabilities.

  • y_pred: the name of the column in the reference data that contains the predicted classes.

  • y_true: the name of the column in the reference data that contains the true classes.

  • timestamp_column_name (Optional): the name of the column in the reference data that contains timestamps.

  • problem_type: the type of problem being monitored. In this example we will monitor a binary classification problem.

  • metrics: a list of metrics to calculate. In this example we will calculate the confusion_matrix metric.

  • normalize_confusion_matrix (Optional): how to normalize the confusion matrix. The normalization options are:

    • None : returns counts for each cell

    • “true” : normalize over the true class of observations.

    • “pred” : normalize over the predicted class of observations

    • “all” : normalize over all observations

  • chunk_size (Optional): the number of observations in each chunk of data used to calculate performance. For more information about chunking other chunking options check out the chunking tutorial.

  • thresholds (Optional): the thresholds used to calculate the alert flag. For more information about thresholds, check out the thresholds tutorial.

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'
...     },
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['confusion_matrix'],
...     normalize_confusion_matrix='all',
...     chunk_size=6000)

The new PerformanceCalculator is fitted using the fit() method on the reference data.

>>> calc.fit(reference_df)

The fitted PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the calculate() method. NannyML can output a dataframe that contains all the results of the analysis data.

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

chunk
key
chunk_index
start_index
end_index
start_date
end_date
period
targets_missing_rate
true_highstreet_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_highstreet_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_highstreet_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert

0

[0:5999]

0

0

5999

2020-09-01 03:10:01

2020-09-13 16:15:10

analysis

0

0.00555482

0.242833

0.262292

0.228341

False

0.00265159

0.0458333

0.0486211

0.0396456

False

0.00262565

0.0416667

0.0470817

0.039385

False

0.00267108

0.0456667

0.0526699

0.0369635

False

0.00562464

0.26

0.265912

0.243455

False

0.00239047

0.034

0.0421872

0.0289128

False

0.00254241

0.0408333

0.046008

0.0348254

False

0.00248954

0.0408333

0.0441763

0.0331904

False

0.00561357

0.248333

0.270106

0.236227

False

1

[6000:11999]

1

6000

11999

2020-09-13 16:15:32

2020-09-25 19:48:42

analysis

0

0.00555482

0.256

0.262292

0.228341

False

0.00265159

0.0435

0.0486211

0.0396456

False

0.00262565

0.0395

0.0470817

0.039385

False

0.00267108

0.0395

0.0526699

0.0369635

False

0.00562464

0.253

0.265912

0.243455

False

0.00239047

0.0335

0.0421872

0.0289128

False

0.00254241

0.042

0.046008

0.0348254

False

0.00248954

0.039

0.0441763

0.0331904

False

0.00561357

0.254

0.270106

0.236227

False

2

[12000:17999]

2

12000

17999

2020-09-25 19:50:04

2020-10-08 02:53:47

analysis

0

0.00555482

0.241833

0.262292

0.228341

False

0.00265159

0.0416667

0.0486211

0.0396456

False

0.00262565

0.0431667

0.0470817

0.039385

False

0.00267108

0.044

0.0526699

0.0369635

False

0.00562464

0.2595

0.265912

0.243455

False

0.00239047

0.0333333

0.0421872

0.0289128

False

0.00254241

0.0418333

0.046008

0.0348254

False

0.00248954

0.0373333

0.0441763

0.0331904

False

0.00561357

0.257333

0.270106

0.236227

False

3

[18000:23999]

3

18000

23999

2020-10-08 02:57:34

2020-10-20 15:48:19

analysis

0

0.00555482

0.241667

0.262292

0.228341

False

0.00265159

0.0413333

0.0486211

0.0396456

False

0.00262565

0.0418333

0.0470817

0.039385

False

0.00267108

0.0395

0.0526699

0.0369635

False

0.00562464

0.262833

0.265912

0.243455

False

0.00239047

0.041

0.0421872

0.0289128

False

0.00254241

0.0436667

0.046008

0.0348254

False

0.00248954

0.0335

0.0441763

0.0331904

False

0.00561357

0.254667

0.270106

0.236227

False

4

[24000:29999]

4

24000

29999

2020-10-20 15:49:06

2020-11-01 22:04:40

analysis

0

0.00555482

0.248

0.262292

0.228341

False

0.00265159

0.04

0.0486211

0.0396456

False

0.00262565

0.0461667

0.0470817

0.039385

False

0.00267108

0.045

0.0526699

0.0369635

False

0.00562464

0.248833

0.265912

0.243455

False

0.00239047

0.0323333

0.0421872

0.0289128

False

0.00254241

0.0403333

0.046008

0.0348254

False

0.00248954

0.0381667

0.0441763

0.0331904

False

0.00561357

0.261167

0.270106

0.236227

False

5

[30000:35999]

5

30000

35999

2020-11-01 22:04:59

2020-11-14 03:55:33

analysis

0

0.00555482

0.220333

0.262292

0.228341

True

0.00265159

0.0701667

0.0486211

0.0396456

True

0.00262565

0.055

0.0470817

0.039385

True

0.00267108

0.0898333

0.0526699

0.0369635

True

0.00562464

0.146333

0.265912

0.243455

True

0.00239047

0.0746667

0.0421872

0.0289128

True

0.00254241

0.0875

0.046008

0.0348254

True

0.00248954

0.062

0.0441763

0.0331904

True

0.00561357

0.194167

0.270106

0.236227

True

6

[36000:41999]

6

36000

41999

2020-11-14 03:55:49

2020-11-26 09:19:06

analysis

0

0.00555482

0.224333

0.262292

0.228341

True

0.00265159

0.0673333

0.0486211

0.0396456

True

0.00262565

0.053

0.0470817

0.039385

True

0.00267108

0.0991667

0.0526699

0.0369635

True

0.00562464

0.151333

0.265912

0.243455

True

0.00239047

0.0698333

0.0421872

0.0289128

True

0.00254241

0.0846667

0.046008

0.0348254

True

0.00248954

0.0636667

0.0441763

0.0331904

True

0.00561357

0.186667

0.270106

0.236227

True

7

[42000:47999]

7

42000

47999

2020-11-26 09:19:22

2020-12-08 14:33:56

analysis

0

0.00555482

0.232833

0.262292

0.228341

False

0.00265159

0.066

0.0486211

0.0396456

True

0.00262565

0.0503333

0.0470817

0.039385

True

0.00267108

0.0916667

0.0526699

0.0369635

True

0.00562464

0.145833

0.265912

0.243455

True

0.00239047

0.0693333

0.0421872

0.0289128

True

0.00254241

0.0891667

0.046008

0.0348254

True

0.00248954

0.0636667

0.0441763

0.0331904

True

0.00561357

0.191167

0.270106

0.236227

True

8

[48000:53999]

8

48000

53999

2020-12-08 14:34:25

2020-12-20 18:30:30

analysis

0

0.00555482

0.2255

0.262292

0.228341

True

0.00265159

0.0686667

0.0486211

0.0396456

True

0.00262565

0.052

0.0470817

0.039385

True

0.00267108

0.0898333

0.0526699

0.0369635

True

0.00562464

0.148

0.265912

0.243455

True

0.00239047

0.0733333

0.0421872

0.0289128

True

0.00254241

0.0855

0.046008

0.0348254

True

0.00248954

0.0628333

0.0441763

0.0331904

True

0.00561357

0.194333

0.270106

0.236227

True

9

[54000:59999]

9

54000

59999

2020-12-20 18:31:09

2021-01-01 22:57:55

analysis

0

0.00555482

0.225667

0.262292

0.228341

True

0.00265159

0.065

0.0486211

0.0396456

True

0.00262565

0.0543333

0.0470817

0.039385

True

0.00267108

0.0946667

0.0526699

0.0369635

True

0.00562464

0.144833

0.265912

0.243455

True

0.00239047

0.0726667

0.0421872

0.0289128

True

0.00254241

0.0855

0.046008

0.0348254

True

0.00248954

0.0618333

0.0441763

0.0331904

True

0.00561357

0.1955

0.270106

0.236227

True

The results from the reference data are also available.

>>> display(results.filter(period='reference').to_df())

chunk
key
chunk_index
start_index
end_index
start_date
end_date
period
targets_missing_rate
true_highstreet_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_highstreet_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_highstreet_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_prepaid_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_highstreet_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_prepaid_card
sampling_error
value
upper_threshold
lower_threshold
alert
true_upmarket_card_pred_upmarket_card
sampling_error
value
upper_threshold
lower_threshold
alert

0

[0:5999]

0

0

5999

2020-05-02 02:01:30

2020-05-14 12:25:35

reference

0

0.00555482

0.244

0.262292

0.228341

False

0.00265159

0.0453333

0.0486211

0.0396456

False

0.00262565

0.0435

0.0470817

0.039385

False

0.00267108

0.05

0.0526699

0.0369635

False

0.00562464

0.2535

0.265912

0.243455

False

0.00239047

0.0346667

0.0421872

0.0289128

False

0.00254241

0.0366667

0.046008

0.0348254

False

0.00248954

0.0393333

0.0441763

0.0331904

False

0.00561357

0.253

0.270106

0.236227

False

1

[6000:11999]

1

6000

11999

2020-05-14 12:29:25

2020-05-26 18:27:42

reference

0

0.00555482

0.253833

0.262292

0.228341

False

0.00265159

0.0458333

0.0486211

0.0396456

False

0.00262565

0.042

0.0470817

0.039385

False

0.00267108

0.0473333

0.0526699

0.0369635

False

0.00562464

0.25

0.265912

0.243455

False

0.00239047

0.0363333

0.0421872

0.0289128

False

0.00254241

0.0408333

0.046008

0.0348254

False

0.00248954

0.0366667

0.0441763

0.0331904

False

0.00561357

0.247167

0.270106

0.236227

False

2

[12000:17999]

2

12000

17999

2020-05-26 18:31:06

2020-06-07 19:55:45

reference

0

0.00555482

0.246333

0.262292

0.228341

False

0.00265159

0.041

0.0486211

0.0396456

False

0.00262565

0.0428333

0.0470817

0.039385

False

0.00267108

0.0455

0.0526699

0.0369635

False

0.00562464

0.2555

0.265912

0.243455

False

0.00239047

0.0303333

0.0421872

0.0289128

False

0.00254241

0.0423333

0.046008

0.0348254

False

0.00248954

0.0408333

0.0441763

0.0331904

False

0.00561357

0.255333

0.270106

0.236227

False

3

[18000:23999]

3

18000

23999

2020-06-07 19:58:39

2020-06-19 19:42:20

reference

0

0.00555482

0.2455

0.262292

0.228341

False

0.00265159

0.0426667

0.0486211

0.0396456

False

0.00262565

0.0426667

0.0470817

0.039385

False

0.00267108

0.0448333

0.0526699

0.0369635

False

0.00562464

0.258167

0.265912

0.243455

False

0.00239047

0.0345

0.0421872

0.0289128

False

0.00254241

0.0435

0.046008

0.0348254

False

0.00248954

0.0415

0.0441763

0.0331904

False

0.00561357

0.246667

0.270106

0.236227

False

4

[24000:29999]

4

24000

29999

2020-06-19 19:44:14

2020-07-02 01:58:05

reference

0

0.00555482

0.242

0.262292

0.228341

False

0.00265159

0.0435

0.0486211

0.0396456

False

0.00262565

0.0421667

0.0470817

0.039385

False

0.00267108

0.0403333

0.0526699

0.0369635

False

0.00562464

0.256833

0.265912

0.243455

False

0.00239047

0.0346667

0.0421872

0.0289128

False

0.00254241

0.0415

0.046008

0.0348254

False

0.00248954

0.0385

0.0441763

0.0331904

False

0.00561357

0.2605

0.270106

0.236227

False

5

[30000:35999]

5

30000

35999

2020-07-02 02:06:56

2020-07-14 08:14:04

reference

0

0.00555482

0.242833

0.262292

0.228341

False

0.00265159

0.0461667

0.0486211

0.0396456

False

0.00262565

0.044

0.0470817

0.039385

False

0.00267108

0.0455

0.0526699

0.0369635

False

0.00562464

0.253667

0.265912

0.243455

False

0.00239047

0.0353333

0.0421872

0.0289128

False

0.00254241

0.0401667

0.046008

0.0348254

False

0.00248954

0.0401667

0.0441763

0.0331904

False

0.00561357

0.252167

0.270106

0.236227

False

6

[36000:41999]

6

36000

41999

2020-07-14 08:14:08

2020-07-26 12:55:42

reference

0

0.00555482

0.247333

0.262292

0.228341

False

0.00265159

0.0446667

0.0486211

0.0396456

False

0.00262565

0.0463333

0.0470817

0.039385

False

0.00267108

0.0428333

0.0526699

0.0369635

False

0.00562464

0.261167

0.265912

0.243455

False

0.00239047

0.0366667

0.0421872

0.0289128

False

0.00254241

0.0411667

0.046008

0.0348254

False

0.00248954

0.0355

0.0441763

0.0331904

False

0.00561357

0.244333

0.270106

0.236227

False

7

[42000:47999]

7

42000

47999

2020-07-26 12:57:37

2020-08-07 16:32:15

reference

0

0.00555482

0.255667

0.262292

0.228341

False

0.00265159

0.0443333

0.0486211

0.0396456

False

0.00262565

0.0431667

0.0470817

0.039385

False

0.00267108

0.0441667

0.0526699

0.0369635

False

0.00562464

0.2475

0.265912

0.243455

False

0.00239047

0.0373333

0.0421872

0.0289128

False

0.00254241

0.0383333

0.046008

0.0348254

False

0.00248954

0.0368333

0.0441763

0.0331904

False

0.00561357

0.252667

0.270106

0.236227

False

8

[48000:53999]

8

48000

53999

2020-08-07 16:33:44

2020-08-20 00:06:08

reference

0

0.00555482

0.236333

0.262292

0.228341

False

0.00265159

0.0446667

0.0486211

0.0396456

False

0.00262565

0.044

0.0470817

0.039385

False

0.00267108

0.0458333

0.0526699

0.0369635

False

0.00562464

0.253833

0.265912

0.243455

False

0.00239047

0.0391667

0.0421872

0.0289128

False

0.00254241

0.0393333

0.046008

0.0348254

False

0.00248954

0.0393333

0.0441763

0.0331904

False

0.00561357

0.2575

0.270106

0.236227

False

9

[54000:59999]

9

54000

59999

2020-08-20 00:07:58

2020-09-01 03:03:23

reference

0

0.00555482

0.239333

0.262292

0.228341

False

0.00265159

0.0431667

0.0486211

0.0396456

False

0.00262565

0.0416667

0.0470817

0.039385

False

0.00267108

0.0418333

0.0526699

0.0369635

False

0.00562464

0.256667

0.265912

0.243455

False

0.00239047

0.0365

0.0421872

0.0289128

False

0.00254241

0.0403333

0.046008

0.0348254

False

0.00248954

0.0381667

0.0441763

0.0331904

False

0.00561357

0.262333

0.270106

0.236227

False

Apart from chunk and period-related columns, the results data have a set of columns for each calculated metric.

  • targets_missing_rate - The fraction of missing target data.

  • value - the realized metric value for a specific chunk.

  • sampling_error - the estimate of the Sampling Error.

  • upper_threshold and lower_threshold - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during fit phase.

  • alert - flag indicating potentially significant performance change. True if estimated performance crosses upper or lower threshold.

The results can be plotted for visual inspection. Our plot contains several key elements.

  • The blue step plot shows the performance in each chunk of the provided data. Thick squared point markers indicate the middle of these chunks.

  • The gray vertical line splits the reference and analysis data periods.

  • The red horizontal dashed lines show upper and lower thresholds that indicate the range of expected performance values.

  • The red diamond-shaped point markers in the middle of a chunk indicate that an alert has been raised. Alerts are caused by the performance crossing the upper or lower threshold.

>>> figure = results.plot()
>>> figure.show()
../../../_images/tutorial-confusion-matrix-calculation-binary-car-loan-analysis.svg

Additional information such as the chunk index range and chunk date range (if timestamps were provided) is shown in the hover for each chunk (these are interactive plots, though only static views are included here).

Insights

After reviewing the performance calculation results, we should be able to clearly see how the model is performing against the targets, according to whatever metrics we wish to track.

What’s Next

If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes.

It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.