Monitoring Realized Performance for Multiclass Classification

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_multiclass_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_multiclass_classification_dataset()[1]
>>> analysis_target_df = nml.load_synthetic_multiclass_classification_dataset()[2]
>>> analysis_df = analysis_df.merge(analysis_target_df, on='identifier')

>>> display(reference_df.head(3))

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'
...     },
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

>>> results = calc.calculate(analysis_df)
>>> display(results.data)

>>> display(results.calculator.previous_reference_results)

>>> for metric in calc.metrics:
...     figure = results.plot(kind='performance', plot_reference=True, metric=metric)
...     figure.show()

Walkthrough

For simplicity the guide is based on a synthetic dataset where the monitored model predicts which type of credit card product new customers should be assigned to. You can learn more about this dataset.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods

The analysis_targets dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation.. But it is required to calculate performance, so the first thing we need to in this case is set up the right data in the right dataframes. The analysis target values are joined on the analysis frame by the identifier column.

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_multiclass_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_multiclass_classification_dataset()[1]
>>> analysis_target_df = nml.load_synthetic_multiclass_classification_dataset()[2]
>>> analysis_df = analysis_df.merge(analysis_target_df, on='identifier')

>>> display(reference_df.head(3))

acq_channel

app_behavioral_score

requested_credit_limit

app_channel

credit_bureau_score

stated_income

is_customer

period

identifier

timestamp

y_pred_proba_prepaid_card

y_pred_proba_highstreet_card

y_pred_proba_upmarket_card

y_pred

y_true

0

Partner3

1.80823

350

web

309

15000

True

reference

60000

2020-05-02 02:01:30

0.97

0.03

0

prepaid_card

prepaid_card

1

Partner2

4.38257

500

mobile

418

23000

True

reference

60001

2020-05-02 02:03:33

0.87

0.13

0

prepaid_card

prepaid_card

2

Partner2

-0.787575

400

web

507

24000

False

reference

60002

2020-05-02 02:04:49

0.47

0.35

0.18

prepaid_card

upmarket_card

Next a PerformanceCalculator is created using a list of metrics to calculate (or just one metric), the data columns required for these metrics, and an optional chunking specification.

The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:

  • roc_auc - one-vs-the-rest, macro-averaged

  • f1 - macro-averaged

  • precision - macro-averaged

  • recall - macro-averaged

  • specificity - macro-averaged

  • accuracy

For more information on metrics, check the metrics module.

>>> calc = nml.PerformanceCalculator(
...     y_pred_proba={
...         'prepaid_card': 'y_pred_proba_prepaid_card',
...         'highstreet_card': 'y_pred_proba_highstreet_card',
...         'upmarket_card': 'y_pred_proba_upmarket_card'
...     },
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='classification_multiclass',
...     metrics=['roc_auc', 'f1', 'precision', 'recall', 'specificity', 'accuracy'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

The new PerformanceCalculator is fitted using the fit() method on the reference data.

The fitted PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the calculate() method. NannyML can output a dataframe that contains all the results of the analysis data.

>>> results = calc.calculate(analysis_df)
>>> display(results.data)

key

start_index

end_index

start_date

end_date

period

targets_missing_rate

roc_auc

roc_auc_lower_threshold

roc_auc_upper_threshold

roc_auc_sampling_error

roc_auc_alert

f1

f1_lower_threshold

f1_upper_threshold

f1_sampling_error

f1_alert

precision

precision_lower_threshold

precision_upper_threshold

precision_sampling_error

precision_alert

recall

recall_lower_threshold

recall_upper_threshold

recall_sampling_error

recall_alert

specificity

specificity_lower_threshold

specificity_upper_threshold

specificity_sampling_error

specificity_alert

accuracy

accuracy_lower_threshold

accuracy_upper_threshold

accuracy_sampling_error

accuracy_alert

0

[0:5999]

0

5999

2020-09-01 03:10:01

2020-09-13 16:15:10

0

0.907595

0.900902

0.913516

0.00214318

False

0.751103

0.741254

0.764944

0.00565227

False

0.75127

0.741261

0.764978

0.00556588

False

0.751033

0.741261

0.765026

0.00556473

False

0.87555

0.870566

0.882586

0.0030024

False

0.751167

0.741231

0.765102

0.00556637

False

1

[6000:11999]

6000

11999

2020-09-13 16:15:32

2020-09-25 19:48:42

0

0.910534

0.900902

0.913516

0.00214318

False

0.763045

0.741254

0.764944

0.00565227

False

0.763125

0.741261

0.764978

0.00556588

False

0.763148

0.741261

0.765026

0.00556473

False

0.881508

0.870566

0.882586

0.0030024

False

0.763

0.741231

0.765102

0.00556637

False

2

[12000:17999]

12000

17999

2020-09-25 19:50:04

2020-10-08 02:53:47

0

0.909414

0.900902

0.913516

0.00214318

False

0.758487

0.741254

0.764944

0.00565227

False

0.758503

0.741261

0.764978

0.00556588

False

0.758484

0.741261

0.765026

0.00556473

False

0.879367

0.870566

0.882586

0.0030024

False

0.758667

0.741231

0.765102

0.00556637

False

3

[18000:23999]

18000

23999

2020-10-08 02:57:34

2020-10-20 15:48:19

0

0.911577

0.900902

0.913516

0.00214318

False

0.758944

0.741254

0.764944

0.00565227

False

0.758973

0.741261

0.764978

0.00556588

False

0.758986

0.741261

0.765026

0.00556473

False

0.87963

0.870566

0.882586

0.0030024

False

0.759167

0.741231

0.765102

0.00556637

False

4

[24000:29999]

24000

29999

2020-10-20 15:49:06

2020-11-01 22:04:40

0

0.907533

0.900902

0.913516

0.00214318

False

0.757964

0.741254

0.764944

0.00565227

False

0.75795

0.741261

0.764978

0.00556588

False

0.757979

0.741261

0.765026

0.00556473

False

0.878986

0.870566

0.882586

0.0030024

False

0.758

0.741231

0.765102

0.00556637

False

5

[30000:35999]

30000

35999

2020-11-01 22:04:59

2020-11-14 03:55:33

0

0.748588

0.900902

0.913516

0.00214318

True

0.557112

0.741254

0.764944

0.00565227

True

0.559696

0.741261

0.764978

0.00556588

True

0.557829

0.741261

0.765026

0.00556473

True

0.779905

0.870566

0.882586

0.0030024

True

0.560833

0.741231

0.765102

0.00556637

True

6

[36000:41999]

36000

41999

2020-11-14 03:55:49

2020-11-26 09:19:06

0

0.751137

0.900902

0.913516

0.00214318

True

0.559148

0.741254

0.764944

0.00565227

True

0.562915

0.741261

0.764978

0.00556588

True

0.56017

0.741261

0.765026

0.00556473

True

0.780676

0.870566

0.882586

0.0030024

True

0.562333

0.741231

0.765102

0.00556637

True

7

[42000:47999]

42000

47999

2020-11-26 09:19:22

2020-12-08 14:33:56

0

0.756399

0.900902

0.913516

0.00214318

True

0.565055

0.741254

0.764944

0.00565227

True

0.569069

0.741261

0.764978

0.00556588

True

0.565943

0.741261

0.765026

0.00556473

True

0.784223

0.870566

0.882586

0.0030024

True

0.569833

0.741231

0.765102

0.00556637

True

8

[48000:53999]

48000

53999

2020-12-08 14:34:25

2020-12-20 18:30:30

0

0.758561

0.900902

0.913516

0.00214318

True

0.563897

0.741254

0.764944

0.00565227

True

0.566673

0.741261

0.764978

0.00556588

True

0.564723

0.741261

0.765026

0.00556473

True

0.783422

0.870566

0.882586

0.0030024

True

0.567833

0.741231

0.765102

0.00556637

True

9

[54000:59999]

54000

59999

2020-12-20 18:31:09

2021-01-01 22:57:55

0

0.753937

0.900902

0.913516

0.00214318

True

0.561644

0.741254

0.764944

0.00565227

True

0.565129

0.741261

0.764978

0.00556588

True

0.562772

0.741261

0.765026

0.00556473

True

0.782429

0.870566

0.882586

0.0030024

True

0.566

0.741231

0.765102

0.00556637

True

There results from the reference data are also available.

>>> display(results.calculator.previous_reference_results)

key

start_index

end_index

start_date

end_date

period

targets_missing_rate

roc_auc

roc_auc_lower_threshold

roc_auc_upper_threshold

roc_auc_sampling_error

roc_auc_alert

f1

f1_lower_threshold

f1_upper_threshold

f1_sampling_error

f1_alert

precision

precision_lower_threshold

precision_upper_threshold

precision_sampling_error

precision_alert

recall

recall_lower_threshold

recall_upper_threshold

recall_sampling_error

recall_alert

specificity

specificity_lower_threshold

specificity_upper_threshold

specificity_sampling_error

specificity_alert

accuracy

accuracy_lower_threshold

accuracy_upper_threshold

accuracy_sampling_error

accuracy_alert

0

[0:5999]

0

5999

2020-05-02 02:01:30

2020-05-14 12:25:35

0

0.90476

0.900902

0.913516

0.00214318

False

0.750532

0.741254

0.764944

0.00565227

False

0.7505

0.741261

0.764978

0.00556588

False

0.750576

0.741261

0.765026

0.00556473

False

0.875226

0.870566

0.882586

0.0030024

False

0.7505

0.741231

0.765102

0.00556637

False

1

[6000:11999]

6000

11999

2020-05-14 12:29:25

2020-05-26 18:27:42

0

0.905917

0.900902

0.913516

0.00214318

False

0.751148

0.741254

0.764944

0.00565227

False

0.751142

0.741261

0.764978

0.00556588

False

0.751157

0.741261

0.765026

0.00556473

False

0.875424

0.870566

0.882586

0.0030024

False

0.751

0.741231

0.765102

0.00556637

False

2

[12000:17999]

12000

17999

2020-05-26 18:31:06

2020-06-07 19:55:45

0

0.909329

0.900902

0.913516

0.00214318

False

0.75714

0.741254

0.764944

0.00565227

False

0.75728

0.741261

0.764978

0.00556588

False

0.757174

0.741261

0.765026

0.00556473

False

0.878628

0.870566

0.882586

0.0030024

False

0.757167

0.741231

0.765102

0.00556637

False

3

[18000:23999]

18000

23999

2020-06-07 19:58:39

2020-06-19 19:42:20

0

0.906731

0.900902

0.913516

0.00214318

False

0.750274

0.741254

0.764944

0.00565227

False

0.750415

0.741261

0.764978

0.00556588

False

0.750241

0.741261

0.765026

0.00556473

False

0.875163

0.870566

0.882586

0.0030024

False

0.750333

0.741231

0.765102

0.00556637

False

4

[24000:29999]

24000

29999

2020-06-19 19:44:14

2020-07-02 01:58:05

0

0.910577

0.900902

0.913516

0.00214318

False

0.759144

0.741254

0.764944

0.00565227

False

0.759175

0.741261

0.764978

0.00556588

False

0.759197

0.741261

0.765026

0.00556473

False

0.879686

0.870566

0.882586

0.0030024

False

0.759333

0.741231

0.765102

0.00556637

False

5

[30000:35999]

30000

35999

2020-07-02 02:06:56

2020-07-14 08:14:04

0

0.904577

0.900902

0.913516

0.00214318

False

0.74863

0.741254

0.764944

0.00565227

False

0.74866

0.741261

0.764978

0.00556588

False

0.748657

0.741261

0.765026

0.00556473

False

0.874329

0.870566

0.882586

0.0030024

False

0.748667

0.741231

0.765102

0.00556637

False

6

[36000:41999]

36000

41999

2020-07-14 08:14:08

2020-07-26 12:55:42

0

0.906673

0.900902

0.913516

0.00214318

False

0.752763

0.741254

0.764944

0.00565227

False

0.752684

0.741261

0.764978

0.00556588

False

0.752944

0.741261

0.765026

0.00556473

False

0.876407

0.870566

0.882586

0.0030024

False

0.752833

0.741231

0.765102

0.00556637

False

7

[42000:47999]

42000

47999

2020-07-26 12:57:37

2020-08-07 16:32:15

0

0.908703

0.900902

0.913516

0.00214318

False

0.755883

0.741254

0.764944

0.00565227

False

0.75582

0.741261

0.764978

0.00556588

False

0.756006

0.741261

0.765026

0.00556473

False

0.877891

0.870566

0.882586

0.0030024

False

0.755833

0.741231

0.765102

0.00556637

False

8

[48000:53999]

48000

53999

2020-08-07 16:33:44

2020-08-20 00:06:08

0

0.905072

0.900902

0.913516

0.00214318

False

0.74742

0.741254

0.764944

0.00565227

False

0.747441

0.741261

0.764978

0.00556588

False

0.747436

0.741261

0.765026

0.00556473

False

0.873832

0.870566

0.882586

0.0030024

False

0.747667

0.741231

0.765102

0.00556637

False

9

[54000:59999]

54000

59999

2020-08-20 00:07:58

2020-09-01 03:03:23

0

0.909749

0.900902

0.913516

0.00214318

False

0.758055

0.741254

0.764944

0.00565227

False

0.758077

0.741261

0.764978

0.00556588

False

0.758052

0.741261

0.765026

0.00556473

False

0.879177

0.870566

0.882586

0.0030024

False

0.758333

0.741231

0.765102

0.00556637

False

Apart from chunking and chunk and period-related columns, the results data have the a set of columns for each calculated metric. When taking roc_auc as an example:

  • targets_missing_rate - The fraction of missing target data.

  • <metric> - The value of the metric for a specific chunk.

  • <metric>_lower_threshold> and <metric>_upper_threshold> - Lower and upper thresholds for performance metric. Crossing them will raise an alert that there is a significant metric change. The thresholds are calculated based on the realized performance of chunks in the reference period. The thresholds are 3 standard deviations away from the mean performance calculated on reference chunks. They are calculated during fit phase.

  • <metric>_alert - A flag indicating potentially significant performance change. True if realized performance crosses upper or lower threshold.

  • <metric>_sampling_error - Estimated Sampling Error for the relevant metric.

The results can be plotted for visual inspection:

>>> for metric in calc.metrics:
...     figure = results.plot(kind='performance', plot_reference=True, metric=metric)
...     figure.show()
../../_images/tutorial-performance-calculation-multiclass-f1.svg../../_images/tutorial-performance-calculation-multiclass-roc_auc.svg

Insights

After reviewing the performance calculation results, we should be able to clearly see how the model is performing against the targets, according to whatever metrics we wish to track.

What Next

If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes.

It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.