Monitoring Realized Performance for Regression

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> import pandas as pd
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_price_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_price_dataset()[1]
>>> analysis_targets_df = nml.load_synthetic_car_price_dataset()[2]
>>> analysis_df = pd.merge(analysis_df, analysis_targets_df, on='id')

>>> display(reference_df.head(3))

>>> calc = nml.PerformanceCalculator(
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='regression',
...     metrics=['mae', 'mape', 'mse', 'msle', 'rmse', 'rmsle'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> display(results.filter(period='reference').to_df())

>>> figure = results.plot(kind='performance')
>>> figure.show()

Advanced configuration

Walkthrough

For simplicity this guide is based on a synthetic dataset included in the library, where the monitored model predicts the market price of a used car. Check out Car Price Dataset to learn more about this dataset.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

The analysis_targets_df dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation. But as it is required to calculate performance, the first thing to do in this case is to join the analysis target values with the analysis data.

>>> import nannyml as nml
>>> import pandas as pd
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_price_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_price_dataset()[1]
>>> analysis_targets_df = nml.load_synthetic_car_price_dataset()[2]
>>> analysis_df = pd.merge(analysis_df, analysis_targets_df, on='id')

>>> display(reference_df.head(3))

id

car_age

km_driven

price_new

accident_count

door_count

fuel

transmission

y_true

y_pred

timestamp

0

0

15

144020

42810

4

3

diesel

automatic

569

1246

2017-01-24 08:00:00.000

1

1

12

57078

31835

3

3

electric

automatic

4277

4924

2017-01-24 08:00:33.600

2

2

2

76288

31851

3

5

diesel

automatic

7011

5744

2017-01-24 08:01:07.200

Next a PerformanceCalculator is created using a list of metrics to calculate (or just one metric), the data columns required for these metrics, an optional chunking specification and the type of machine learning problem being addressed.

The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:

  • mae - mean absolute error

  • mape - mean absolute percentage error

  • mse - mean squared error

  • rmse - root mean squared error

  • msle - mean squared logarithmic error

  • rmsle - root mean squared logarithmic error

For more information on metrics, check the metrics module.

>>> calc = nml.PerformanceCalculator(
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='regression',
...     metrics=['mae', 'mape', 'mse', 'msle', 'rmse', 'rmsle'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

The new PerformanceCalculator is fitted using the fit() method on the reference data.

The fitted PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the calculate() method. NannyML can output a dataframe that contains all the results of the analysis data.

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

chunk
key
chunk_index
start_index
end_index
start_date
end_date
period
targets_missing_rate
mae
sampling_error
value
upper_threshold
lower_threshold
alert
mape
sampling_error
value
upper_threshold
lower_threshold
alert
mse
sampling_error
value
upper_threshold
lower_threshold
alert
msle
sampling_error
value
upper_threshold
lower_threshold
alert
rmse
sampling_error
value
upper_threshold
lower_threshold
alert
rmsle
sampling_error
value
upper_threshold
lower_threshold
alert

0

[0:5999]

0

0

5999

2017-02-16 16:00:00

2017-02-18 23:59:26.400000

analysis

0

8.21576

853.4

874.805

817.855

False

0.00248466

0.228707

0.237019

0.229456

True

21915

1.14313e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0704883

0.0737091

0.0696521

False

10.348

1069.17

1103.31

1014.28

False

0.002239

0.265496

0.271511

0.263948

False

1

[6000:11999]

1

6000

11999

2017-02-19 00:00:00

2017-02-21 07:59:26.400000

analysis

0

8.21576

853.137

874.805

817.855

False

0.00248466

0.230818

0.237019

0.229456

False

21915

1.13987e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0699896

0.0737091

0.0696521

False

10.348

1067.65

1103.31

1014.28

False

0.002239

0.264556

0.271511

0.263948

False

2

[12000:17999]

2

12000

17999

2017-02-21 08:00:00

2017-02-23 15:59:26.400000

analysis

0

8.21576

846.304

874.805

817.855

False

0.00248466

0.229042

0.237019

0.229456

True

21915

1.12872e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0696923

0.0737091

0.0696521

False

10.348

1062.41

1103.31

1014.28

False

0.002239

0.263993

0.271511

0.263948

False

3

[18000:23999]

3

18000

23999

2017-02-23 16:00:00

2017-02-25 23:59:26.400000

analysis

0

8.21576

855.495

874.805

817.855

False

0.00248466

0.233624

0.237019

0.229456

False

21915

1.15829e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0719322

0.0737091

0.0696521

False

10.348

1076.24

1103.31

1014.28

False

0.002239

0.268202

0.271511

0.263948

False

4

[24000:29999]

4

24000

29999

2017-02-26 00:00:00

2017-02-28 07:59:26.400000

analysis

0

8.21576

849.33

874.805

817.855

False

0.00248466

0.233887

0.237019

0.229456

False

21915

1.12429e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0724877

0.0737091

0.0696521

False

10.348

1060.32

1103.31

1014.28

False

0.002239

0.269235

0.271511

0.263948

False

5

[30000:35999]

5

30000

35999

2017-02-28 08:00:00

2017-03-02 15:59:26.400000

analysis

0

8.21576

702.518

874.805

817.855

True

0.00248466

0.262864

0.237019

0.229456

True

21915

829589

1.21572e+06

1.02681e+06

True

0.0011989

0.104949

0.0737091

0.0696521

True

10.348

910.818

1103.31

1014.28

True

0.002239

0.323958

0.271511

0.263948

True

6

[36000:41999]

6

36000

41999

2017-03-02 16:00:00

2017-03-04 23:59:26.400000

analysis

0

8.21576

700.736

874.805

817.855

True

0.00248466

0.26346

0.237019

0.229456

True

21915

829693

1.21572e+06

1.02681e+06

True

0.0011989

0.104814

0.0737091

0.0696521

True

10.348

910.875

1103.31

1014.28

True

0.002239

0.32375

0.271511

0.263948

True

7

[42000:47999]

7

42000

47999

2017-03-05 00:00:00

2017-03-07 07:59:26.400000

analysis

0

8.21576

684.702

874.805

817.855

True

0.00248466

0.26095

0.237019

0.229456

True

21915

792287

1.21572e+06

1.02681e+06

True

0.0011989

0.104347

0.0737091

0.0696521

True

10.348

890.105

1103.31

1014.28

True

0.002239

0.323027

0.271511

0.263948

True

8

[48000:53999]

8

48000

53999

2017-03-07 08:00:00

2017-03-09 15:59:26.400000

analysis

0

8.21576

705.814

874.805

817.855

True

0.00248466

0.265371

0.237019

0.229456

True

21915

835917

1.21572e+06

1.02681e+06

True

0.0011989

0.104714

0.0737091

0.0696521

True

10.348

914.285

1103.31

1014.28

True

0.002239

0.323596

0.271511

0.263948

True

9

[54000:59999]

9

54000

59999

2017-03-09 16:00:00

2017-03-11 23:59:26.400000

analysis

0

8.21576

698.344

874.805

817.855

True

0.00248466

0.265757

0.237019

0.229456

True

21915

825936

1.21572e+06

1.02681e+06

True

0.0011989

0.105882

0.0737091

0.0696521

True

10.348

908.81

1103.31

1014.28

True

0.002239

0.325394

0.271511

0.263948

True

The results from the reference data are also available.

>>> display(results.filter(period='reference').to_df())

chunk
key
chunk_index
start_index
end_index
start_date
end_date
period
targets_missing_rate
mae
sampling_error
value
upper_threshold
lower_threshold
alert
mape
sampling_error
value
upper_threshold
lower_threshold
alert
mse
sampling_error
value
upper_threshold
lower_threshold
alert
msle
sampling_error
value
upper_threshold
lower_threshold
alert
rmse
sampling_error
value
upper_threshold
lower_threshold
alert
rmsle
sampling_error
value
upper_threshold
lower_threshold
alert

0

[0:5999]

0

0

5999

2017-01-24 08:00:00

2017-01-26 15:59:26.400000

reference

0

8.21576

863.932

874.805

817.855

False

0.00248466

0.23274

0.237019

0.229456

False

21915

1.18007e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0715427

0.0737091

0.0696521

False

10.348

1086.31

1103.31

1014.28

False

0.002239

0.267475

0.271511

0.263948

False

1

[6000:11999]

1

6000

11999

2017-01-26 16:00:00

2017-01-28 23:59:26.400000

reference

0

8.21576

844.491

874.805

817.855

False

0.00248466

0.234282

0.237019

0.229456

False

21915

1.12407e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0721316

0.0737091

0.0696521

False

10.348

1060.22

1103.31

1014.28

False

0.002239

0.268573

0.271511

0.263948

False

2

[12000:17999]

2

12000

17999

2017-01-29 00:00:00

2017-01-31 07:59:26.400000

reference

0

8.21576

830.578

874.805

817.855

False

0.00248466

0.231986

0.237019

0.229456

False

21915

1.07831e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0709387

0.0737091

0.0696521

False

10.348

1038.42

1103.31

1014.28

False

0.002239

0.266343

0.271511

0.263948

False

3

[18000:23999]

3

18000

23999

2017-01-31 08:00:00

2017-02-02 15:59:26.400000

reference

0

8.21576

838.746

874.805

817.855

False

0.00248466

0.231618

0.237019

0.229456

False

21915

1.07827e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0709489

0.0737091

0.0696521

False

10.348

1038.4

1103.31

1014.28

False

0.002239

0.266362

0.271511

0.263948

False

4

[24000:29999]

4

24000

29999

2017-02-02 16:00:00

2017-02-04 23:59:26.400000

reference

0

8.21576

857.765

874.805

817.855

False

0.00248466

0.235091

0.237019

0.229456

False

21915

1.14923e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0727984

0.0737091

0.0696521

False

10.348

1072.02

1103.31

1014.28

False

0.002239

0.269812

0.271511

0.263948

False

5

[30000:35999]

5

30000

35999

2017-02-05 00:00:00

2017-02-07 07:59:26.400000

reference

0

8.21576

852.697

874.805

817.855

False

0.00248466

0.232364

0.237019

0.229456

False

21915

1.15555e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0712554

0.0737091

0.0696521

False

10.348

1074.97

1103.31

1014.28

False

0.002239

0.266937

0.271511

0.263948

False

6

[36000:41999]

6

36000

41999

2017-02-07 08:00:00

2017-02-09 15:59:26.400000

reference

0

8.21576

842.253

874.805

817.855

False

0.00248466

0.232789

0.237019

0.229456

False

21915

1.12037e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0715653

0.0737091

0.0696521

False

10.348

1058.48

1103.31

1014.28

False

0.002239

0.267517

0.271511

0.263948

False

7

[42000:47999]

7

42000

47999

2017-02-09 16:00:00

2017-02-11 23:59:26.400000

reference

0

8.21576

837.9

874.805

817.855

False

0.00248466

0.235516

0.237019

0.229456

False

21915

1.10396e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0729194

0.0737091

0.0696521

False

10.348

1050.7

1103.31

1014.28

False

0.002239

0.270036

0.271511

0.263948

False

8

[48000:53999]

8

48000

53999

2017-02-12 00:00:00

2017-02-14 07:59:26.400000

reference

0

8.21576

844.266

874.805

817.855

False

0.00248466

0.232423

0.237019

0.229456

False

21915

1.09914e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0711648

0.0737091

0.0696521

False

10.348

1048.4

1103.31

1014.28

False

0.002239

0.266767

0.271511

0.263948

False

9

[54000:59999]

9

54000

59999

2017-02-14 08:00:00

2017-02-16 15:59:26.400000

reference

0

8.21576

850.673

874.805

817.855

False

0.00248466

0.233561

0.237019

0.229456

False

21915

1.12369e+06

1.21572e+06

1.02681e+06

False

0.0011989

0.0715405

0.0737091

0.0696521

False

10.348

1060.04

1103.31

1014.28

False

0.002239

0.267471

0.271511

0.263948

False

Apart from chunking and chunk and period-related columns, the results data have a set of columns for each calculated metric.

  • targets_missing_rate - The fraction of missing target data.

  • value - the realized metric value for a specific chunk.

  • sampling_error - the estimate of the Sampling Error.

  • upper_threshold and lower_threshold - crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in the reference partition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated during fit phase. You can also set up custom thresholds using constant or standard deviations thresholds, to learn more about it check out our tutorial on thresholds.

  • alert - flag indicating potentially significant performance change. True if estimated performance crosses upper or lower threshold.

The results can be plotted for visual inspection:

>>> figure = results.plot(kind='performance')
>>> figure.show()
../../_images/tutorial-performance-calculation-regression.svg

Insights

From looking at the RMSE and RMSLE performance results we can observe an interesting effect. We know that RMSE penalizes mispredictions symmetrically while RMSLE penalizes underprediction more than overprediction. Hence while our model has become a little bit more accurate according to RMSE, the increase in RMSLE tells us that our model is now underpredicting more than it was before!

What Next

If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes. We can also plot the realized performance and compare it with the estimated results.

It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.