# Estimating Performance for Regression¶

This tutorial explains how to use NannyML to estimate the performance of regression models in the absence of target data. To find out how DLE estimates performance, read the explanation of how Direct Loss Estimation works.

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

## Just The Code¶

```>>> import nannyml as nml
>>> from IPython.display import display

>>> estimator = nml.DLE(
...     feature_column_names=['car_age', 'km_driven', 'price_new', 'accident_count', 'door_count', 'fuel', 'transmission'],
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     metrics=['rmse', 'rmsle'],
...     chunk_size=6000,
...     tune_hyperparameters=False
>>> )

>>> estimator.fit(reference_df)
>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> display(results.filter(period='reference').to_df())

>>> metric_fig = results.plot()
>>> metric_fig.show()
```

## Walkthrough¶

For simplicity this guide is based on a synthetic dataset included in the library, where the monitored model predicts whether the market price of a used car. You can read more about this synthetic dataset here.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

```>>> import nannyml as nml
>>> from IPython.display import display

```

car_age

km_driven

price_new

accident_count

door_count

fuel

transmission

y_true

y_pred

timestamp

0

15

144020

42810

4

3

diesel

automatic

569

1246

2017-01-24 08:00:00.000

1

12

57078

31835

3

3

electric

automatic

4277

4924

2017-01-24 08:00:33.600

2

2

76288

31851

3

5

diesel

automatic

7011

5744

2017-01-24 08:01:07.200

The next step is to instantiate the Direct Error Estimation (`DLE`) estimator. For the instantiation we need to provide:

• The list of column names for the features our model uses.

• The column name for the model output.

• The column name for the model targets.

• The list of regression performance metrics we are interested in estimating. Currently the supported metrics are:

• `mae` - mean absolute error

• `mape` - mean absolute percentage error

• `mse` - mean squared error

• `rmse` - root mean squared error

• `msle` - mean squared logarithmic error

• `rmsle` - root mean squared logarithmic error

• Optionally we can provide a chunking specification, otherwise the NannyML default will be used. For more information about chunking you can check the setting up page and advanced guide.

• Optionally we can provide selected hyperparamters for the model that will make the error estimation. If not, the LGBMRegressor defaults will be used.

• Optionally we can tell the estimator to use FLAML to perform hyperparamter tuning. By default no hyperparamter tuning is performed.

• Optionally we can provide configuration options to perform hyperparamter tuning instead of using the ones set by NannyML.

More information can be found on the API documentation for the `DLE` estimator. During this tutorial the NannyML default settings are used regarding hyperparameter tuning.

```>>> estimator = nml.DLE(
...     feature_column_names=['car_age', 'km_driven', 'price_new', 'accident_count', 'door_count', 'fuel', 'transmission'],
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     metrics=['rmse', 'rmsle'],
...     chunk_size=6000,
...     tune_hyperparameters=False
>>> )
```

The new `DLE` is fitted using the `fit()` method on the `reference` data.

The fitted `estimator` can then be used to calculate estimated performance metrics on all data which has target values available with the `estimate()` method. NannyML can output a dataframe that contains all the results of the analysis data.

```>>> estimator.fit(reference_df)
>>> results = estimator.estimate(analysis_df)
>>> display(results.filter(period='analysis').to_df())
```

(‘chunk’, ‘key’)

(‘chunk’, ‘chunk_index’)

(‘chunk’, ‘start_index’)

(‘chunk’, ‘end_index’)

(‘chunk’, ‘start_date’)

(‘chunk’, ‘end_date’)

(‘chunk’, ‘period’)

(‘rmse’, ‘sampling_error’)

(‘rmse’, ‘realized’)

(‘rmse’, ‘value’)

(‘rmse’, ‘upper_confidence_boundary’)

(‘rmse’, ‘lower_confidence_boundary’)

(‘rmse’, ‘upper_threshold’)

(‘rmse’, ‘lower_threshold’)

(‘rmsle’, ‘sampling_error’)

(‘rmsle’, ‘realized’)

(‘rmsle’, ‘value’)

(‘rmsle’, ‘upper_confidence_boundary’)

(‘rmsle’, ‘lower_confidence_boundary’)

(‘rmsle’, ‘upper_threshold’)

(‘rmsle’, ‘lower_threshold’)

0

[0:5999]

0

0

5999

2017-02-16 16:00:00

2017-02-18 23:59:26.400000

analysis

10.348

nan

1067.42

1098.46

1036.37

1103.31

1014.28

False

0.002239

nan

0.265777

0.272494

0.25906

0.271511

0.263948

False

1

[6000:11999]

1

6000

11999

2017-02-19 00:00:00

2017-02-21 07:59:26.400000

analysis

10.348

nan

1062.74

1093.79

1031.7

1103.31

1014.28

False

0.002239

nan

0.266766

0.273483

0.260049

0.271511

0.263948

False

2

[12000:17999]

2

12000

17999

2017-02-21 08:00:00

2017-02-23 15:59:26.400000

analysis

10.348

nan

1054.53

1085.58

1023.49

1103.31

1014.28

False

0.002239

nan

0.267806

0.274523

0.261089

0.271511

0.263948

False

3

[18000:23999]

3

18000

23999

2017-02-23 16:00:00

2017-02-25 23:59:26.400000

analysis

10.348

nan

1062.54

1093.58

1031.49

1103.31

1014.28

False

0.002239

nan

0.265634

0.272351

0.258917

0.271511

0.263948

False

4

[24000:29999]

4

24000

29999

2017-02-26 00:00:00

2017-02-28 07:59:26.400000

analysis

10.348

nan

1054.79

1085.83

1023.74

1103.31

1014.28

False

0.002239

nan

0.268348

0.275065

0.261631

0.271511

0.263948

False

5

[30000:35999]

5

30000

35999

2017-02-28 08:00:00

2017-03-02 15:59:26.400000

analysis

10.348

nan

930.497

961.541

899.453

1103.31

1014.28

True

0.002239

nan

0.305148

0.311865

0.298431

0.271511

0.263948

True

6

[36000:41999]

6

36000

41999

2017-03-02 16:00:00

2017-03-04 23:59:26.400000

analysis

10.348

nan

930.34

961.384

899.296

1103.31

1014.28

True

0.002239

nan

0.306772

0.313489

0.300055

0.271511

0.263948

True

7

[42000:47999]

7

42000

47999

2017-03-05 00:00:00

2017-03-07 07:59:26.400000

analysis

10.348

nan

928.593

959.637

897.549

1103.31

1014.28

True

0.002239

nan

0.306629

0.313346

0.299912

0.271511

0.263948

True

8

[48000:53999]

8

48000

53999

2017-03-07 08:00:00

2017-03-09 15:59:26.400000

analysis

10.348

nan

932.237

963.281

901.193

1103.31

1014.28

True

0.002239

nan

0.30611

0.312827

0.299393

0.271511

0.263948

True

9

[54000:59999]

9

54000

59999

2017-03-09 16:00:00

2017-03-11 23:59:26.400000

analysis

10.348

nan

921.73

952.774

890.686

1103.31

1014.28

True

0.002239

nan

0.308825

0.315542

0.302108

0.271511

0.263948

True

There results from the reference data are also available.

```>>> display(results.filter(period='reference').to_df())
```

(‘chunk’, ‘key’)

(‘chunk’, ‘chunk_index’)

(‘chunk’, ‘start_index’)

(‘chunk’, ‘end_index’)

(‘chunk’, ‘start_date’)

(‘chunk’, ‘end_date’)

(‘chunk’, ‘period’)

(‘rmse’, ‘sampling_error’)

(‘rmse’, ‘realized’)

(‘rmse’, ‘value’)

(‘rmse’, ‘upper_confidence_boundary’)

(‘rmse’, ‘lower_confidence_boundary’)

(‘rmse’, ‘upper_threshold’)

(‘rmse’, ‘lower_threshold’)

(‘rmsle’, ‘sampling_error’)

(‘rmsle’, ‘realized’)

(‘rmsle’, ‘value’)

(‘rmsle’, ‘upper_confidence_boundary’)

(‘rmsle’, ‘lower_confidence_boundary’)

(‘rmsle’, ‘upper_threshold’)

(‘rmsle’, ‘lower_threshold’)

0

[0:5999]

0

0

5999

2017-01-24 08:00:00

2017-01-26 15:59:26.400000

reference

10.348

1086.31

1073.4

1104.44

1042.35

1103.31

1014.28

False

0.002239

0.267475

0.266626

0.273343

0.259909

0.271511

0.263948

False

1

[6000:11999]

1

6000

11999

2017-01-26 16:00:00

2017-01-28 23:59:26.400000

reference

10.348

1060.22

1056.19

1087.24

1025.15

1103.31

1014.28

False

0.002239

0.268573

0.268918

0.275635

0.262201

0.271511

0.263948

False

2

[12000:17999]

2

12000

17999

2017-01-29 00:00:00

2017-01-31 07:59:26.400000

reference

10.348

1038.42

1054.96

1086.01

1023.92

1103.31

1014.28

False

0.002239

0.266343

0.268806

0.275523

0.262089

0.271511

0.263948

False

3

[18000:23999]

3

18000

23999

2017-01-31 08:00:00

2017-02-02 15:59:26.400000

reference

10.348

1038.4

1055.05

1086.09

1024

1103.31

1014.28

False

0.002239

0.266362

0.267155

0.273872

0.260438

0.271511

0.263948

False

4

[24000:29999]

4

24000

29999

2017-02-02 16:00:00

2017-02-04 23:59:26.400000

reference

10.348

1072.02

1066.57

1097.61

1035.52

1103.31

1014.28

False

0.002239

0.269812

0.26687

0.273587

0.260153

0.271511

0.263948

False

5

[30000:35999]

5

30000

35999

2017-02-05 00:00:00

2017-02-07 07:59:26.400000

reference

10.348

1074.97

1064.57

1095.61

1033.52

1103.31

1014.28

False

0.002239

0.266937

0.266295

0.273012

0.259578

0.271511

0.263948

False

6

[36000:41999]

6

36000

41999

2017-02-07 08:00:00

2017-02-09 15:59:26.400000

reference

10.348

1058.48

1058.17

1089.21

1027.12

1103.31

1014.28

False

0.002239

0.267517

0.267456

0.274173

0.260739

0.271511

0.263948

False

7

[42000:47999]

7

42000

47999

2017-02-09 16:00:00

2017-02-11 23:59:26.400000

reference

10.348

1050.7

1054.74

1085.78

1023.69

1103.31

1014.28

False

0.002239

0.270036

0.268755

0.275472

0.262038

0.271511

0.263948

False

8

[48000:53999]

8

48000

53999

2017-02-12 00:00:00

2017-02-14 07:59:26.400000

reference

10.348

1048.4

1051.64

1082.68

1020.59

1103.31

1014.28

False

0.002239

0.266767

0.267832

0.274549

0.261115

0.271511

0.263948

False

9

[54000:59999]

9

54000

59999

2017-02-14 08:00:00

2017-02-16 15:59:26.400000

reference

10.348

1060.04

1053.51

1084.55

1022.47

1103.31

1014.28

False

0.002239

0.267471

0.268593

0.27531

0.261876

0.271511

0.263948

False

Apart from chunk-related data, the results data have the following columns for each metric that was estimated:

• `realized_<metric>` - when `target` values are available for a chunk, the realized performance metric will also be calculated and included within the results.

• `estimated_<metric>` - the estimate of a metric for a specific chunk,

• `upper_confidence_<metric>` and `lower_confidence_<metric>` - these correspond to the sampling error and are equal to +/- 3 times the calculated standard error.

• `<metric>_lower_threshold>` and `<metric>_upper_threshold>` - Lower and upper thresholds for performance metric. Crossing them will raise an alert that there is a significant metric change. The thresholds are calculated based on the realized performance of chunks in the `reference` period. The thresholds are 3 standard deviations away from the mean performance calculated on `reference` chunks. They are calculated during `fit` phase.

• `alert_<metric>` - flag indicating potentially significant performance change. `True` if estimated performance crosses upper or lower threshold.

These results can be also plotted. Our plot contains several key elements.

• The purple dashed step plot shows the estimated performance in each chunk of the analysis period. Thick squared point markers indicate the middle of these chunks.

• The low-saturated purple area around the estimated performance indicates the sampling error.

• The red horizontal dashed lines show upper and lower thresholds for alerting purposes.

• If the estimated performance crosses the upper or lower threshold an alert is raised which is indicated with a red, low-saturated background in the whole width of the relevant chunk. This is additionally indicated by a red, diamond-shaped point marker in the middle of the chunk.

Additional information is shown in the hover (these are interactive plots, though only static views are included here). The plots can be created with the following code:

```>>> metric_fig = results.plot()
>>> metric_fig.show()
```

## Insights¶

From looking at the RMSE and RMSLE performance results we can observe an interesting effect. We know that RMSE penalizes mispredictions symmetrically while RMSLE penalizes underprediction more than overprediction. Hence performance estimator tells us that while our model will become a little bit more accurate according to RMSE, the increase in RMSLE suggests us that our model will be underpredicting more than it was before!

## What’s next¶

The Data Drift functionality can help us to understand whether data drift is causing the performance problem. When the target values become available they can be compared with the estimated results.

You can learn more about Direct Error Estimation and its limitations in the How it Works page.