Monitoring Realized Performance for Regression

Note

The following example uses timestamps. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements.

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_price_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_price_dataset()[1]
>>> analysis_target_df = nml.load_synthetic_car_price_dataset()[2]
>>> analysis_df = analysis_df.join(analysis_target_df)

>>> display(reference_df.head(3))

>>> calc = nml.PerformanceCalculator(
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='regression',
...     metrics=['mae', 'mape', 'mse', 'msle', 'rmse', 'rmsle'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

>>> display(results.filter(period='reference').to_df())

>>> figure = results.plot(kind='performance')
>>> figure.show()

Walkthrough

For simplicity the guide is based on a synthetic dataset where the monitored model predicts the selling price of a used car. You can learn more about this dataset.

In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods.

The analysis_targets dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation. But as it is required to calculate performance, the first thing to do in this case is to join the analysis target values with the analysis data.

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference_df = nml.load_synthetic_car_price_dataset()[0]
>>> analysis_df = nml.load_synthetic_car_price_dataset()[1]
>>> analysis_target_df = nml.load_synthetic_car_price_dataset()[2]
>>> analysis_df = analysis_df.join(analysis_target_df)

>>> display(reference_df.head(3))

	car_age	km_driven	price_new	accident_count	door_count	fuel	transmission	y_true	y_pred	timestamp
0	15	144020	42810	4	3	diesel	automatic	569	1246	2017-01-24 08:00:00.000
1	12	57078	31835	3	3	electric	automatic	4277	4924	2017-01-24 08:00:33.600
2	2	76288	31851	3	5	diesel	automatic	7011	5744	2017-01-24 08:01:07.200

Next a PerformanceCalculator is created using a list of metrics to calculate (or just one metric), the data columns required for these metrics, an optional chunking specification and the type of machine learning problem being addressed.

The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:

• mae - mean absolute error

• mape - mean absolute percentage error

• mse - mean squared error

• rmse - root mean squared error

• msle - mean squared logarithmic error

• rmsle - root mean squared logarithmic error

For more information on metrics, check the metrics module.

>>> calc = nml.PerformanceCalculator(
...     y_pred='y_pred',
...     y_true='y_true',
...     timestamp_column_name='timestamp',
...     problem_type='regression',
...     metrics=['mae', 'mape', 'mse', 'msle', 'rmse', 'rmsle'],
...     chunk_size=6000)

>>> calc.fit(reference_df)

The new PerformanceCalculator is fitted using the fit() method on the reference data.

The fitted PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the calculate() method. NannyML can output a dataframe that contains all the results of the analysis data.

>>> results = calc.calculate(analysis_df)
>>> display(results.filter(period='analysis').to_df())

	(‘chunk’, ‘key’)	(‘chunk’, ‘chunk_index’)	(‘chunk’, ‘start_index’)	(‘chunk’, ‘end_index’)	(‘chunk’, ‘start_date’)	(‘chunk’, ‘end_date’)	(‘chunk’, ‘period’)	(‘chunk’, ‘targets_missing_rate’)	(‘mae’, ‘sampling_error’)	(‘mae’, ‘value’)	(‘mae’, ‘upper_threshold’)	(‘mae’, ‘lower_threshold’)	(‘mae’, ‘alert’)	(‘mape’, ‘sampling_error’)	(‘mape’, ‘value’)	(‘mape’, ‘upper_threshold’)	(‘mape’, ‘lower_threshold’)	(‘mape’, ‘alert’)	(‘mse’, ‘sampling_error’)	(‘mse’, ‘value’)	(‘mse’, ‘upper_threshold’)	(‘mse’, ‘lower_threshold’)	(‘mse’, ‘alert’)	(‘msle’, ‘sampling_error’)	(‘msle’, ‘value’)	(‘msle’, ‘upper_threshold’)	(‘msle’, ‘lower_threshold’)	(‘msle’, ‘alert’)	(‘rmse’, ‘sampling_error’)	(‘rmse’, ‘value’)	(‘rmse’, ‘upper_threshold’)	(‘rmse’, ‘lower_threshold’)	(‘rmse’, ‘alert’)	(‘rmsle’, ‘sampling_error’)	(‘rmsle’, ‘value’)	(‘rmsle’, ‘upper_threshold’)	(‘rmsle’, ‘lower_threshold’)	(‘rmsle’, ‘alert’)
0	[0:5999]	0	0	5999	2017-02-16 16:00:00	2017-02-18 23:59:26.400000	analysis	0	8.21576	853.4	874.805	817.855	False	0.00248466	0.228707	0.237019	0.229456	True	21915	1.14313e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0704883	0.0737091	0.0696521	False	10.348	1069.17	1103.31	1014.28	False	0.002239	0.265496	0.271511	0.263948	False
1	[6000:11999]	1	6000	11999	2017-02-19 00:00:00	2017-02-21 07:59:26.400000	analysis	0	8.21576	853.137	874.805	817.855	False	0.00248466	0.230818	0.237019	0.229456	False	21915	1.13987e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0699896	0.0737091	0.0696521	False	10.348	1067.65	1103.31	1014.28	False	0.002239	0.264556	0.271511	0.263948	False
2	[12000:17999]	2	12000	17999	2017-02-21 08:00:00	2017-02-23 15:59:26.400000	analysis	0	8.21576	846.304	874.805	817.855	False	0.00248466	0.229042	0.237019	0.229456	True	21915	1.12872e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0696923	0.0737091	0.0696521	False	10.348	1062.41	1103.31	1014.28	False	0.002239	0.263993	0.271511	0.263948	False
3	[18000:23999]	3	18000	23999	2017-02-23 16:00:00	2017-02-25 23:59:26.400000	analysis	0	8.21576	855.495	874.805	817.855	False	0.00248466	0.233624	0.237019	0.229456	False	21915	1.15829e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0719322	0.0737091	0.0696521	False	10.348	1076.24	1103.31	1014.28	False	0.002239	0.268202	0.271511	0.263948	False
4	[24000:29999]	4	24000	29999	2017-02-26 00:00:00	2017-02-28 07:59:26.400000	analysis	0	8.21576	849.33	874.805	817.855	False	0.00248466	0.233887	0.237019	0.229456	False	21915	1.12429e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0724877	0.0737091	0.0696521	False	10.348	1060.32	1103.31	1014.28	False	0.002239	0.269235	0.271511	0.263948	False
5	[30000:35999]	5	30000	35999	2017-02-28 08:00:00	2017-03-02 15:59:26.400000	analysis	0	8.21576	702.518	874.805	817.855	True	0.00248466	0.262864	0.237019	0.229456	True	21915	829589	1.21572e+06	1.02681e+06	True	0.0011989	0.104949	0.0737091	0.0696521	True	10.348	910.818	1103.31	1014.28	True	0.002239	0.323958	0.271511	0.263948	True
6	[36000:41999]	6	36000	41999	2017-03-02 16:00:00	2017-03-04 23:59:26.400000	analysis	0	8.21576	700.736	874.805	817.855	True	0.00248466	0.26346	0.237019	0.229456	True	21915	829693	1.21572e+06	1.02681e+06	True	0.0011989	0.104814	0.0737091	0.0696521	True	10.348	910.875	1103.31	1014.28	True	0.002239	0.32375	0.271511	0.263948	True
7	[42000:47999]	7	42000	47999	2017-03-05 00:00:00	2017-03-07 07:59:26.400000	analysis	0	8.21576	684.702	874.805	817.855	True	0.00248466	0.26095	0.237019	0.229456	True	21915	792287	1.21572e+06	1.02681e+06	True	0.0011989	0.104347	0.0737091	0.0696521	True	10.348	890.105	1103.31	1014.28	True	0.002239	0.323027	0.271511	0.263948	True
8	[48000:53999]	8	48000	53999	2017-03-07 08:00:00	2017-03-09 15:59:26.400000	analysis	0	8.21576	705.814	874.805	817.855	True	0.00248466	0.265371	0.237019	0.229456	True	21915	835917	1.21572e+06	1.02681e+06	True	0.0011989	0.104714	0.0737091	0.0696521	True	10.348	914.285	1103.31	1014.28	True	0.002239	0.323596	0.271511	0.263948	True
9	[54000:59999]	9	54000	59999	2017-03-09 16:00:00	2017-03-11 23:59:26.400000	analysis	0	8.21576	698.344	874.805	817.855	True	0.00248466	0.265757	0.237019	0.229456	True	21915	825936	1.21572e+06	1.02681e+06	True	0.0011989	0.105882	0.0737091	0.0696521	True	10.348	908.81	1103.31	1014.28	True	0.002239	0.325394	0.271511	0.263948	True

There results from the reference data are also available.

>>> display(results.filter(period='reference').to_df())

	(‘chunk’, ‘key’)	(‘chunk’, ‘chunk_index’)	(‘chunk’, ‘start_index’)	(‘chunk’, ‘end_index’)	(‘chunk’, ‘start_date’)	(‘chunk’, ‘end_date’)	(‘chunk’, ‘period’)	(‘chunk’, ‘targets_missing_rate’)	(‘mae’, ‘sampling_error’)	(‘mae’, ‘value’)	(‘mae’, ‘upper_threshold’)	(‘mae’, ‘lower_threshold’)	(‘mae’, ‘alert’)	(‘mape’, ‘sampling_error’)	(‘mape’, ‘value’)	(‘mape’, ‘upper_threshold’)	(‘mape’, ‘lower_threshold’)	(‘mape’, ‘alert’)	(‘mse’, ‘sampling_error’)	(‘mse’, ‘value’)	(‘mse’, ‘upper_threshold’)	(‘mse’, ‘lower_threshold’)	(‘mse’, ‘alert’)	(‘msle’, ‘sampling_error’)	(‘msle’, ‘value’)	(‘msle’, ‘upper_threshold’)	(‘msle’, ‘lower_threshold’)	(‘msle’, ‘alert’)	(‘rmse’, ‘sampling_error’)	(‘rmse’, ‘value’)	(‘rmse’, ‘upper_threshold’)	(‘rmse’, ‘lower_threshold’)	(‘rmse’, ‘alert’)	(‘rmsle’, ‘sampling_error’)	(‘rmsle’, ‘value’)	(‘rmsle’, ‘upper_threshold’)	(‘rmsle’, ‘lower_threshold’)	(‘rmsle’, ‘alert’)
0	[0:5999]	0	0	5999	2017-01-24 08:00:00	2017-01-26 15:59:26.400000	reference	0	8.21576	863.932	874.805	817.855	False	0.00248466	0.23274	0.237019	0.229456	False	21915	1.18007e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0715427	0.0737091	0.0696521	False	10.348	1086.31	1103.31	1014.28	False	0.002239	0.267475	0.271511	0.263948	False
1	[6000:11999]	1	6000	11999	2017-01-26 16:00:00	2017-01-28 23:59:26.400000	reference	0	8.21576	844.491	874.805	817.855	False	0.00248466	0.234282	0.237019	0.229456	False	21915	1.12407e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0721316	0.0737091	0.0696521	False	10.348	1060.22	1103.31	1014.28	False	0.002239	0.268573	0.271511	0.263948	False
2	[12000:17999]	2	12000	17999	2017-01-29 00:00:00	2017-01-31 07:59:26.400000	reference	0	8.21576	830.578	874.805	817.855	False	0.00248466	0.231986	0.237019	0.229456	False	21915	1.07831e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0709387	0.0737091	0.0696521	False	10.348	1038.42	1103.31	1014.28	False	0.002239	0.266343	0.271511	0.263948	False
3	[18000:23999]	3	18000	23999	2017-01-31 08:00:00	2017-02-02 15:59:26.400000	reference	0	8.21576	838.746	874.805	817.855	False	0.00248466	0.231618	0.237019	0.229456	False	21915	1.07827e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0709489	0.0737091	0.0696521	False	10.348	1038.4	1103.31	1014.28	False	0.002239	0.266362	0.271511	0.263948	False
4	[24000:29999]	4	24000	29999	2017-02-02 16:00:00	2017-02-04 23:59:26.400000	reference	0	8.21576	857.765	874.805	817.855	False	0.00248466	0.235091	0.237019	0.229456	False	21915	1.14923e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0727984	0.0737091	0.0696521	False	10.348	1072.02	1103.31	1014.28	False	0.002239	0.269812	0.271511	0.263948	False
5	[30000:35999]	5	30000	35999	2017-02-05 00:00:00	2017-02-07 07:59:26.400000	reference	0	8.21576	852.697	874.805	817.855	False	0.00248466	0.232364	0.237019	0.229456	False	21915	1.15555e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0712554	0.0737091	0.0696521	False	10.348	1074.97	1103.31	1014.28	False	0.002239	0.266937	0.271511	0.263948	False
6	[36000:41999]	6	36000	41999	2017-02-07 08:00:00	2017-02-09 15:59:26.400000	reference	0	8.21576	842.253	874.805	817.855	False	0.00248466	0.232789	0.237019	0.229456	False	21915	1.12037e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0715653	0.0737091	0.0696521	False	10.348	1058.48	1103.31	1014.28	False	0.002239	0.267517	0.271511	0.263948	False
7	[42000:47999]	7	42000	47999	2017-02-09 16:00:00	2017-02-11 23:59:26.400000	reference	0	8.21576	837.9	874.805	817.855	False	0.00248466	0.235516	0.237019	0.229456	False	21915	1.10396e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0729194	0.0737091	0.0696521	False	10.348	1050.7	1103.31	1014.28	False	0.002239	0.270036	0.271511	0.263948	False
8	[48000:53999]	8	48000	53999	2017-02-12 00:00:00	2017-02-14 07:59:26.400000	reference	0	8.21576	844.266	874.805	817.855	False	0.00248466	0.232423	0.237019	0.229456	False	21915	1.09914e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0711648	0.0737091	0.0696521	False	10.348	1048.4	1103.31	1014.28	False	0.002239	0.266767	0.271511	0.263948	False
9	[54000:59999]	9	54000	59999	2017-02-14 08:00:00	2017-02-16 15:59:26.400000	reference	0	8.21576	850.673	874.805	817.855	False	0.00248466	0.233561	0.237019	0.229456	False	21915	1.12369e+06	1.21572e+06	1.02681e+06	False	0.0011989	0.0715405	0.0737091	0.0696521	False	10.348	1060.04	1103.31	1014.28	False	0.002239	0.267471	0.271511	0.263948	False

Apart from chunking and chunk and period-related columns, the results data have a set of columns for each calculated metric. When taking mae as an example:

• targets_missing_rate - The fraction of missing target data.

• <metric> - The value of the metric for a specific chunk.

• <metric>_lower_threshold> and <metric>_upper_threshold> - Lower and upper thresholds for performance metric. Crossing them will raise an alert that there is a significant metric change. The thresholds are calculated based on the realized performance of chunks in the reference period. The thresholds are 3 standard deviations away from the mean performance calculated on reference chunks. They are calculated during fit phase.

• <metric>_alert - A flag indicating potentially significant performance change. True if realized performance crosses upper or lower threshold.

• <metric>_sampling_error - Estimated Sampling Error for the relevant metric.

The results can be plotted for visual inspection:

>>> figure = results.plot(kind='performance')
>>> figure.show()

Insights

From looking at the RMSE and RMSLE performance results we can observe an interesting effect. We know that RMSE penalizes mispredictions symmetrically while RMSLE penalizes underprediction more than overprediction. Hence while our model has become a little bit more accurate according to RMSE, the increase in RMSLE tells us that our model is now underpredicting more than it was before!

What Next

If we decide further investigation is needed, the Data Drift functionality can help us to see what feature changes may be contributing to any performance changes.

It is also wise to check whether the model’s performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.