Why Monitoring Realized Performance
The realized performance of a machine learning model is typically a good proxy for the business impact of the model. A significant drop in performance normally means a lot of value generated by the model is at risk, so close monitoring and quick resolution of issues are essential.
This guide shows how to use NannyML to calculate the Realized Performance of a model. Target values need to be available in both the reference and analysis data. All monitoring metrics available by NannyML for monitoring will be showed.
The performance monitoring process requires no missing values in the target data on the reference dataset. However, the analysis data can contain missing values. The entries with missing values will simply be ignored when calculating the performance results. If there are so many missing values that the available data are below the Minimum chunk size then the performance results are omitted from the resulting visualizations because they are too noisy, to be reliable.