Business Value Estimation and Calculation
The business_value metric provides a way to tie the performance of a model to monetary or business oriented outcomes. In this page, we will discuss how the business_value metric works under the hood.
Introduction to Business Value
The business_value metric offers a way to quantify the value of a model in terms of the business’s own metrics. At the core, if the business value (or cost) of each outcome in the confusion matrix is known, then the business value of a model can either be calculated using the realized Performance Calculator if the ground truth labels are available or estimated using Performance Estimation if the ground truth labels are not available.
More specifically, we know that each prediction made by a binary classification models can be one of four outcomes:
True Positive (TP): The model correctly predicts a positive outcome.
True Negative (TN): The model correctly predicts a negative outcome.
False Positive (FP): The model incorrectly predicts a positive outcome.
False Negative (FN): The model incorrectly predicts a negative outcome.
The business value of each of these four outcomes can be calculated according to actual business results and costs. The total business value of a model can then be calculated by summing the business value of each prediction.
For example, if the value of a true positive is $100,000, the value of a true negative is $0, the value of a false positive is $1,000, and the value of a false negative is $10,000, then the business value of a can be calculated as follows:
Business Value Formula
We can formalize the intuition above as follows:
where \(\text{business_value}_{i,j}\) is the business value of a cell in the confusion matrix, and \(\text{confusion_matrix}_{i,j}\) is the count of observations in that cell of the confusion matrix. Using the confusion matrix notation the element on the i-th row and j-column of the business value matrix tells us the value of the i-th target when we have predicted the j-th value.
Note
In Multiclass classification the classes are ordered alphanumerically. This is used in the creation of the confusion matrix. The rows of the confusion matrix represent target values in the corresponding alphanumerical order. And the columns of the confusion matrix represent predicted classes in the same alphanumerical order. Therefore the elements of the business value matrix should be constructed accordingly.
For binary classification this formula is easier to manage hence we will use it as an example. Classificatio problems with more classes follow the same pattern. Using the sklearn confusion matrix convention we designate label 0 as negative and label 1 as positive. Hence we can write the confusion matrix as:
Note that target values are represented by rows and predicted values are represented by columns. This means that the first row contains values that have resulted in the negative outcome while the first column contains values that were predicted with negative label. The correspondings business value matrix is:
The business value of a binary classification model can thus be generally expressed as:
Calculation of Business Value For Classification
When the ground truth labels are available, the business value of a model can be calculated by using the values from the realized confusion matrix, and then using the business value formula above to calculate the business value.
For a tutorial on how to calculate the business value of a model, see our Calculating Business Value for Binary Classification and Calculating Business Value for Multiclass Classification tutorials.
Estimation of Business Value For Classification
In cases where ground truth labels of the data are unavailable, we can still estimate the business value of a model. This is done by using the CBPE (Confidence-Based Performance Estimation) algorithm to estimate the confusion matrix, and then using the business value formula above to obtain a business value estimate. To read more about the CBPE (Confidence-Based Performance Estimation) algorithm, see our performance estimation deep dive.
For a tutorial on how to estimate the business value of a model, see our Estimating Business Value for Binary Classification and Estimating Business Value for Multiclass Classification tutorials.
Normalization
The business_value metric can be normalized so that the value returned is the business value per prediction. The advantage of this is that it allows for easy comparison of the business value of different models, even if they have different numbers of predictions. Further, it allows for easy comparison of the business value of the same model on different chunks of data, if they have different numbers of predictions as is often the case when using period-based chunking.
Under the hood normalization is quite simple. The total business_value metric is calculated or estimated as described above, and then divided by the number of predictions in a given chunk.
Normalization is supported for both estimation and calculation of business value. Check out the Calculating Business Value for Binary Classification tutorial and the Estimating Business Value for Binary Classification tutorial for examples of how to normalize the business value metric.