nannyml.performance_estimation.confidence_based.cbpe module

Implementation of the CBPE estimator.

class nannyml.performance_estimation.confidence_based.cbpe.CBPE(metrics: Union[str, List[str]], y_pred: str, y_pred_proba: Union[str, Dict[str, str]], y_true: str, problem_type: Union[str, nannyml._typing.ProblemType], timestamp_column_name: Optional[str] = None, chunk_size: Optional[int] = None, chunk_number: Optional[int] = None, chunk_period: Optional[str] = None, chunker: Optional[nannyml.chunk.Chunker] = None, calibration: Optional[str] = None, calibrator: Optional[nannyml.calibration.Calibrator] = None, thresholds: Optional[Dict[str, nannyml.thresholds.Threshold]] = None, normalize_confusion_matrix: Optional[str] = None, business_value_matrix: Optional[Union[List, numpy.ndarray]] = None, normalize_business_value: Optional[str] = None)[source]

Bases: nannyml.base.AbstractEstimator

Performance estimator using the Confidence Based Performance Estimation (CBPE) technique.

Initializes a new CBPE performance estimator.

Parameters

  • y_true (str) – The name of the column containing target values (that are provided in reference data during fitting).

  • y_pred_proba (ModelOutputsType) – Name(s) of the column(s) containing your model output. Pass a single string when there is only a single model output column, e.g. in binary classification cases. Pass a dictionary when working with multiple output columns, e.g. in multiclass classification cases. The dictionary maps a class/label string to the column name containing model outputs for that class/label.

  • y_pred (str) – The name of the column containing your model predictions.

  • timestamp_column_name (str, default=None) – The name of the column containing the timestamp of the model prediction.

  • metrics (Union[str, List[str]]) – A metric or list of metrics to calculate.

  • chunk_size (int, default=None) – Splits the data into chunks containing chunks_size observations. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_number (int, default=None) – Splits the data into chunk_number pieces. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_period (str, default=None) – Splits the data according to the given period. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunker (Chunker, default=None) – The Chunker used to split the data sets into a lists of chunks.

  • calibration (str, default='isotonic') – Determines which calibration will be applied to the model predictions. Defaults to isotonic, currently the only supported value.

  • calibrator (Calibrator, default=None) – A specific instance of a Calibrator to be applied to the model predictions. If not set NannyML will use the value of the calibration variable instead.

  • thresholds (dict, default={ 'roc_auc': StandardDeviationThreshold(), 'f1': StandardDeviationThreshold(), 'precision': StandardDeviationThreshold(), 'recall': StandardDeviationThreshold(), 'specificity': StandardDeviationThreshold(), 'accuracy': StandardDeviationThreshold(), 'confusion_matrix': StandardDeviationThreshold(), 'business_cost': StandardDeviationThreshold(), }) –

    A dictionary allowing users to set a custom threshold for each method. It links a Threshold subclass to a method name. This dictionary is optional. When a dictionary is given its values will override the default values. If no dictionary is given a default will be applied. The default method thresholds are as follows:

    • roc_auc: StandardDeviationThreshold()

    • f1: StandardDeviationThreshold()

    • precision: StandardDeviationThreshold()

    • recall: StandardDeviationThreshold()

    • specificity: StandardDeviationThreshold()

    • accuracy: StandardDeviationThreshold()

    • confusion_matrix: StandardDeviationThreshold()

    • mape: StandardDeviationThreshold()

    • business_cost: StandardDeviationThreshold()

  • problem_type (Union[str, ProblemType]) – Determines which CBPE implementation to use. Allowed problem type values are ‘classification_binary’ and ‘classification_multiclass’.

  • normalize_confusion_matrix (str, default=None) – Determines how the confusion matrix will be normalized. Allowed values are None, ‘all’, ‘true’ and ‘predicted’. If None, the confusion matrix will not be normalized and the counts for each cell of the matrix will be returned. If ‘all’, the confusion matrix will be normalized by the total number of observations. If ‘true’, the confusion matrix will be normalized by the total number of observations for each true class. If ‘predicted’, the confusion matrix will be normalized by the total number of observations for each predicted class.

  • business_value_matrix (Optional[Union[List, np.ndarray]], default=None) – A matrix containing the business value for each combination of true and predicted class. The i-th row and j-th column entry of the matrix contains the business value for predicting the i-th class as the j-th class.

  • normalize_business_value (str, default=None) – Determines how the business value will be normalized. Allowed values are None and ‘per_prediction’. If None, the business value will not be normalized and the value returned will be the total value per chunk. If ‘per_prediction’, the value will be normalized by the number of predictions in the chunk.

Examples

>>> import nannyml as nml
>>> from IPython.display import display
>>> reference_df = nml.load_synthetic_binary_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_binary_classification_dataset()[1]
>>> display(reference_df.head(3))
>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='work_home_actual',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=5000,
...     problem_type='classification_binary',
>>> )
>>> estimator.fit(reference_df)
>>> results = estimator.estimate(analysis_df)
>>> display(results.data)
>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', metric=metric)
...     metric_fig.show()
>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', plot_reference=True, metric=metric)
...     metric_fig.show()