nannyml.performance_estimation.confidence_based.cbpe module

Implementation of the CBPE estimator.

class nannyml.performance_estimation.confidence_based.cbpe.CBPE(metrics: Union[str, List[str]], y_pred: str, y_pred_proba: Union[str, Dict[str, str]], y_true: str, problem_type: Union[str, nannyml._typing.ProblemType], timestamp_column_name: Optional[str] = None, chunk_size: Optional[int] = None, chunk_number: Optional[int] = None, chunk_period: Optional[str] = None, chunker: Optional[nannyml.chunk.Chunker] = None, calibration: Optional[str] = None, calibrator: Optional[nannyml.calibration.Calibrator] = None)[source]

Bases: nannyml.base.AbstractEstimator

Performance estimator using the Confidence Based Performance Estimation (CBPE) technique.

Initializes a new CBPE performance estimator.

Parameters

  • y_true (str) – The name of the column containing target values (that are provided in reference data during fitting).

  • y_pred_proba (ModelOutputsType) – Name(s) of the column(s) containing your model output. Pass a single string when there is only a single model output column, e.g. in binary classification cases. Pass a dictionary when working with multiple output columns, e.g. in multiclass classification cases. The dictionary maps a class/label string to the column name containing model outputs for that class/label.

  • y_pred (str) – The name of the column containing your model predictions.

  • timestamp_column_name (str, default=None) – The name of the column containing the timestamp of the model prediction.

  • metrics (Union[str, List[str]]) – A metric or list of metrics to calculate.

  • chunk_size (int, default=None) – Splits the data into chunks containing chunks_size observations. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_number (int, default=None) – Splits the data into chunk_number pieces. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_period (str, default=None) – Splits the data according to the given period. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunker (Chunker, default=None) – The Chunker used to split the data sets into a lists of chunks.

  • calibration (str, default='isotonic') – Determines which calibration will be applied to the model predictions. Defaults to isotonic, currently the only supported value.

  • calibrator (Calibrator, default=None) – A specific instance of a Calibrator to be applied to the model predictions. If not set NannyML will use the value of the calibration variable instead.

  • problem_type (Union[str, ProblemType]) – Determines which CBPE implementation to use. Allowed problem type values are ‘classification_binary’ and ‘classification_multiclass’.

Examples

>>> import nannyml as nml
>>> from IPython.display import display
>>> reference_df = nml.load_synthetic_binary_classification_dataset()[0]
>>> analysis_df = nml.load_synthetic_binary_classification_dataset()[1]
>>> display(reference_df.head(3))
>>> estimator = nml.CBPE(
...     y_pred_proba='y_pred_proba',
...     y_pred='y_pred',
...     y_true='work_home_actual',
...     timestamp_column_name='timestamp',
...     metrics=['roc_auc', 'f1'],
...     chunk_size=5000,
...     problem_type='classification_binary',
>>> )
>>> estimator.fit(reference_df)
>>> results = estimator.estimate(analysis_df)
>>> display(results.data)
>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', metric=metric)
...     metric_fig.show()
>>> for metric in estimator.metrics:
...     metric_fig = results.plot(kind='performance', plot_reference=True, metric=metric)
...     metric_fig.show()