nannyml.performance_calculation.calculator module

Calculates realized performance metrics when target data is available.

class nannyml.performance_calculation.calculator.PerformanceCalculator(timestamp_column_name: str, metrics: List[str], y_true: str, y_pred_proba: Optional[Union[str, Dict[str, str]]], y_pred: Optional[str], chunk_size: Optional[int] = None, chunk_number: Optional[int] = None, chunk_period: Optional[str] = None, chunker: Optional[Chunker] = None)[source]

Bases: AbstractCalculator

Calculates realized performance metrics when target data is available.

Creates a new performance calculator.

Parameters

  • y_true (str) – The name of the column containing target values.

  • y_pred_proba (ModelOutputsType) – Name(s) of the column(s) containing your model output. Pass a single string when there is only a single model output column, e.g. in binary classification cases. Pass a dictionary when working with multiple output columns, e.g. in multiclass classification cases. The dictionary maps a class/label string to the column name containing model outputs for that class/label.

  • y_pred (str) – The name of the column containing your model predictions.

  • timestamp_column_name (str) – The name of the column containing the timestamp of the model prediction.

  • metrics (List[str]) – A list of metrics to calculate.

  • chunk_size (int) – Splits the data into chunks containing chunks_size observations. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_number (int) – Splits the data into chunk_number pieces. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunk_period (str) – Splits the data according to the given period. Only one of chunk_size, chunk_number or chunk_period should be given.

  • chunker (Chunker) – The Chunker used to split the data sets into a lists of chunks.

Examples

>>> import nannyml as nml
>>>
>>> reference_df, analysis_df, target_df = nml.load_synthetic_binary_classification_dataset()
>>>
>>> calc = nml.PerformanceCalculator(y_true='work_home_actual', y_pred='y_pred', y_pred_proba='y_pred_proba',
>>>                                  timestamp_column_name='timestamp', metrics=['f1', 'roc_auc'])
>>>
>>> calc.fit(reference_df)
>>>
>>> results = calc.calculate(analysis_df.merge(target_df, on='identifier'))
>>> print(results.data)
             key  start_index  ...  roc_auc_upper_threshold roc_auc_alert
0       [0:4999]            0  ...                  0.97866         False
1    [5000:9999]         5000  ...                  0.97866         False
2  [10000:14999]        10000  ...                  0.97866         False
3  [15000:19999]        15000  ...                  0.97866         False
4  [20000:24999]        20000  ...                  0.97866         False
5  [25000:29999]        25000  ...                  0.97866          True
6  [30000:34999]        30000  ...                  0.97866          True
7  [35000:39999]        35000  ...                  0.97866          True
8  [40000:44999]        40000  ...                  0.97866          True
9  [45000:49999]        45000  ...                  0.97866          True
>>> for metric in calc.metrics:
>>>     results.plot(metric=metric, plot_reference=True).show()