nannyml.performance_estimation.confidence_based.cbpe module
Implementation of the CBPE estimator.
- class nannyml.performance_estimation.confidence_based.cbpe.CBPE(y_pred_proba: Union[str, Dict[str, str]], *args, **kwargs)[source]
Bases:
AbstractEstimator
Performance estimator using the Confidence Based Performance Estimation (CBPE) technique.
Initializes a new CBPE performance estimator.
- Parameters
metrics (List[str]) – A list of metrics to calculate.
chunk_size (int, default=None) – Splits the data into chunks containing chunks_size observations. Only one of chunk_size, chunk_number or chunk_period should be given.
chunk_number (int, default=None) – Splits the data into chunk_number pieces. Only one of chunk_size, chunk_number or chunk_period should be given.
chunk_period (str, default=None) – Splits the data according to the given period. Only one of chunk_size, chunk_number or chunk_period should be given.
chunker (Chunker, default=None) – The Chunker used to split the data sets into a lists of chunks.
calibration (str, default='isotonic') – Determines which calibration will be applied to the model predictions. Defaults to
isotonic
, currently the only supported value.calibrator (Calibrator, default=None) – A specific instance of a Calibrator to be applied to the model predictions. If not set NannyML will use the value of the
calibration
variable instead.
Examples
>>> import nannyml as nml >>> >>> reference_df, analysis_df, target_df = nml.load_synthetic_binary_classification_dataset() >>> >>> estimator = nml.CBPE( >>> y_true='work_home_actual', >>> y_pred='y_pred', >>> y_pred_proba='y_pred_proba', >>> timestamp_column_name='timestamp', >>> metrics=['f1', 'roc_auc'] >>> ) >>> >>> estimator.fit(reference_df) >>> >>> results = estimator.estimate(analysis_df) >>> print(results.data) key start_index ... lower_threshold_roc_auc alert_roc_auc 0 [0:4999] 0 ... 0.97866 False 1 [5000:9999] 5000 ... 0.97866 False 2 [10000:14999] 10000 ... 0.97866 False 3 [15000:19999] 15000 ... 0.97866 False 4 [20000:24999] 20000 ... 0.97866 False 5 [25000:29999] 25000 ... 0.97866 True 6 [30000:34999] 30000 ... 0.97866 True 7 [35000:39999] 35000 ... 0.97866 True 8 [40000:44999] 40000 ... 0.97866 True 9 [45000:49999] 45000 ... 0.97866 True >>> for metric in estimator.metrics: >>> results.plot(metric=metric, plot_reference=True).show()