nannyml.performance_estimation.confidence_based.results module

Module containing CBPE estimation results and plotting implementations.

class nannyml.performance_estimation.confidence_based.results.Result(results_data: pandas.core.frame.DataFrame, metrics: List[nannyml.performance_estimation.confidence_based.metrics.Metric], y_pred: str, y_pred_proba: Union[str, Dict[str, str]], y_true: str, chunker: nannyml.chunk.Chunker, problem_type: nannyml._typing.ProblemType, timestamp_column_name: Optional[str] = None)[source]

Bases: nannyml.base.PerMetricResult[nannyml.performance_estimation.confidence_based.metrics.Metric], nannyml.plots.blueprints.comparisons.ResultCompareMixin

Contains results for CBPE estimation and adds filtering and plotting functionality.

Parameters

  • results_data (pd.DataFrame) – Results data returned by a CBPE estimator.

  • metrics (List[nannyml.performance_estimation.confidence_based.metrics.Metric]) – List of metrics to evaluate.

  • y_pred (str) – The name of the column containing your model predictions.

  • y_pred_proba (Union[str, Dict[str, str]]) –

    Name(s) of the column(s) containing your model output.

    • For binary classification, pass a single string refering to the model output column.

    • For multiclass classification, pass a dictionary that maps a class string to the column name containing model outputs for that class.

  • y_true (str) – The name of the column containing target values (that are provided in reference data during fitting).

  • chunker (Chunker) – The Chunker used to split the data sets into a lists of chunks.

  • problem_type (ProblemType) – Determines which CBPE implementation to use. Allowed problem type values are ‘classification_binary’ and ‘classification_multiclass’.

  • timestamp_column_name (str, default=None) – The name of the column containing the timestamp of the model prediction. If not given, plots will not use a time-based x-axis but will use the index of the chunks instead.

keys() → List[nannyml._typing.Key][source]

Creates a list of keys where each Key is a namedtuple(‘Key’, ‘properties display_names’)

plot(kind: str = 'performance', *args, **kwargs) → plotly.graph_objs._figure.Figure[source]

Render plots based on CBPE estimation results.

This function will return a plotly.graph_objects.Figure object. The following kinds of plots are available:

Parameters

kind (str, default='performance') –

Raises

InvalidArgumentsException – when an unknown plot kind is provided.:

Returns

fig – A Figure object containing the requested drift plot.

Can be saved to disk using the write_image() method or shown rendered on screen using the show() method.

Return type

plotly.graph_objs._figure.Figure

Examples

>>> import nannyml as nml
>>>
>>> reference_df, analysis_df, target_df = nml.load_synthetic_binary_classification_dataset()
>>>
>>> estimator = nml.CBPE(
>>>     y_true='work_home_actual',
>>>     y_pred='y_pred',
>>>     y_pred_proba='y_pred_proba',
>>>     timestamp_column_name='timestamp',
>>>     metrics=['f1', 'roc_auc']
>>> )
>>>
>>> estimator.fit(reference_df)
>>>
>>> results = estimator.estimate(analysis_df)
>>> results.plot().show()