nannyml.performance_estimation.confidence_based.metrics module

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationAUROC(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationAccuracy(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationBusinessValue(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, business_value_matrix: Union[List, numpy.ndarray], normalize_business_value: Optional[str] = None, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationConfusionMatrix(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, normalize_confusion_matrix: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

fit(reference_data: pandas.core.frame.DataFrame)[source]: Fits a Metric on reference data. :param reference_data: The reference data used for fitting. Must have target data available. :type reference_data: pd.DataFrame

get_chunk_record(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

get_false_neg_info(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

get_false_negative_estimate(chunk_data: pandas.core.frame.DataFrame) → float[source]

get_false_pos_info(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

get_false_positive_estimate(chunk_data: pandas.core.frame.DataFrame) → float[source]

get_true_neg_info(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

get_true_negative_estimate(chunk_data: pandas.core.frame.DataFrame) → float[source]

get_true_pos_info(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

get_true_positive_estimate(chunk_data: pandas.core.frame.DataFrame) → float[source]

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationF1(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationPrecision(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationRecall(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationSpecificity(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.Metric(name: str, y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, components: List[Tuple[str, str]], timestamp_column_name: Optional[str] = None, lower_threshold_value_limit: Optional[float] = None, upper_threshold_value_limit: Optional[float] = None, **kwargs)[source]

Bases: abc.ABC

A performance metric used to calculate realized model performance.

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

alert(value: float) → bool[source]

property column_name: str

property column_names

property display_name: str

property display_names

fit(reference_data: pandas.core.frame.DataFrame)[source]

Fits a Metric on reference data.

Parameters: reference_data (pd.DataFrame) – The reference data used for fitting. Must have target data available.

get_chunk_record(chunk_data: pandas.core.frame.DataFrame) → Dict[source]

class nannyml.performance_estimation.confidence_based.metrics.MetricFactory[source]

Bases: object

A factory class that produces Metric instances based on a given magic string or a metric specification.

classmethod create(key: str, use_case: nannyml._typing.ProblemType, **kwargs) → nannyml.performance_estimation.confidence_based.metrics.Metric[source]

classmethod register(metric: str, use_case: nannyml._typing.ProblemType) → Callable[source]

registry: Dict[str, Dict[nannyml._typing.ProblemType, Type[nannyml.performance_estimation.confidence_based.metrics.Metric]]] = {'accuracy': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationAccuracy'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationAccuracy'>}, 'business_value': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationBusinessValue'>}, 'confusion_matrix': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationConfusionMatrix'>}, 'f1': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationF1'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationF1'>}, 'precision': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationPrecision'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationPrecision'>}, 'recall': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationRecall'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationRecall'>}, 'roc_auc': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationAUROC'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationAUROC'>}, 'specificity': {ProblemType.CLASSIFICATION_BINARY: <class 'nannyml.performance_estimation.confidence_based.metrics.BinaryClassificationSpecificity'>, ProblemType.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationSpecificity'>}}

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationAUROC(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationAccuracy(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationF1(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationPrecision(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationRecall(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

class nannyml.performance_estimation.confidence_based.metrics.MulticlassClassificationSpecificity(y_pred_proba: Union[str, Dict[str, str]], y_pred: str, y_true: str, chunker: nannyml.chunk.Chunker, threshold: nannyml.thresholds.Threshold, timestamp_column_name: Optional[str] = None, **kwargs)[source]

Bases: nannyml.performance_estimation.confidence_based.metrics.Metric

Creates a new Metric instance.

Parameters: name (str) – The name used to indicate the metric in columns of a DataFrame.

nannyml.performance_estimation.confidence_based.metrics.estimate_business_value(y_pred: numpy.ndarray, y_pred_proba: numpy.ndarray, normalize_business_value: Optional[str], business_value_matrix: numpy.ndarray)[source]

nannyml.performance_estimation.confidence_based.metrics.estimate_f1(y_pred: pandas.core.frame.DataFrame, y_pred_proba: pandas.core.frame.DataFrame) → float[source]

nannyml.performance_estimation.confidence_based.metrics.estimate_precision(y_pred: pandas.core.frame.DataFrame, y_pred_proba: pandas.core.frame.DataFrame) → float[source]

nannyml.performance_estimation.confidence_based.metrics.estimate_recall(y_pred: pandas.core.frame.DataFrame, y_pred_proba: pandas.core.frame.DataFrame) → float[source]

nannyml.performance_estimation.confidence_based.metrics.estimate_roc_auc(y_pred_proba: pandas.core.series.Series) → float[source]

nannyml.performance_estimation.confidence_based.metrics.estimate_specificity(y_pred: pandas.core.frame.DataFrame, y_pred_proba: pandas.core.frame.DataFrame) → float[source]