nannyml.performance_calculation.metrics module

Module containing metric utilities and implementations.

class nannyml.performance_calculation.metrics.BinaryClassificationAUROC(calculator)

Bases: Metric

Area under Receiver Operating Curve metric.

Creates a new AUROC instance.

class nannyml.performance_calculation.metrics.BinaryClassificationAccuracy(calculator)

Bases: Metric

Accuracy metric.

Creates a new Accuracy instance.

class nannyml.performance_calculation.metrics.BinaryClassificationF1(calculator)

Bases: Metric

F1 score metric.

Creates a new F1 instance.

class nannyml.performance_calculation.metrics.BinaryClassificationPrecision(calculator)

Bases: Metric

Precision metric.

Creates a new Precision instance.

class nannyml.performance_calculation.metrics.BinaryClassificationRecall(calculator)

Bases: Metric

Recall metric, also known as ‘sensitivity’.

Creates a new Recall instance.

class nannyml.performance_calculation.metrics.BinaryClassificationSpecificity(calculator)

Bases: Metric

Specificity metric.

Creates a new F1 instance.

class nannyml.performance_calculation.metrics.Metric(display_name: str, column_name: str, calculator: AbstractCalculator, upper_threshold: Optional[float] = None, lower_threshold: Optional[float] = None)

Bases: ABC

A performance metric used to calculate realized model performance.

Creates a new Metric instance.

Parameters

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

• calculator (AbstractCalculator) – The calculator using the Metric instance.

• upper_threshold (float, default=None) – An optional upper threshold for the performance metric.

• lower_threshold (float, default=None) – An optional lower threshold for the performance metric.

__eq__(other)

Establishes equality by comparing all properties.

calculate(data: DataFrame)

Calculates performance metrics on data.

Parameters

data (pd.DataFrame) – The data to calculate performance metrics on. Requires presence of either the predicted labels or prediction scores/probabilities (depending on the metric to be calculated), as well as the target data.

fit(reference_data: DataFrame, chunker: Chunker)

Fits a Metric on reference data.

Parameters

• reference_data (pd.DataFrame) – The reference data used for fitting. Must have target data available.

• chunker (Chunker) – The Chunker used to split the reference data into chunks. This value is provided by the calling PerformanceCalculator.

minimum_chunk_size() → int

Determines the minimum number of observations a chunk should ideally for this metric to be trustworthy.

class nannyml.performance_calculation.metrics.MetricFactory

Bases: object

A factory class that produces Metric instances based on a given magic string or a metric specification.

classmethod create(key: str, use_case: UseCase, kwargs: Dict[str, Any] = {}) → Metric

Returns a Metric instance for a given key.

classmethod register(metric: str, use_case: UseCase) → Callable

registry: Dict[str, Dict[UseCase, Metric]] = {'accuracy': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationAccuracy'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationAccuracy'>}, 'f1': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationF1'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationF1'>}, 'precision': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationPrecision'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationPrecision'>}, 'recall': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationRecall'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationRecall'>}, 'roc_auc': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationAUROC'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationAUROC'>}, 'specificity': {UseCase.CLASSIFICATION_BINARY: <class 'nannyml.performance_calculation.metrics.BinaryClassificationSpecificity'>, UseCase.CLASSIFICATION_MULTICLASS: <class 'nannyml.performance_calculation.metrics.MulticlassClassificationSpecificity'>}}

class nannyml.performance_calculation.metrics.MulticlassClassificationAUROC(calculator)

Bases: Metric

Area under Receiver Operating Curve metric.

Creates a new AUROC instance.

class nannyml.performance_calculation.metrics.MulticlassClassificationAccuracy(calculator)

Bases: Metric

Accuracy metric.

Creates a new Accuracy instance.

class nannyml.performance_calculation.metrics.MulticlassClassificationF1(calculator)

Bases: Metric

F1 score metric.

Creates a new F1 instance.

class nannyml.performance_calculation.metrics.MulticlassClassificationPrecision(calculator)

Bases: Metric

Precision metric.

Creates a new Precision instance.

class nannyml.performance_calculation.metrics.MulticlassClassificationRecall(calculator)

Bases: Metric

Recall metric, also known as ‘sensitivity’.

Creates a new Recall instance.

class nannyml.performance_calculation.metrics.MulticlassClassificationSpecificity(calculator)

Bases: Metric

Specificity metric.

Creates a new Specificity instance.