nannyml.performance_calculation.metrics module
Module containing metric utilities and implementations.
- class nannyml.performance_calculation.metrics.BinaryClassificationAUROC(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Area under Receiver Operating Curve metric.
Creates a new AUROC instance.
- class nannyml.performance_calculation.metrics.BinaryClassificationAccuracy(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Accuracy metric.
Creates a new Accuracy instance.
- class nannyml.performance_calculation.metrics.BinaryClassificationF1(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
F1 score metric.
Creates a new F1 instance.
- class nannyml.performance_calculation.metrics.BinaryClassificationPrecision(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Precision metric.
Creates a new Precision instance.
- class nannyml.performance_calculation.metrics.BinaryClassificationRecall(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Recall metric, also known as ‘sensitivity’.
Creates a new Recall instance.
- class nannyml.performance_calculation.metrics.BinaryClassificationSpecificity(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Specificity metric.
Creates a new F1 instance.
- class nannyml.performance_calculation.metrics.Metric(display_name: str, column_name: str, metadata: nannyml.metadata.base.ModelMetadata, upper_threshold: Optional[float] = None, lower_threshold: Optional[float] = None)[source]
Bases:
abc.ABC
Represents a performance metric.
Creates a new Metric instance.
- Parameters
display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the
calculation_function
.column_name (str) – The name used to indicate the metric in columns of a DataFrame.
metadata (ModelMetadata) – Metadata describing the model being monitored.
upper_threshold (float, default=None) – An optional upper threshold for the performance metric.
lower_threshold (float, default=None) – An optional lower threshold for the performance metric.
- calculate(data: pandas.core.frame.DataFrame)[source]
Calculates performance metrics on data.
- Parameters
data (pd.DataFrame) – The data to calculate performance metrics on. Requires presence of either the predicted labels or prediction scores/probabilities (depending on the metric to be calculated), as well as the target data.
- fit(reference_data: pandas.core.frame.DataFrame, chunker: nannyml.chunk.Chunker)[source]
Fits a Metric on reference data.
- Parameters
reference_data (pd.DataFrame) – The reference data used for fitting. Must have target data available.
chunker (Chunker) – The
Chunker
used to split the reference data into chunks. This value is provided by the callingPerformanceCalculator
.
- class nannyml.performance_calculation.metrics.MetricFactory[source]
Bases:
object
A factory class that produces Metric instances based on a given magic string or a metric specification.
- classmethod create(key: str, metadata: nannyml.metadata.base.ModelMetadata) nannyml.performance_calculation.metrics.Metric [source]
Returns a Metric instance for a given key.
- class nannyml.performance_calculation.metrics.MulticlassClassificationAUROC(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Area under Receiver Operating Curve metric.
Creates a new AUROC instance.
- class nannyml.performance_calculation.metrics.MulticlassClassificationAccuracy(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Accuracy metric.
Creates a new Accuracy instance.
- class nannyml.performance_calculation.metrics.MulticlassClassificationF1(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
F1 score metric.
Creates a new F1 instance.
- class nannyml.performance_calculation.metrics.MulticlassClassificationPrecision(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Precision metric.
Creates a new Precision instance.
- class nannyml.performance_calculation.metrics.MulticlassClassificationRecall(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Recall metric, also known as ‘sensitivity’.
Creates a new Recall instance.
- class nannyml.performance_calculation.metrics.MulticlassClassificationSpecificity(metadata: nannyml.metadata.base.ModelMetadata)[source]
Bases:
nannyml.performance_calculation.metrics.Metric
Specificity metric.
Creates a new Specificity instance.