nannyml.performance_estimation.direct_loss_estimation.metrics module

class nannyml.performance_estimation.direct_loss_estimation.metrics.MAE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float

class nannyml.performance_estimation.direct_loss_estimation.metrics.MAPE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float

class nannyml.performance_estimation.direct_loss_estimation.metrics.MSE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float

class nannyml.performance_estimation.direct_loss_estimation.metrics.MSLE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float

class nannyml.performance_estimation.direct_loss_estimation.metrics.Metric(display_name: str, column_name: str, feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any], upper_value_limit: float | None = None, lower_value_limit: float = 0.0)

Bases: ABC

A performance metric used to estimate regression performance.

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

__eq__(other)

Establishes equality by comparing all properties.

estimate(data: DataFrame)

Calculates performance metrics on data.

Parameters:

data (pd.DataFrame) – The data to estimate performance metrics for. Requires presence of either the predicted labels or prediction scores/probabilities (depending on the metric to be calculated).

fit(reference_data: DataFrame)

Fits a Metric on reference data.

Parameters:

reference_data (pd.DataFrame) – The reference data used for fitting. Must have target data available.

abstract realized_performance(data: DataFrame) → float

sampling_error(data: DataFrame)

Calculates the sampling error with respect to the reference data for a given chunk of data.

Parameters:

data (pd.DataFrame) – The data to calculate the sampling error on, with respect to the reference data.

Returns:

sampling_error – The expected sampling error.

Return type:

float

class nannyml.performance_estimation.direct_loss_estimation.metrics.MetricFactory

Bases: object

A factory class that produces Metric instances based on a given magic string or a metric specification.

classmethod create(key: str, problem_type: ProblemType, **kwargs) → Metric

Returns a Metric instance for a given key.

classmethod register(metric: str, problem_type: ProblemType) → Callable

registry: Dict[str, Dict[ProblemType, Metric]] = {'mae': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.MAE'>}, 'mape': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.MAPE'>}, 'mse': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.MSE'>}, 'msle': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.MSLE'>}, 'rmse': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.RMSE'>}, 'rmsle': {ProblemType.REGRESSION: <class 'nannyml.performance_estimation.direct_loss_estimation.metrics.RMSLE'>}}

class nannyml.performance_estimation.direct_loss_estimation.metrics.RMSE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float

class nannyml.performance_estimation.direct_loss_estimation.metrics.RMSLE(feature_column_names: List[str], y_true: str, y_pred: str, chunker: Chunker, tune_hyperparameters: bool, hyperparameter_tuning_config: Dict[str, Any], hyperparameters: Dict[str, Any])

Bases: Metric

Creates a new Metric instance.

Parameters:

• display_name (str) – The name of the metric. Used to display in plots. If not given this name will be derived from the calculation_function.

• column_name (str) – The name used to indicate the metric in columns of a DataFrame.

realized_performance(data: DataFrame) → float