NannyML
main
Contents:
Quickstart
What is NannyML?
Installing NannyML
Contents of the quickstart
Just the code
Walkthrough
Estimating Performance without Targets
Detecting Data Drift
Insights
What next
Tutorials
Setting Up
Data requirements
Data Periods
Common columns
Binary classification columns
Multiclass classification columns
Regression columns
What next
Providing metadata
Why is providing metadata required?
Metadata for binary classification
Metadata for multiclass classification
Metadata for regression
Insights
What next
Chunking
Why do we need chunks?
Walkthrough on creating chunks
Chunks on plots with results
Estimating Performance
Estimating Performance for Binary Classification
Why Perform Performance Estimation
Insights
What’s next
Estimating Performance for Multiclass Classification
Why Perform Performance Estimation
Insights
What’s next
Estimating Performance for Multiclass Classification
Monitoring Realized Performance
Monitoring Realized Performance for Binary Classification
Why Monitor Realized Performance
Insights
What Next
Monitoring Realized Performance for Multiclass Classification
Why Monitor Realized Performance
Insights
What Next
Monitoring Realized Performance for Regression
Comparing Estimated and Realized Performance
Detecting Data Drift
Univariate Drift Detection
Why Perform Univariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Multivariate Data Drift Detection
Why Perform Multivariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
Insights
What Next
Adjusting Plots
How It Works
Metadata extraction
Naming conventions
Common metadata columns
Binary classification columns
Multiclass classification columns
Feature type detection
Data Reconstruction with PCA
Limitations of Univariate Drift Detection
“Butterfly” Dataset
Data Reconstruction with PCA
Understanding Reconstruction Error with PCA
Reconstruction Error with PCA on the butterfly dataset
Confidence-based Performance Estimation (CBPE)
CBPE algorithm
Binary classification
Multiclass Classification
Assumptions and Limitations
Appendix: Probability calibration
Chunking data
Chunking considerations
Different periods within one chunk
Underpopulated chunks
Not enough chunks
Minimum chunk size
Minimum Chunk Size for Performance Estimation and Performance Monitoring
Minimum Chunk Size for Multivariate Drift
Minimum Chunk for Univariate Drift
Examples
Binary Classification: California Housing Dataset
Load and prepare data
Performance Estimation
Comparison with the actual performance
Drift detection
Troubleshooting
Dealing with a MissingMetadataException
The problem
The solution
Related reads
Example Datasets
Synthetic Binary Classification Dataset
Problem Description
Dataset Description
Metadata Extraction
Synthetic Multiclass Classification Dataset
Problem Description
Dataset Description
Metadata Extraction
California Housing Dataset
Modifying California Housing Dataset
Enriching the data
Training a Machine Learning Model
Meeting NannyML Data Requirements
Glossary
API reference
nannyml package
Subpackages
nannyml.datasets package
nannyml.drift package
nannyml.metadata package
nannyml.performance_calculation package
nannyml.performance_estimation package
nannyml.plots package
Submodules
nannyml.calibration module
nannyml.chunk module
nannyml.exceptions module
nannyml.preprocessing module
Module contents
Contributing
Spread the word
Be a part of the team
Contribute to the codebase
Get started coding
Pull Request Guidelines
Tips
NannyML
»
Index
Edit on GitHub
Index
_
|
A
|
B
|
C
|
D
|
E
|
F
|
G
|
I
|
K
|
L
|
M
|
N
|
O
|
P
|
R
|
S
|
T
|
U
|
V
_
__eq__() (nannyml.performance_calculation.metrics.Metric method)
__len__() (nannyml.chunk.Chunk method)
__new__() (nannyml.performance_estimation.confidence_based.cbpe.CBPE static method)
__repr__() (nannyml.chunk.Chunk method)
(nannyml.drift.model_inputs.univariate.statistical.results.UnivariateDriftResult method)
(nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.feature.Feature method)
__str__() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.feature.Feature method)
A
Alert
ALERT_COLUMN_SUFFIX (nannyml.drift.ranking.AlertCountRanking attribute)
AlertCountRanking (class in nannyml.drift.ranking)
B
BinaryClassificationAccuracy (class in nannyml.performance_calculation.metrics)
BinaryClassificationAUROC (class in nannyml.performance_calculation.metrics)
BinaryClassificationF1 (class in nannyml.performance_calculation.metrics)
BinaryClassificationMetadata (class in nannyml.metadata.binary_classification)
BinaryClassificationPrecision (class in nannyml.performance_calculation.metrics)
BinaryClassificationRecall (class in nannyml.performance_calculation.metrics)
BinaryClassificationSpecificity (class in nannyml.performance_calculation.metrics)
BLUE_SKY_CRAYOLA (nannyml.plots.colors.Colors attribute)
Butterfly dataset
by() (nannyml.drift.ranking.Ranker class method)
C
calculate() (nannyml.drift.base.DriftCalculator method)
(nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator.DataReconstructionDriftCalculator method)
(nannyml.drift.model_inputs.univariate.statistical.calculator.UnivariateStatisticalDriftCalculator method)
(nannyml.drift.target.target_distribution.calculator.TargetDistributionCalculator method)
(nannyml.performance_calculation.calculator.PerformanceCalculator method)
(nannyml.performance_calculation.metrics.Metric method)
CalculatorException
CalculatorNotFittedException
calibrate() (nannyml.calibration.Calibrator method)
(nannyml.calibration.IsotonicCalibrator method)
(nannyml.calibration.NoopCalibrator method)
Calibrator (class in nannyml.calibration)
CalibratorFactory (class in nannyml.calibration)
CATEGORICAL (nannyml.metadata.feature.FeatureType attribute)
categorical_features (nannyml.metadata.base.ModelMetadata property)
CBPE (class in nannyml.performance_estimation.confidence_based.cbpe)
(Confidence-Based Performance Estimation)
CBPEPerformanceEstimatorResult (class in nannyml.performance_estimation.confidence_based.results)
Chi Squared test
Chunk (class in nannyml.chunk)
Chunker (class in nannyml.chunk)
ChunkerException
class_labels() (nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
CLASSIFICATION_BINARY (nannyml.metadata.base.ModelType attribute)
CLASSIFICATION_MULTICLASS (nannyml.metadata.base.ModelType attribute)
Colors (class in nannyml.plots.colors)
Concept Drift
CONTINUOUS (nannyml.metadata.feature.FeatureType attribute)
continuous_features (nannyml.metadata.base.ModelMetadata property)
CountBasedChunker (class in nannyml.chunk)
create() (nannyml.calibration.CalibratorFactory class method)
(nannyml.metadata.extraction.ModelMetadataFactory class method)
(nannyml.performance_calculation.metrics.MetricFactory class method)
D
Data Chunk
Data Drift
Data Period
DataReconstructionDriftCalculator (class in nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator)
DataReconstructionDriftCalculatorResult (class in nannyml.drift.model_inputs.multivariate.data_reconstruction.results)
DEFAULT_CHUNK_COUNT (nannyml.chunk.DefaultChunker attribute)
DefaultChunker (class in nannyml.chunk)
DriftCalculator (class in nannyml.drift.base)
DriftResult (class in nannyml.drift.base)
E
enrich() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata method)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
(nannyml.metadata.regression.RegressionMetadata method)
estimate() (nannyml.performance_estimation.base.PerformanceEstimator method)
(nannyml.performance_estimation.confidence_based.cbpe.CBPE method)
Estimated Performance
extract() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata method)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
(nannyml.metadata.regression.RegressionMetadata method)
extract_metadata() (in module nannyml.metadata.extraction)
F
Feature
(class in nannyml.metadata.feature)
feature() (nannyml.metadata.base.ModelMetadata method)
FeatureType (class in nannyml.metadata.feature)
fit() (nannyml.calibration.Calibrator method)
(nannyml.calibration.IsotonicCalibrator method)
(nannyml.calibration.NoopCalibrator method)
(nannyml.drift.base.DriftCalculator method)
(nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator.DataReconstructionDriftCalculator method)
(nannyml.drift.model_inputs.univariate.statistical.calculator.UnivariateStatisticalDriftCalculator method)
(nannyml.drift.target.target_distribution.calculator.TargetDistributionCalculator method)
(nannyml.performance_calculation.calculator.PerformanceCalculator method)
(nannyml.performance_calculation.metrics.Metric method)
(nannyml.performance_estimation.base.PerformanceEstimator method)
(nannyml.performance_estimation.confidence_based.cbpe.CBPE method)
G
GRAY (nannyml.plots.colors.Colors attribute)
GRAY_DARK (nannyml.plots.colors.Colors attribute)
GREEN_SEA (nannyml.plots.colors.Colors attribute)
Ground truth
I
Identifier
Imputation
INDIGO_PERSIAN (nannyml.plots.colors.Colors attribute)
InvalidArgumentsException
InvalidReferenceDataException
is_complete() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata method)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
(nannyml.metadata.regression.RegressionMetadata method)
IsotonicCalibrator (class in nannyml.calibration)
K
Kolmogorov-Smirnov test
L
Latent space
LIGHT_GRAY (nannyml.plots.colors.Colors attribute)
load_csv_file_to_df() (in module nannyml.datasets.datasets)
load_modified_california_housing_dataset() (in module nannyml.datasets.datasets)
load_synthetic_binary_classification_dataset() (in module nannyml.datasets.datasets)
load_synthetic_multiclass_classification_dataset() (in module nannyml.datasets.datasets)
M
mapping (nannyml.metadata.extraction.ModelMetadataFactory attribute)
metadata_columns (nannyml.metadata.base.ModelMetadata property)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata property)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata property)
(nannyml.metadata.regression.RegressionMetadata property)
Metric (class in nannyml.performance_calculation.metrics)
MetricFactory (class in nannyml.performance_calculation.metrics)
minimum_chunk_size() (nannyml.performance_calculation.metrics.Metric method)
MissingMetadataException
Model inputs
Model Metadata
Model outputs
Model predictions
ModelMetadata (class in nannyml.metadata.base)
ModelMetadataFactory (class in nannyml.metadata.extraction)
ModelType (class in nannyml.metadata.base)
module
nannyml
nannyml.calibration
nannyml.chunk
nannyml.datasets
nannyml.datasets.data
nannyml.datasets.datasets
nannyml.drift
nannyml.drift.base
nannyml.drift.model_inputs
nannyml.drift.model_inputs.multivariate
nannyml.drift.model_inputs.multivariate.data_reconstruction
nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator
nannyml.drift.model_inputs.multivariate.data_reconstruction.results
nannyml.drift.model_inputs.univariate
nannyml.drift.model_inputs.univariate.statistical
nannyml.drift.model_inputs.univariate.statistical.calculator
nannyml.drift.model_inputs.univariate.statistical.results
nannyml.drift.model_outputs
nannyml.drift.ranking
nannyml.drift.target
nannyml.drift.target.target_distribution
nannyml.drift.target.target_distribution.calculator
nannyml.drift.target.target_distribution.result
nannyml.exceptions
nannyml.metadata
nannyml.metadata.base
nannyml.metadata.binary_classification
nannyml.metadata.extraction
nannyml.metadata.feature
nannyml.metadata.multiclass_classification
nannyml.metadata.regression
nannyml.performance_calculation
nannyml.performance_calculation.calculator
nannyml.performance_calculation.metrics
nannyml.performance_calculation.result
nannyml.performance_estimation
nannyml.performance_estimation.base
nannyml.performance_estimation.confidence_based
nannyml.performance_estimation.confidence_based.cbpe
nannyml.performance_estimation.confidence_based.results
nannyml.plots
nannyml.plots.colors
nannyml.preprocessing
MulticlassClassificationAccuracy (class in nannyml.performance_calculation.metrics)
MulticlassClassificationAUROC (class in nannyml.performance_calculation.metrics)
MulticlassClassificationF1 (class in nannyml.performance_calculation.metrics)
MulticlassClassificationMetadata (class in nannyml.metadata.multiclass_classification)
MulticlassClassificationPrecision (class in nannyml.performance_calculation.metrics)
MulticlassClassificationRecall (class in nannyml.performance_calculation.metrics)
MulticlassClassificationSpecificity (class in nannyml.performance_calculation.metrics)
Multivariate Drift Detection
N
nannyml
module
nannyml.calibration
module
nannyml.chunk
module
nannyml.datasets
module
nannyml.datasets.data
module
nannyml.datasets.datasets
module
nannyml.drift
module
nannyml.drift.base
module
nannyml.drift.model_inputs
module
nannyml.drift.model_inputs.multivariate
module
nannyml.drift.model_inputs.multivariate.data_reconstruction
module
nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator
module
nannyml.drift.model_inputs.multivariate.data_reconstruction.results
module
nannyml.drift.model_inputs.univariate
module
nannyml.drift.model_inputs.univariate.statistical
module
nannyml.drift.model_inputs.univariate.statistical.calculator
module
nannyml.drift.model_inputs.univariate.statistical.results
module
nannyml.drift.model_outputs
module
nannyml.drift.ranking
module
nannyml.drift.target
module
nannyml.drift.target.target_distribution
module
nannyml.drift.target.target_distribution.calculator
module
nannyml.drift.target.target_distribution.result
module
nannyml.exceptions
module
nannyml.metadata
module
nannyml.metadata.base
module
nannyml.metadata.binary_classification
module
nannyml.metadata.extraction
module
nannyml.metadata.feature
module
nannyml.metadata.multiclass_classification
module
nannyml.metadata.regression
module
nannyml.performance_calculation
module
nannyml.performance_calculation.calculator
module
nannyml.performance_calculation.metrics
module
nannyml.performance_calculation.result
module
nannyml.performance_estimation
module
nannyml.performance_estimation.base
module
nannyml.performance_estimation.confidence_based
module
nannyml.performance_estimation.confidence_based.cbpe
module
nannyml.performance_estimation.confidence_based.results
module
nannyml.plots
module
nannyml.plots.colors
module
nannyml.preprocessing
module
needs_calibration() (in module nannyml.calibration)
NoopCalibrator (class in nannyml.calibration)
NotFittedException
O
ORDINAL (nannyml.metadata.feature.FeatureType attribute)
P
parse() (nannyml.metadata.base.ModelType static method)
Partition Column
partition_column_name (nannyml.metadata.base.ModelMetadata property)
PCA
Performance Estimation
PerformanceCalculator (class in nannyml.performance_calculation.calculator)
PerformanceCalculatorResult (class in nannyml.performance_calculation.result)
PerformanceEstimator (class in nannyml.performance_estimation.base)
PerformanceEstimatorResult (class in nannyml.performance_estimation.base)
PeriodBasedChunker (class in nannyml.chunk)
plot() (nannyml.drift.base.DriftResult method)
(nannyml.drift.model_inputs.multivariate.data_reconstruction.results.DataReconstructionDriftCalculatorResult method)
(nannyml.drift.model_inputs.univariate.statistical.results.UnivariateDriftResult method)
(nannyml.drift.target.target_distribution.result.TargetDistributionResult method)
(nannyml.performance_calculation.result.PerformanceCalculatorResult method)
(nannyml.performance_estimation.base.PerformanceEstimatorResult method)
(nannyml.performance_estimation.confidence_based.results.CBPEPerformanceEstimatorResult method)
Predicted labels
Predicted probabilities
Predicted scores
predicted_class_probability_metadata_columns() (nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
predicted_probabilities_column_names (nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata property)
predicted_probability_column_name (nannyml.metadata.binary_classification.BinaryClassificationMetadata property)
prediction_column_name (nannyml.metadata.binary_classification.BinaryClassificationMetadata property)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata property)
(nannyml.metadata.regression.RegressionMetadata property)
Predictions
preprocess() (in module nannyml.preprocessing)
print() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.feature.Feature method)
R
rank() (nannyml.drift.ranking.AlertCountRanking method)
(nannyml.drift.ranking.Ranking method)
Ranker (class in nannyml.drift.ranking)
Ranking (class in nannyml.drift.ranking)
Realized Performance
Reconstruction Error
RED_IMPERIAL (nannyml.plots.colors.Colors attribute)
register_calibrator() (nannyml.calibration.CalibratorFactory class method)
register_ranking() (nannyml.drift.ranking.Ranker class method)
REGRESSION (nannyml.metadata.base.ModelType attribute)
RegressionMetadata (class in nannyml.metadata.regression)
S
SAFFRON (nannyml.plots.colors.Colors attribute)
SizeBasedChunker (class in nannyml.chunk)
split() (nannyml.chunk.Chunker method)
T
Target
target_column_name (nannyml.metadata.base.ModelMetadata property)
TargetDistributionCalculator (class in nannyml.drift.target.target_distribution.calculator)
TargetDistributionResult (class in nannyml.drift.target.target_distribution.result)
Timestamp
timestamp_column_name (nannyml.metadata.base.ModelMetadata property)
to_df() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata method)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
(nannyml.metadata.regression.RegressionMetadata method)
to_dict() (nannyml.metadata.base.ModelMetadata method)
(nannyml.metadata.binary_classification.BinaryClassificationMetadata method)
(nannyml.metadata.feature.Feature method)
(nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
(nannyml.metadata.regression.RegressionMetadata method)
U
Univariate Drift Detection
UnivariateDriftResult (class in nannyml.drift.model_inputs.univariate.statistical.results)
UnivariateStatisticalDriftCalculator (class in nannyml.drift.model_inputs.univariate.statistical.calculator)
UNKNOWN (nannyml.metadata.feature.FeatureType attribute)
V
validate_predicted_class_labels_in_class_probability_mapping() (nannyml.metadata.multiclass_classification.MulticlassClassificationMetadata method)
Read the Docs
v: main
Versions
latest
stable
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.1
v0.2.0
main
Downloads
On Read the Docs
Project Home
Builds