NannyML
main
Contents:
Quickstart
What is NannyML?
Installing NannyML
Contents of the quickstart
Just the code
Walkthrough
Estimating Performance without Targets
Detecting Data Drift
Insights
What next
Tutorials
Setting Up
Data requirements
Data Periods
Common columns
Binary classification columns
Multiclass classification columns
Regression columns
What next
Providing metadata
Why is providing metadata required?
Metadata for binary classification
Metadata for multiclass classification
Metadata for regression
Insights
What next
Chunking
Why do we need chunks?
Walkthrough on creating chunks
Chunks on plots with results
Estimating Performance
Estimating Performance for Binary Classification
Why Perform Performance Estimation
Insights
What’s next
Estimating Performance for Multiclass Classification
Why Perform Performance Estimation
Insights
What’s next
Estimating Performance for Multiclass Classification
Monitoring Realized Performance
Monitoring Realized Performance for Binary Classification
Why Monitor Realized Performance
Insights
What Next
Monitoring Realized Performance for Multiclass Classification
Why Monitor Realized Performance
Insights
What Next
Monitoring Realized Performance for Regression
Comparing Estimated and Realized Performance
Detecting Data Drift
Univariate Drift Detection
Why Perform Univariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Multivariate Data Drift Detection
Why Perform Multivariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
Insights
What Next
Adjusting Plots
How It Works
Metadata extraction
Naming conventions
Common metadata columns
Binary classification columns
Multiclass classification columns
Feature type detection
Data Reconstruction with PCA
Limitations of Univariate Drift Detection
“Butterfly” Dataset
Data Reconstruction with PCA
Understanding Reconstruction Error with PCA
Reconstruction Error with PCA on the butterfly dataset
Confidence-based Performance Estimation (CBPE)
CBPE algorithm
Binary classification
Multiclass Classification
Assumptions and Limitations
Appendix: Probability calibration
Chunking data
Chunking considerations
Different periods within one chunk
Underpopulated chunks
Not enough chunks
Minimum chunk size
Minimum Chunk Size for Performance Estimation and Performance Monitoring
Minimum Chunk Size for Multivariate Drift
Minimum Chunk for Univariate Drift
Examples
Binary Classification: California Housing Dataset
Load and prepare data
Performance Estimation
Comparison with the actual performance
Drift detection
Troubleshooting
Dealing with a MissingMetadataException
The problem
The solution
Related reads
Example Datasets
Synthetic Binary Classification Dataset
Problem Description
Dataset Description
Metadata Extraction
Synthetic Multiclass Classification Dataset
Problem Description
Dataset Description
Metadata Extraction
California Housing Dataset
Modifying California Housing Dataset
Enriching the data
Training a Machine Learning Model
Meeting NannyML Data Requirements
Glossary
API reference
nannyml package
Subpackages
nannyml.datasets package
nannyml.drift package
nannyml.metadata package
nannyml.performance_calculation package
nannyml.performance_estimation package
nannyml.plots package
Submodules
nannyml.calibration module
nannyml.chunk module
nannyml.exceptions module
nannyml.preprocessing module
Module contents
Contributing
Spread the word
Be a part of the team
Contribute to the codebase
Get started coding
Pull Request Guidelines
Tips
NannyML
»
Tutorials
Edit on GitHub
Tutorials
Setting Up
Data requirements
Providing metadata
Chunking
Estimating Performance
Estimating Performance for Binary Classification
Estimating Performance for Multiclass Classification
Estimating Performance for Multiclass Classification
Monitoring Realized Performance
Monitoring Realized Performance for Binary Classification
Monitoring Realized Performance for Multiclass Classification
Monitoring Realized Performance for Regression
Comparing Estimated and Realized Performance
Detecting Data Drift
Univariate Drift Detection
Multivariate Data Drift Detection
Drift Detection for Model Outputs
Drift Detection for Model Targets
Adjusting Plots
Read the Docs
v: main
Versions
latest
stable
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.1
v0.2.0
main
Downloads
On Read the Docs
Project Home
Builds