NannyML
v0.5.3
Contents:
Quickstart
What is NannyML?
Installing NannyML
Contents of the quickstart
Just the code
Walkthrough
Estimating Performance without Targets
Detecting Data Drift
Insights
What next
Tutorials
Data requirements
Data Periods
Reference Period
Analysis Period
Columns
Timestamp
Target
Features
Model Output columns
Predicted class probabilities
Prediction class labels
NannyML Functionality Requirements
What next
Estimating Performance
Why Perform Performance Estimation
Estimating Performance for Binary Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Performance for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Performance for Regression
Monitoring Realized Performance
Why Monitoring Realized Performance
Monitoring Realized Performance for Binary Classification
Just The Code
Walkthrough
Insights
What Next
Monitoring Realized Performance for Multiclass Classification
Just The Code
Walkthrough
Insights
What Next
Monitoring Realized Performance for Regression
Comparing Estimated and Realized Performance
Detecting Data Drift
Univariate Drift Detection
Why Perform Univariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Multivariate Data Drift Detection
Why Perform Multivariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Outputs
Drift Detection for Binary Classification Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Multiclass Classification Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
What Next
Drift Detection for Model Targets
Drift Detection for Binary Classification Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Multiclass Classification Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
What Next
Adjusting Plots
Chunking
Why do we need chunks?
Walkthrough on creating chunks
Time-based chunking
Size-based chunking
Number-based chunking
Automatic chunking
Chunks on plots with results
How It Works
Confidence-based Performance Estimation (CBPE)
CBPE algorithm
Binary classification
Multiclass Classification
Assumptions and Limitations
Appendix: Probability calibration
Data Reconstruction with PCA
Limitations of Univariate Drift Detection
“Butterfly” Dataset
Data Reconstruction with PCA
Understanding Reconstruction Error with PCA
Reconstruction Error with PCA on the butterfly dataset
Chunking Considerations
Not Enough Chunks
Not Enough Observations in Chunk
Impact of Chunk Size on Reliability of Results
Estimation of Standard Error
Adapting Standard Error of the Mean Formula
Standard Error Estimation and Interpretation for NannyML features
Performance Estimation
Performance Monitoring
Multivariate Drift Detection with PCA
Univariate Drift Detection
Assumptions and Limitations
Examples
Binary Classification: California Housing Dataset
Load and prepare data
Performance Estimation
Comparison with the actual performance
Drift detection
Example Datasets
Synthetic Binary Classification Dataset
Problem Description
Dataset Description
Synthetic Multiclass Classification Dataset
Problem Description
Dataset Description
California Housing Dataset
Modifying California Housing Dataset
Enriching the data
Training a Machine Learning Model
Meeting NannyML Data Requirements
Glossary
CLI nml
Running the CLI
Installation
Configuration
Configuration file
Locations
Format
Input section
Output section
Column mapping section
Chunker section
Standalone parameters section
Templating paths
Examples
Command overview
run
Syntax
Options
Example
API reference
nannyml package
Subpackages
nannyml.cli package
Submodules
Module contents
nannyml.datasets package
Subpackages
Submodules
Module contents
nannyml.drift package
Subpackages
Submodules
Module contents
nannyml.io package
Submodules
Module contents
nannyml.performance_calculation package
Submodules
Module contents
nannyml.performance_estimation package
Subpackages
Module contents
nannyml.plots package
Submodules
Module contents
nannyml.sampling_error package
Submodules
Module contents
Submodules
nannyml.base module
nannyml.calibration module
nannyml.chunk module
nannyml.config module
nannyml.exceptions module
nannyml.runner module
Module contents
Contributing
Spread the word
Be a part of the team
Contribute to the codebase
Get started coding
Pull Request Guidelines
Tips
NannyML
»
Tutorials
»
Estimating Performance
»
Estimating Performance for Regression
Edit on GitHub
Estimating Performance for Regression
Warning
Performance estimation does not support regression use cases yet.
Read the Docs
v: v0.5.3
Versions
latest
stable
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.1
v0.2.0
main
Downloads
On Read the Docs
Project Home
Builds