NannyML
0.13.1
Contents:
Introduction
What is NannyML?
Key features
➖ Performance Estimation and Calculation
➖ Business Value Estimation and Calculation
➖ Data Quality
➖ Multivariate Drift Detection
➖ Univariate Drift Detection
➖ Custom Thresholds
Next steps
Get early access to NannyML Web App
Installing NannyML
Extras
Quickstart
What is NannyML?
Exemplary Workflow with NannyML
Loading data
Estimating Performance without Targets
Investigating Data Distribution Shifts
Comparing Estimated with Realized Performance when Targets Arrive
What’s next?
Tutorials
Data requirements
Data Periods
Reference Period
Analysis Period
Columns
Timestamp
Target
Features
Model Output columns
Predicted class probabilities
Prediction class labels
NannyML Functionality Requirements
What’s next
Estimating Performance
Why Estimate Performance
Estimating Performance for Binary Classification
Estimating Standard Performance Metrics for Binary Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Confusion Matrix Elements for Binary Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Business Value for Binary Classification
Just The Code
Walkthrough
Insights
What’s next
Creating and Estimating a Custom Binary Classification Metric
Just the Code
Walkthrough
Insights
What’s next
Estimating Performance for Multiclass Classification
Estimating Performance for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Confusion Matrix Elements for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Business Value for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Performance for Regression
Just The Code
Walkthrough
Insights
What’s next
Monitoring Realized Performance
Why Monitor Realized Performance
Monitoring Realized Performance for Binary Classification
Calculating Standard Performance Metrics for Binary Classification
Just The Code
Walkthrough
Insights
What’s Next
Calculating Confusion Matrix Elements for Binary Classification
Just The Code
Walkthrough
Insights
What’s Next
Calculating Business Value for Binary Classification
Just The Code
Walkthrough
Insights
What’s Next
Monitoring Realized Performance for Multiclass Classification
Calculating Standard Performance Metrics for Multiclass Classification
Just The Code
Walkthrough
Insights
What Next
Calculating Confusion Matrix Elements for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s Next
Calculating Business Value for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s Next
Monitoring Realized Performance for Regression
Just The Code
Walkthrough
Insights
What Next
Comparing Estimated and Realized Performance
Just the code
Walkthrough
Estimating performance without targets
Comparing to realized performance
Detecting Data Drift
Univariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Multivariate Drift Detection
Data Reconstruction with PCA
Just The Code
Walkthrough
Insights
What Next
Domain Classifier
Just The Code
Walkthrough
Insights
What Next
Ranking
Just The Code
Walkthrough
Alert Count Ranking
Correlation Ranking
Insights
What’s Next
Data Quality Checks
Missing Values Detection
Just The Code
Walkthrough
Insights
What Next
Unseen Values Detection
Just The Code
Walkthrough
Insights
What Next
Summary Statistics
Summation
Just The Code
Walkthrough
Insights
What Next
Average
Just The Code
Walkthrough
Insights
What Next
Standard Deviation
Just The Code
Walkthrough
Insights
What Next
Median
Just The Code
Walkthrough
Insights
What Next
Rows Count
Just The Code
Walkthrough
Insights
What Next
Storing and loading calculators
Just the code
Walkthrough
What’s Next
Working with results
What are NannyML Results?
Just the code
Walkthrough
The data structure
Filtering
Plotting
Comparing
Exporting
Adjusting Plots
Chunking
Why do we need chunks?
Walkthrough on creating chunks
Time-based chunking
Size-based chunking
Number-based chunking
Automatic chunking
Customize chunk behavior
Chunks on plots with results
Thresholds
Just the code
Walkthrough
Constant thresholds
Standard deviation thresholds
Setting custom thresholds for calculators and estimators
Default thresholds
What’s next?
How It Works
Estimation of Performance of the Monitored Model
Confidence-based Performance Estimation (CBPE)
The Intuition
Implementation details
Binary classification
Multiclass Classification
Assumptions and Limitations
Appendix: Probability calibration
Direct Loss Estimation (DLE)
The Intuition
Implementation details
Assumptions and limitations
Other Approaches to Estimate Performance of Regression Models
Bayesian approaches
Conformalized Quantile Regression
Conclusions from Bayesian and Conformalized Quantile Regression approaches
Business Value Estimation and Calculation
Introduction to Business Value
Business Value Formula
Calculation of Business Value For Classification
Estimation of Business Value For Classification
Normalization
Presenting Univariate Drift Detection Methods
Methods for Continuous Features
Kolmogorov-Smirnov Test
Jensen-Shannon Distance
Wasserstein Distance
Hellinger Distance
Methods for Categorical Variables
Chi-squared Test
Jensen-Shannon Distance
Hellinger Distance
L-Infinity Distance
Choosing Univariate Drift Detection Methods
Comparison of Methods for Continuous Variables
Shifting the Mean of the Analysis Data Set
Shifting the Standard Deviation of the Analysis Data Set
Tradeoffs of The Kolmogorov-Smirnov Statistic
Tradeoffs of Jensen-Shannon Distance and Hellinger Distance
Experiment 1
Experiment 2
Tradeoffs of Wasserstein Distance
Experiment 1
Experiment 2
Experiment 3
Comparison of Methods for Categorical Variables
Sensitivity to Sample Size of Different Drift Measures
Behavior When a Category Slowly Disappears
Behavior When Observations from a New Category Occur
Effect of Sample Size on Different Drift Measures
Effect of the Number of Categories on Different Drift Measures
Comparison of Drift Methods on Data Sets with Many Categories
Results Summary (TLDR)
Methods for Continuous Variables
Methods For Categorical Variables
Ranking
Alert Count Ranking
Correlation Ranking
Multivariate Drift Detection
Limitations of Univariate Drift Detection
“Butterfly” Dataset
Data Reconstruction with PCA
Understanding Reconstruction Error with PCA
Reconstruction Error with PCA on the butterfly dataset
Domain Classifier
Understanding Domain Classifier
Domain Classifier on the butterfly dataset
Chunking Considerations
Not Enough Chunks
Not Enough Observations in Chunk
Impact of Chunk Size on Reliability of Results
Calculating Sampling Error
Defining Sampling Error from Standard Error of the Mean
Sampling Error Estimation and Interpretation for NannyML features
Performance Estimation
Performance Monitoring
Multivariate Drift Detection with PCA
Univariate Drift Detection
Summary Statistics
Average
Summation
Standard Deviation
Median
Assumptions and Limitations
Thresholds
Threshold basics
Constant thresholds
Standard deviation thresholds
Examples
Binary Classification: California Housing Dataset
Load and prepare data
Performance Estimation
Comparison with the actual performance
Drift detection
Full Monitoring Workflow - Regression: NYC Green Taxi Dataset
Import libraries
Load the data
Preprocessing the data
Exploring the training data
Training a model
Evaluating the model
Deploying the model
Analysing ML model performance in production
Estimating the model’s performance
Detecting multivariate data drift
Detecting univariate data drift
Bonus: Comparing realized and estimated performance
Conclusion
Example Datasets
US Census Employment dataset
Data Source
Dataset Description
Preparing Data for NannyML
Fetching the Data
Defining Partitions and Preprocessing
Developing ML Model and Making Predictions
Splitting and Storing the Data
Appendix: Feature description
References
Synthetic Binary Classification Car Loan Dataset
Problem Description
Dataset Description
Data Quality Version
Synthetic Multiclass Classification Dataset
Problem Description
Dataset Description
Synthetic Regression Dataset
Problem Description
Dataset Description
California Housing Dataset
Modifying California Housing Dataset
Enriching the data
Training a Machine Learning Model
Meeting NannyML Data Requirements
Titanic Dataset
Problem Description
Dataset Description
Glossary
Command Line Interface (CLI)
Running the CLI
Installation
Configuration
Configuration file
Locations
Format
Input section
Output section
Writing to filesystem
Writing to a pickle file
Writing to a relational database
Column mapping section
Store section
Chunker section
Scheduling section
Standalone parameters section
Templating paths
Examples
Command overview
run
Syntax
Options
Example
Usage logging in NannyML
TLDR
What do we mean by usage statistics?
What about personal data
What about my dataset?
Why are we doing this?
Improving NannyML and prioritizing new features
Surviving as a company
How usage logging works
To opt in
or
not to opt in
, that’s the question
How to disable usage logging
Setting the environment variable
Providing a
.env
file
Turning off user analytics in code
API reference
nannyml package
Subpackages
nannyml.cli package
Submodules
Module contents
nannyml.data_quality package
Subpackages
Module contents
nannyml.datasets package
Subpackages
Submodules
Module contents
nannyml.distribution package
Subpackages
Module contents
nannyml.drift package
Subpackages
Submodules
Module contents
nannyml.io package
Subpackages
Submodules
Module contents
nannyml.performance_calculation package
Subpackages
Submodules
Module contents
nannyml.performance_estimation package
Subpackages
Module contents
nannyml.plots package
Subpackages
Submodules
Module contents
nannyml.sampling_error package
Submodules
Module contents
nannyml.stats package
Subpackages
Module contents
Submodules
nannyml.analytics module
nannyml.base module
nannyml.calibration module
nannyml.chunk module
nannyml.config module
nannyml.exceptions module
nannyml.runner module
nannyml.thresholds module
nannyml.usage_logging module
Module contents
Contributing
Spread the word
Be a part of the team
Contribute to the codebase
Get started coding
Pull Request Guidelines
Tips
NannyML
nannyml
nannyml package
nannyml.data_quality package
nannyml.data_quality.range package
View page source
nannyml.data_quality.range package
Submodules
nannyml.data_quality.range.calculator module
nannyml.data_quality.range.result module
Module contents
Package containing the Data Quality Calculators implementation.