NannyML
v0.6.1

Contents:

  • Quickstart
    • What is NannyML?
    • Installing NannyML
    • Contents of the quickstart
    • Just the code
    • Walkthrough
      • Estimating Performance without Targets
      • Detecting Data Drift
    • Insights
    • What next
  • Tutorials
    • Data requirements
      • Data Periods
        • Reference Period
        • Analysis Period
      • Columns
        • Timestamp
        • Target
        • Features
      • Model Output columns
        • Predicted class probabilities
        • Prediction class labels
      • NannyML Functionality Requirements
      • What next
    • Estimating Performance
      • Why Perform Performance Estimation
      • Estimating Performance for Binary Classification
        • Just The Code
        • Walkthrough
        • Insights
        • What’s next
      • Estimating Performance for Multiclass Classification
        • Just The Code
        • Walkthrough
        • Insights
        • What’s next
      • Estimating Performance for Regression
        • Just The Code
        • Walkthrough
        • Insights
        • What’s next
    • Monitoring Realized Performance
      • Why Monitoring Realized Performance
      • Monitoring Realized Performance for Binary Classification
        • Just The Code
        • Walkthrough
        • Insights
        • What Next
      • Monitoring Realized Performance for Multiclass Classification
        • Just The Code
        • Walkthrough
        • Insights
        • What Next
      • Monitoring Realized Performance for Regression
        • Just The Code
        • Walkthrough
        • Insights
        • What Next
    • Comparing Estimated and Realized Performance
    • Detecting Data Drift
      • Univariate Drift Detection
        • Why Perform Univariate Drift Detection
        • Just The Code
        • Walkthrough
        • Insights
        • What Next
      • Multivariate Data Drift Detection
        • Why Perform Multivariate Drift Detection
        • Just The Code
        • Walkthrough
        • Insights
        • What Next
      • Drift Detection for Model Outputs
        • Drift Detection for Binary Classification Model Outputs
          • Why Perform Drift Detection for Model Outputs
          • Just The Code
          • Walkthrough
          • Insights
          • What Next
        • Drift Detection for Multiclass Classification Model Outputs
          • Why Perform Drift Detection for Model Outputs
          • Just The Code
          • Walkthrough
          • What Next
        • Drift Detection for Regression Model Outputs
          • Why Perform Drift Detection for Model Outputs
          • Just The Code
          • Walkthrough
          • Insights
          • What Next
      • Drift Detection for Model Targets
        • Drift Detection for Binary Classification Model Targets
          • Why Perform Drift Detection for Model Targets
          • Just The Code
          • Walkthrough
          • Insights
          • What Next
        • Drift Detection for Multiclass Classification Model Targets
          • Why Perform Drift Detection for Model Targets
          • Just The Code
          • Walkthrough
          • What Next
        • Drift Detection for Regression Model Targets
          • Why Perform Drift Detection for Model Targets
          • Just The Code
          • Walkthrough
          • Insights
          • What Next
    • Adjusting Plots
    • Chunking
      • Why do we need chunks?
      • Walkthrough on creating chunks
        • Time-based chunking
        • Size-based chunking
        • Number-based chunking
        • Automatic chunking
      • Chunks on plots with results
  • How It Works
    • Estimation of Performance of the Monitored Model
      • Confidence-based Performance Estimation (CBPE)
        • The Intuition
        • Implementation details
          • Binary classification
          • Multiclass Classification
        • Assumptions and Limitations
        • Appendix: Probability calibration
      • Direct Loss Estimation (DLE)
        • The Intuition
        • Implementation details
        • Assumptions and limitations
      • Other Approaches to Estimate Performance of Regression Models
        • Bayesian approaches
        • Conformalized Quantile Regression
        • Conclusions from Bayesian and Conformalized Quantile Regression approaches
    • Data Reconstruction with PCA
      • Limitations of Univariate Drift Detection
        • “Butterfly” Dataset
      • Data Reconstruction with PCA
      • Understanding Reconstruction Error with PCA
        • Reconstruction Error with PCA on the butterfly dataset
    • Chunking Considerations
      • Not Enough Chunks
      • Not Enough Observations in Chunk
      • Impact of Chunk Size on Reliability of Results
    • Calculating Sampling Error
      • Defining Sampling Error from Standard Error of the Mean
      • Sampling Error Estimation and Interpretation for NannyML features
        • Performance Estimation
        • Performance Monitoring
        • Multivariate Drift Detection with PCA
        • Univariate Drift Detection
      • Assumptions and Limitations
  • Examples
    • Binary Classification: California Housing Dataset
      • Load and prepare data
      • Performance Estimation
      • Comparison with the actual performance
      • Drift detection
  • Example Datasets
    • Synthetic Binary Classification Dataset
      • Problem Description
      • Dataset Description
    • Synthetic Multiclass Classification Dataset
      • Problem Description
      • Dataset Description
    • California Housing Dataset
      • Modifying California Housing Dataset
      • Enriching the data
      • Training a Machine Learning Model
      • Meeting NannyML Data Requirements
    • Synthetic Regression Dataset
      • Problem Description
      • Dataset Description
  • Glossary
  • Command Line Interface (CLI)
    • Running the CLI
      • Installation
      • Configuration
    • Configuration file
      • Locations
      • Format
        • Input section
        • Output section
        • Column mapping section
        • Chunker section
        • Standalone parameters section
      • Templating paths
      • Examples
    • Command overview
      • run
        • Syntax
        • Options
        • Example
  • API reference
    • nannyml package
      • Subpackages
        • nannyml.cli package
          • Submodules
          • Module contents
        • nannyml.datasets package
          • Subpackages
          • Submodules
          • Module contents
        • nannyml.drift package
          • Subpackages
          • Submodules
          • Module contents
        • nannyml.io package
          • Submodules
          • Module contents
        • nannyml.performance_calculation package
          • Subpackages
          • Submodules
          • Module contents
        • nannyml.performance_estimation package
          • Subpackages
          • Module contents
        • nannyml.plots package
          • Submodules
          • Module contents
        • nannyml.sampling_error package
          • Submodules
          • Module contents
      • Submodules
        • nannyml.base module
        • nannyml.calibration module
        • nannyml.chunk module
        • nannyml.config module
        • nannyml.exceptions module
        • nannyml.runner module
      • Module contents
  • Contributing
    • Spread the word
    • Be a part of the team
    • Contribute to the codebase
      • Get started coding
      • Pull Request Guidelines
      • Tips
NannyML
  • »
  • Welcome to NannyML’s documentation!
  • Edit on GitHub

Welcome to NannyML’s documentation!

PyPi Supported versions coverage

Contents:

  • Quickstart
    • What is NannyML?
    • Installing NannyML
    • Contents of the quickstart
    • Just the code
    • Walkthrough
    • Insights
    • What next
  • Tutorials
    • Data requirements
    • Estimating Performance
    • Monitoring Realized Performance
    • Comparing Estimated and Realized Performance
    • Detecting Data Drift
    • Adjusting Plots
    • Chunking
  • How It Works
    • Estimation of Performance of the Monitored Model
    • Data Reconstruction with PCA
    • Chunking Considerations
    • Calculating Sampling Error
  • Examples
    • Binary Classification: California Housing Dataset
  • Example Datasets
    • Synthetic Binary Classification Dataset
    • Synthetic Multiclass Classification Dataset
    • California Housing Dataset
    • Synthetic Regression Dataset
  • Glossary
  • Command Line Interface (CLI)
    • Running the CLI
    • Configuration file
    • Command overview
  • API reference
    • nannyml package
  • Contributing
    • Spread the word
    • Be a part of the team
    • Contribute to the codebase

Indices and tables

  • Index

  • Module Index

  • Search Page

Next

© Copyright 2022, NannyML. Revision bc84cd1c.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: v0.6.1
Versions
latest
stable
v0.6.1
v0.6.0
v0.5.3
v0.5.2
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.1
v0.2.0
main
Downloads
On Read the Docs
Project Home
Builds