How It Works

• Estimation of Performance of the Monitored Model
  • Confidence-based Performance Estimation (CBPE)
  • Direct Loss Estimation (DLE)
  • Other Approaches to Estimate Performance of Regression Models
• Presenting Univariate Drift Detection Methods
  • Methods for Continuous Features
  • Methods for Categorical Variables
• Choosing Univariate Drift Detection Methods
  • Comparison of Methods for Continuous Variables
  • Comparison of Methods for Categorical Variables
  • Results Summary (TLDR)
• Ranking
  • Alert Count Ranking
  • Correlation Ranking
• Data Reconstruction with PCA
  • Limitations of Univariate Drift Detection
  • Data Reconstruction with PCA
  • Understanding Reconstruction Error with PCA
• Chunking Considerations
  • Not Enough Chunks
  • Not Enough Observations in Chunk
  • Impact of Chunk Size on Reliability of Results
• Calculating Sampling Error
  • Defining Sampling Error from Standard Error of the Mean
  • Sampling Error Estimation and Interpretation for NannyML features
  • Assumptions and Limitations