NannyML
v0.5.1
Contents:
Quickstart
What is NannyML?
Installing NannyML
Contents of the quickstart
Just the code
Walkthrough
Estimating Performance without Targets
Detecting Data Drift
Insights
What next
Tutorials
Data requirements
Data Periods
Reference Period
Analysis Period
Columns
Timestamp
Target
Features
Model Output columns
Predicted class probabilities
Prediction class labels
NannyML Functionality Requirements
What next
Estimating Performance
Why Perform Performance Estimation
Estimating Performance for Binary Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Performance for Multiclass Classification
Just The Code
Walkthrough
Insights
What’s next
Estimating Performance for Regression
Monitoring Realized Performance
Why Monitoring Realized Performance
Monitoring Realized Performance for Binary Classification
Just The Code
Walkthrough
Insights
What Next
Monitoring Realized Performance for Multiclass Classification
Just The Code
Walkthrough
Insights
What Next
Monitoring Realized Performance for Regression
Comparing Estimated and Realized Performance
Detecting Data Drift
Univariate Drift Detection
Why Perform Univariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Multivariate Data Drift Detection
Why Perform Multivariate Drift Detection
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Model Outputs
Drift Detection for Binary Classification Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Multiclass Classification Model Outputs
Why Perform Drift Detection for Model Outputs
Just The Code
Walkthrough
What Next
Drift Detection for Model Targets
Drift Detection for Binary Classification Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
Insights
What Next
Drift Detection for Multiclass Classification Model Targets
Why Perform Drift Detection for Model Targets
Just The Code
Walkthrough
What Next
Adjusting Plots
Chunking
Why do we need chunks?
Walkthrough on creating chunks
Time-based chunking
Size-based chunking
Number-based chunking
Automatic chunking
Chunks on plots with results
How It Works
Data Reconstruction with PCA
Limitations of Univariate Drift Detection
“Butterfly” Dataset
Data Reconstruction with PCA
Understanding Reconstruction Error with PCA
Reconstruction Error with PCA on the butterfly dataset
Confidence-based Performance Estimation (CBPE)
CBPE algorithm
Binary classification
Multiclass Classification
Assumptions and Limitations
Appendix: Probability calibration
Chunking data
Chunking considerations
Underpopulated chunks
Not enough chunks
Minimum chunk size
Minimum Chunk Size for Performance Estimation and Performance Monitoring
Minimum Chunk Size for Multivariate Drift
Minimum Chunk for Univariate Drift
Examples
Binary Classification: California Housing Dataset
Load and prepare data
Performance Estimation
Comparison with the actual performance
Drift detection
Example Datasets
Synthetic Binary Classification Dataset
Problem Description
Dataset Description
Synthetic Multiclass Classification Dataset
Problem Description
Dataset Description
California Housing Dataset
Modifying California Housing Dataset
Enriching the data
Training a Machine Learning Model
Meeting NannyML Data Requirements
Glossary
CLI nml
Running the CLI
Installation
Configuration
Configuration file
Locations
Format
Input section
Output section
Column mapping section
Standalone parameters section
Templating paths
Examples
Command overview
run
Syntax
Options
Example
API reference
nannyml package
Subpackages
nannyml.cli package
Submodules
Module contents
nannyml.datasets package
Subpackages
Submodules
Module contents
nannyml.drift package
Subpackages
Submodules
Module contents
nannyml.io package
Submodules
Module contents
nannyml.performance_calculation package
Submodules
Module contents
nannyml.performance_estimation package
Subpackages
Module contents
nannyml.plots package
Submodules
Module contents
Submodules
nannyml.base module
nannyml.calibration module
nannyml.chunk module
nannyml.config module
nannyml.exceptions module
nannyml.runner module
Module contents
Contributing
Spread the word
Be a part of the team
Contribute to the codebase
Get started coding
Pull Request Guidelines
Tips
NannyML
»
Index
Edit on GitHub
Index
_
|
A
|
B
|
C
|
D
|
E
|
F
|
G
|
I
|
J
|
K
|
L
|
M
|
N
|
O
|
P
|
R
|
S
|
T
|
U
|
W
|
Y
_
__eq__() (nannyml.performance_calculation.metrics.Metric method)
__len__() (nannyml.chunk.Chunk method)
__new__() (nannyml.performance_estimation.confidence_based.cbpe.CBPE static method)
__repr__() (nannyml.chunk.Chunk method)
A
AbstractCalculator (class in nannyml.base)
AbstractCalculatorResult (class in nannyml.base)
AbstractEstimator (class in nannyml.base)
AbstractEstimatorResult (class in nannyml.base)
Alert
ALERT_COLUMN_SUFFIX (nannyml.drift.ranking.AlertCountRanking attribute)
AlertCountRanking (class in nannyml.drift.ranking)
analysis_data (nannyml.config.InputConfig attribute)
B
BinaryClassificationAccuracy (class in nannyml.performance_calculation.metrics)
BinaryClassificationAUROC (class in nannyml.performance_calculation.metrics)
BinaryClassificationF1 (class in nannyml.performance_calculation.metrics)
BinaryClassificationPrecision (class in nannyml.performance_calculation.metrics)
BinaryClassificationRecall (class in nannyml.performance_calculation.metrics)
BinaryClassificationSpecificity (class in nannyml.performance_calculation.metrics)
BLUE_SKY_CRAYOLA (nannyml.plots.colors.Colors attribute)
Butterfly dataset
by() (nannyml.drift.ranking.Ranker class method)
C
calculate() (nannyml.base.AbstractCalculator method)
(nannyml.performance_calculation.metrics.Metric method)
calculator_name (nannyml.base.AbstractCalculatorResult property)
(nannyml.drift.model_inputs.multivariate.data_reconstruction.results.DataReconstructionDriftCalculatorResult property)
(nannyml.drift.model_inputs.univariate.statistical.results.UnivariateStatisticalDriftCalculatorResult property)
(nannyml.drift.model_outputs.univariate.statistical.results.UnivariateDriftResult property)
(nannyml.drift.target.target_distribution.result.TargetDistributionResult property)
(nannyml.performance_calculation.result.PerformanceCalculatorResult property)
CalculatorException
CalculatorNotFittedException
calibrate() (nannyml.calibration.Calibrator method)
(nannyml.calibration.IsotonicCalibrator method)
(nannyml.calibration.NoopCalibrator method)
Calibrator (class in nannyml.calibration)
CalibratorFactory (class in nannyml.calibration)
CBPE (class in nannyml.performance_estimation.confidence_based.cbpe)
(Confidence-Based Performance Estimation)
CBPEPerformanceEstimatorResult (class in nannyml.performance_estimation.confidence_based.results)
Chi Squared test
Chunk (class in nannyml.chunk)
chunk_count (nannyml.config.ChunkerConfig attribute)
chunk_period (nannyml.config.ChunkerConfig attribute)
chunk_size (nannyml.config.ChunkerConfig attribute)
Chunker (class in nannyml.chunk)
chunker (nannyml.config.Config attribute)
ChunkerConfig (class in nannyml.config)
ChunkerException
ChunkerFactory (class in nannyml.chunk)
Colors (class in nannyml.plots.colors)
column_mapping (nannyml.config.Config attribute)
ColumnMapping (class in nannyml.config)
Concept Drift
Config (class in nannyml.config)
CountBasedChunker (class in nannyml.chunk)
create() (nannyml.calibration.CalibratorFactory class method)
(nannyml.performance_calculation.metrics.MetricFactory class method)
credentials (nannyml.config.InputDataConfig attribute)
(nannyml.config.OutputConfig attribute)
D
Data Chunk
Data Drift
Data Period
DataReconstructionDriftCalculator (class in nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator)
DataReconstructionDriftCalculatorResult (class in nannyml.drift.model_inputs.multivariate.data_reconstruction.results)
DEFAULT_CHUNK_COUNT (nannyml.chunk.DefaultChunker attribute)
DefaultChunker (class in nannyml.chunk)
E
estimate() (nannyml.base.AbstractEstimator method)
Estimated Performance
estimator_name (nannyml.base.AbstractEstimatorResult property)
(nannyml.performance_estimation.confidence_based.results.CBPEPerformanceEstimatorResult property)
F
Feature
feature_drift() (in module nannyml.plots.feature_drift)
features (nannyml.config.ColumnMapping attribute)
FileReader (class in nannyml.io.file_reader)
FileWriter (class in nannyml.io.file_writer)
fit() (nannyml.base.AbstractCalculator method)
(nannyml.base.AbstractEstimator method)
(nannyml.calibration.Calibrator method)
(nannyml.calibration.IsotonicCalibrator method)
(nannyml.calibration.NoopCalibrator method)
(nannyml.performance_calculation.metrics.Metric method)
format (nannyml.config.OutputConfig attribute)
G
get_chunker() (nannyml.chunk.ChunkerFactory class method)
get_config_path() (in module nannyml.config)
get_filepath_str() (in module nannyml.io.base)
get_protocol_and_path() (in module nannyml.io.base)
GRAY (nannyml.plots.colors.Colors attribute)
GRAY_DARK (nannyml.plots.colors.Colors attribute)
GREEN_SEA (nannyml.plots.colors.Colors attribute)
Ground truth
I
Identifier
ignore_errors (nannyml.config.Config attribute)
Imputation
INDIGO_PERSIAN (nannyml.plots.colors.Colors attribute)
input (nannyml.config.Config attribute)
InputConfig (class in nannyml.config)
InputDataConfig (class in nannyml.config)
InvalidArgumentsException
InvalidReferenceDataException
IOException
IsotonicCalibrator (class in nannyml.calibration)
J
join_column (nannyml.config.TargetDataConfig attribute)
K
Kolmogorov-Smirnov test
L
Latent space
LIGHT_GRAY (nannyml.plots.colors.Colors attribute)
load() (nannyml.config.Config class method)
load_csv_file_to_df() (in module nannyml.datasets.datasets)
load_modified_california_housing_dataset() (in module nannyml.datasets.datasets)
load_synthetic_binary_classification_dataset() (in module nannyml.datasets.datasets)
load_synthetic_car_loan_dataset() (in module nannyml.datasets.datasets)
load_synthetic_multiclass_classification_dataset() (in module nannyml.datasets.datasets)
M
Metric (class in nannyml.performance_calculation.metrics)
MetricFactory (class in nannyml.performance_calculation.metrics)
minimum_chunk_size() (nannyml.performance_calculation.metrics.Metric method)
MissingMetadataException
Model inputs
Model outputs
Model predictions
module
nannyml
nannyml.base
nannyml.calibration
nannyml.chunk
nannyml.cli
nannyml.cli.cli
nannyml.cli.run
nannyml.config
nannyml.datasets
nannyml.datasets.data
nannyml.datasets.datasets
nannyml.drift
nannyml.drift.model_inputs
nannyml.drift.model_inputs.multivariate
nannyml.drift.model_inputs.multivariate.data_reconstruction
nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator
nannyml.drift.model_inputs.multivariate.data_reconstruction.results
nannyml.drift.model_inputs.univariate
nannyml.drift.model_inputs.univariate.statistical
nannyml.drift.model_inputs.univariate.statistical.calculator
nannyml.drift.model_inputs.univariate.statistical.results
nannyml.drift.model_outputs
nannyml.drift.model_outputs.univariate
nannyml.drift.model_outputs.univariate.statistical
nannyml.drift.model_outputs.univariate.statistical.calculator
nannyml.drift.model_outputs.univariate.statistical.results
nannyml.drift.ranking
nannyml.drift.target
nannyml.drift.target.target_distribution
nannyml.drift.target.target_distribution.calculator
nannyml.drift.target.target_distribution.result
nannyml.exceptions
nannyml.io
nannyml.io.base
nannyml.io.file_reader
nannyml.io.file_writer
nannyml.performance_calculation
nannyml.performance_calculation.calculator
nannyml.performance_calculation.metrics
nannyml.performance_calculation.result
nannyml.performance_estimation
nannyml.performance_estimation.confidence_based
nannyml.performance_estimation.confidence_based.cbpe
nannyml.performance_estimation.confidence_based.results
nannyml.plots
nannyml.plots.colors
nannyml.plots.feature_drift
nannyml.runner
MulticlassClassificationAccuracy (class in nannyml.performance_calculation.metrics)
MulticlassClassificationAUROC (class in nannyml.performance_calculation.metrics)
MulticlassClassificationF1 (class in nannyml.performance_calculation.metrics)
MulticlassClassificationPrecision (class in nannyml.performance_calculation.metrics)
MulticlassClassificationRecall (class in nannyml.performance_calculation.metrics)
MulticlassClassificationSpecificity (class in nannyml.performance_calculation.metrics)
Multivariate Drift Detection
N
nannyml
module
nannyml.base
module
nannyml.calibration
module
nannyml.chunk
module
nannyml.cli
module
nannyml.cli.cli
module
nannyml.cli.run
module
nannyml.config
module
nannyml.datasets
module
nannyml.datasets.data
module
nannyml.datasets.datasets
module
nannyml.drift
module
nannyml.drift.model_inputs
module
nannyml.drift.model_inputs.multivariate
module
nannyml.drift.model_inputs.multivariate.data_reconstruction
module
nannyml.drift.model_inputs.multivariate.data_reconstruction.calculator
module
nannyml.drift.model_inputs.multivariate.data_reconstruction.results
module
nannyml.drift.model_inputs.univariate
module
nannyml.drift.model_inputs.univariate.statistical
module
nannyml.drift.model_inputs.univariate.statistical.calculator
module
nannyml.drift.model_inputs.univariate.statistical.results
module
nannyml.drift.model_outputs
module
nannyml.drift.model_outputs.univariate
module
nannyml.drift.model_outputs.univariate.statistical
module
nannyml.drift.model_outputs.univariate.statistical.calculator
module
nannyml.drift.model_outputs.univariate.statistical.results
module
nannyml.drift.ranking
module
nannyml.drift.target
module
nannyml.drift.target.target_distribution
module
nannyml.drift.target.target_distribution.calculator
module
nannyml.drift.target.target_distribution.result
module
nannyml.exceptions
module
nannyml.io
module
nannyml.io.base
module
nannyml.io.file_reader
module
nannyml.io.file_writer
module
nannyml.performance_calculation
module
nannyml.performance_calculation.calculator
module
nannyml.performance_calculation.metrics
module
nannyml.performance_calculation.result
module
nannyml.performance_estimation
module
nannyml.performance_estimation.confidence_based
module
nannyml.performance_estimation.confidence_based.cbpe
module
nannyml.performance_estimation.confidence_based.results
module
nannyml.plots
module
nannyml.plots.colors
module
nannyml.plots.feature_drift
module
nannyml.runner
module
needs_calibration() (in module nannyml.calibration)
NoopCalibrator (class in nannyml.calibration)
NotFittedException
O
output (nannyml.config.Config attribute)
OutputConfig (class in nannyml.config)
P
Partition Column
path (nannyml.config.InputDataConfig attribute)
(nannyml.config.OutputConfig attribute)
PCA
Performance Estimation
PerformanceCalculator (class in nannyml.performance_calculation.calculator)
PerformanceCalculatorResult (class in nannyml.performance_calculation.result)
PeriodBasedChunker (class in nannyml.chunk)
plot() (nannyml.base.AbstractCalculatorResult method)
(nannyml.base.AbstractEstimatorResult method)
(nannyml.drift.model_inputs.multivariate.data_reconstruction.results.DataReconstructionDriftCalculatorResult method)
(nannyml.drift.model_inputs.univariate.statistical.results.UnivariateStatisticalDriftCalculatorResult method)
(nannyml.drift.model_outputs.univariate.statistical.results.UnivariateDriftResult method)
(nannyml.drift.target.target_distribution.result.TargetDistributionResult method)
(nannyml.performance_calculation.result.PerformanceCalculatorResult method)
(nannyml.performance_estimation.confidence_based.results.CBPEPerformanceEstimatorResult method)
Predicted labels
Predicted probabilities
Predicted scores
Predictions
R
rank() (nannyml.drift.ranking.AlertCountRanking method)
(nannyml.drift.ranking.Ranking method)
Ranker (class in nannyml.drift.ranking)
Ranking (class in nannyml.drift.ranking)
read() (nannyml.io.base.Reader method)
read_args (nannyml.config.InputDataConfig attribute)
Reader (class in nannyml.io.base)
Realized Performance
Reconstruction Error
RED_IMPERIAL (nannyml.plots.colors.Colors attribute)
reference_data (nannyml.config.InputConfig attribute)
register() (nannyml.drift.ranking.Ranker class method)
(nannyml.performance_calculation.metrics.MetricFactory class method)
register_calibrator() (nannyml.calibration.CalibratorFactory class method)
registry (nannyml.drift.ranking.Ranker attribute)
(nannyml.performance_calculation.metrics.MetricFactory attribute)
run() (in module nannyml.runner)
S
SAFFRON (nannyml.plots.colors.Colors attribute)
SizeBasedChunker (class in nannyml.chunk)
split() (nannyml.chunk.Chunker method)
StatisticalOutputDriftCalculator (class in nannyml.drift.model_outputs.univariate.statistical.calculator)
T
Target
target_data (nannyml.config.InputConfig attribute)
TargetDataConfig (class in nannyml.config)
TargetDistributionCalculator (class in nannyml.drift.target.target_distribution.calculator)
TargetDistributionResult (class in nannyml.drift.target.target_distribution.result)
Timestamp
timestamp (nannyml.config.ColumnMapping attribute)
U
Univariate Drift Detection
UnivariateDriftResult (class in nannyml.drift.model_outputs.univariate.statistical.results)
UnivariateStatisticalDriftCalculator (class in nannyml.drift.model_inputs.univariate.statistical.calculator)
UnivariateStatisticalDriftCalculatorResult (class in nannyml.drift.model_inputs.univariate.statistical.results)
W
write() (nannyml.io.base.Writer method)
write_args (nannyml.config.OutputConfig attribute)
Writer (class in nannyml.io.base)
WriterException
Y
y_pred (nannyml.config.ColumnMapping attribute)
y_pred_proba (nannyml.config.ColumnMapping attribute)
y_true (nannyml.config.ColumnMapping attribute)
Read the Docs
v: v0.5.1
Versions
latest
stable
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.1
v0.2.0
main
Downloads
On Read the Docs
Project Home
Builds