An alert refers to a variable at a particular chunk that gets flagged for possible data drift. The alert is raised after the drift functionality of NannyML finds the drift characteristics for this variable and chunk to be suspect.
- Butterfly dataset
A dataset used in Data Reconstruction with PCA to give an example where univariate drift statistics are insufficient in detecting complex data drifts in multidimensional data.
- CBPE (Confidence-Based Performance Estimation)
A family of methods to estimate model performance in the absence of ground truth that take advantage of the confidence which is expressed in the monitored model output probability/score prediction.
- Concept Drift
- Data Drift
A change in joint distribution of Model Inputs (P(X)).
- Data Chunk
Data chunk is simply a sample of data. All the results generated by NannyML are calculated and presented on the level of chunk i.e. a chunk is a single data point on the monitoring results. Chunks are usually created based on time periods - they contain all the observations and predictions from single hour, day, month etc. depending on the selected interval. They can be also size-based so that each chunk contains n observations, or number-based so the whole data is splt into k chunks. In each case chronology of data between chunks is maintained.
- Estimated Performance
The performance the monitored model is expected to have as a result of the Performance Estimation process. Estimated performance can be available immediately after predictions are made.
A variable used by our machine learning model. The model inputs consist of features.
- Latent space
A space of reduced dimensionality, compared to the model input space, that can represent our input data. This space is the result of a representation learning algorithm. Data points that are close together in the model input space are also close together in the latent space.
- Ground truth
A synonym for Target.
Usually a single column, but can be multiple columns where necessary. It is used uniquely identify an observation. When providing Target data at a later point in time, this value can help refer back to the original prediction.
Being able to uniquely identify each row of data can help reference any particular issues NannyML might identify and make resolving issues easier for you. As we add functionality to provide
targetdata afterwards your data will already be in the correct shape to support it!
No specific format. Any str or int value is possible.
An existing identifier from your business case. A technical identifier such as a globally unique identifier (GUID). A hash of some (or all) of your column values, using a hashing function with appropriate collision properties, e.g. the SHA-2 and SHA-3 families. A concatenation of your dataset name and a row number.
The process of substituting missing values with actual values on a dataset.
- Model inputs
Every Feature used by the model.
- Model Metadata
Additional information regarding the model inputs such as the type of a particular feature (continuous, categorical, nominal, ordinal). Information regarding when a prediction was made as well as which data partition it is assigned to are also considered metadata.
- Model outputs
The scores or probabilities that your model predicts for its target outcome.
- Model predictions
A synonym for Model outputs.
- Multivariate Drift Detection
Drift Detection steps that involve all model features in order to create appropriate drift measures.
- Data Period
A data period is a subset of the data used to monitor a model. NannyML expects the provided data to be in one of two data periods.
The first data period is called the
referenceperiod. It contains all the observations for a period with an accepted level of performance. It most likely also includes
The second subset of the data is the
analysisperiod. It contains the observations you want NannyML to analyse. In the absence of targets performance in the analysis period can be estimated.
NannyML needs the period information to understand which data it can use as a reference to compare other periods by. This is contained in the Partition Column.
We currently only support the following partition column values:
Please map your own values to them accordingly.
- Partition Column
A column that tells us what Data Period the data is in. A partition column is necessary for NannyML in order to produce model monitoring results.
Principal Component Analysis. A method used for dimensionality reduction.
- Performance Estimation
Estimating performance of a deployed ML model without having access to Target.
A synonym for Model outputs.
- Predicted labels
The outcome a machine learning model predicts for the event it was called to predict. Predicted labels are a two value categorical variable. They can be represented by integers, usually 0 and 1, booleans, meaning True or False, or strings. For NannyML, in a binary classification problem, it is ideal if predicted labels are presented as integers with 1 representing the positive outcome.
- Predicted probabilities
The probabilities assigned by a machine learning model regarding the chance that a positive event materializes for the binary outcome it was called to predict.
- Predicted scores
Sometimes the prediction of a machine learning model is transformed into a continuous range of real numbers. Those scores take values outside the [0,1] range that is allowed for probabilities. The higher the score the more likely the positive outcome should be.
- Realized Performance
- Reconstruction Error
The average euclidean distance between the original and the reconstructed data points in a dataset.
The actual outcome of the event the machine learning model is trying to predict. Also referred to as Ground truth.
Usually a single column, but can be multiple columns where necessary. This provides NannyML with the date and time that the prediction was made.
NannyML need to understand when predictions were made, and how you record this, so it can bucket observations in time periods.
Any format supported by Pandas, most likely:
ISO 8601, e.g.
Unix-epoch in units of seconds, e.g.
- Univariate Drift Detection
Drift Detection methods that use each model feature individually in order to detect change in Model Inputs.