Chunking data
Chunking considerations
For an introduction on chunks please have a look at Setting Up: Chunking. This guide focuses on the potential issues that may arise from using chunks. They are described below.
Underpopulated chunks
Depending on the selected chunking method and the provided datasets, some chunks may not have a lot of observations inside them. In fact, they might be so small that results obtained are governed by noise rather than actual signal. NannyML estimates a minimum chunk size based on the reference data (see how in the section on minimum chunk size). If some of the created chunks are smaller than the minimum chunk size, a warning will be raised.
>>> import pandas as pd
>>> import nannyml as nml
>>> reference, analysis, _ = nml.datasets.load_synthetic_binary_classification_dataset()
>>>
>>> cbpe = nml.CBPE(
... y_pred_proba='y_pred_proba',
... y_pred='y_pred',
... y_true='work_home_actual',
... timestamp_column_name='timestamp',
... chunk_period="Q",
... metrics=['roc_auc']
>>> ).fit(reference_data=reference)
>>> est_perf = cbpe.estimate(analysis)
UserWarning: The resulting list of chunks contains 1 underpopulated chunks. They contain too few records to be statistically robust and might negatively influence the quality of calculations. Please consider splitting your data in a different way or continue at your own risk.
When the warning is about a single chunk, it is usually the last chunk. This is due to the reasons described in the chunking tutorial.
When there are more than one underpopulated chunks staying with the selected chunking method may be suboptimal. Read minimum chunk size to get more information about the effect of small chunks. Beware of the trade-offs involved when selecting the chunking method.
Not enough chunks
Sometimes the selected chunking method might not generate enough chunks in the reference period.
NannyML calculates thresholds based on the variability of metrics measured in the reference chunks (see how thresholds
are calculated for performance estimation). Having 6 chunks is
far from optimal, but is a reasonable minimum. If there are less than 6 chunks, a warning will be raised.
>>> cbpe = nml.CBPE(
... y_pred_proba='y_pred_proba',
... y_pred='y_pred',
... y_true='work_home_actual',
... timestamp_column_name='timestamp',
... chunk_number=5,
... metrics=['roc_auc']
>>> ).fit(reference_data=reference)
>>> est_perf = cbpe.estimate(analysis)
UserWarning: The resulting number of chunks is too low. Please consider splitting your data in a different way or continue at your own risk.
Minimum chunk size
Small sample size strongly affects the reliability of any ML or statistical analysis, including data drift detection and performance estimation. NannyML allows splitting data in chunks in different ways to let users choose chunks that are meaningful for them. However, when the chunks are too small, statistical results may become unreliable. In this case NannyML will issue a warning. The user can then chose to ignore it and continue or use a chunking method that will result in bigger chunks.
Minimum Chunk Size for Performance Estimation and Performance Monitoring
When the chunk size is small what looks like a significant drop in performance of the monitored model may only be a sampling effect. To better understand that, have a look at the histogram below.
It shows dispersion of accuracy for a random model predicting a random binary target (which by definition should be 0.5) for a sample of 100 observations. It is not uncommon to get accuracy of 0.6 for some samples. The effect is even stronger for more complex metrics like ROC AUC.
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> from sklearn.metrics import accuracy_score
>>> sample_size = 100
>>> dataset_size = 10_000
>>> # random model
>>> y_true = np.random.binomial(1, 0.5, dataset_size)
>>> y_pred = np.random.binomial(1, 0.5, dataset_size)
>>> accuracy_scores = []
>>> for experiment in range(10_000):
>>> subset_indexes = np.random.choice(dataset_size, sample_size, replace=False) # get random indexes
>>> y_true_subset = y_true[subset_indexes]
>>> y_pred_subset = y_pred[subset_indexes]
>>> accuracy_scores.append(accuracy_score(y_true_subset, y_pred_subset))
>>> plt.hist(accuracy_scores, bins=20, density=True)
>>> plt.title("Accuracy of random classifier\n for randomly selected samples of 100 observations.");
When there are many chunks, it is easy to spot the noisy nature of fluctuations. However, with only a few chunks, it is difficult to tell whether the observed changes are significant. To minimize this risk, NannyML estimates a minimum chunk size for the monitored data and raises a warning if the selected chunking method results in chunks that are smaller.
The minimum chunk size is estimated in order to keep variation of performance of the monitored model low. The variation is expressed in terms of standard deviation and it is considered low when it is below 0.02. In other words, for the selected evaluation metric, NannyML estimates a chunk size for which the standard deviation of performance on chunks resulting purely from sampling is lower than 0.02.
Let’s go through the process of estimating the accuracy score from the example above. Selecting a chunk in the data and calculating performance for it is similar to sampling a set from a population and calculating a statistic. When the statistic is a mean, the Standard Error (SE) formula 1 can be used to estimate the standard deviation of the sampled means.
\[{\sigma }_{\bar {x}}\ ={\frac {\sigma }{\sqrt {n}}}\]
In order to take advantage of the SE formula, accuracy for each observation separately needs to be calculated. Accuracy for a single observation is simply equal to 1 when the prediction is correct and equal to 0 otherwise. With observation-level accuracies in place, accuracy for the whole sample can be calculated as the mean of them. After this transformation the SE formula can be used directly to estimate the standard error of accuracy as a function of the sample.
>>> obs_level_accuracy = y_true == y_pred
>>> np.mean(obs_level_accuracy), accuracy_score(y_true, y_pred)
(0.4988, 0.4988)
Now the SE formula can be used to estimate the standard deviation and compare it with the standard deviation from the sampling experiments above.
>>> SE_std = np.std(obs_level_accuracy)/np.sqrt(sample_size)
>>> SE_std, np.std(accuracy_scores)
(0.04999932399543018, 0.04946720594494903)
The same formula can be used to estimate the sample size for the required standard deviation.
>>> required_std = 0.02
>>> sample_size = (np.std(correct_predictions)**2)/required_std**2
>>> sample_size
624.99
So for the analyzed case, the chunk size of 625 observations will result in a standard error of accuracy equal to 0.02. In other words, if we calculate the accuracy of this model on a large number of samples with 625 observations each, standard deviation of these accuracies will be about 0.02. This dispersion will be purely the effect of sampling because model quality and data distribution remain unchanged. In the current NannyML implementation, the estimated chunk size is rounded to full hundredths, 600 in the example above. Additionally, if the estimation returns a number lower than 300, the minimum chunk size suggested is 300.
Generally the SE formula gives the exact value when
The standard deviation of the population is known.
The samples are statistically independent.
Both of these requirements are in fact violated. When the data is split into chunks it is not sampled from population, it comes from a finite set. Therefore standard deviation of population is unknown. Also, chunks are not independent - observations in chunks are selected chronologically, not randomly. They are also drawn without replacement, meaning the same observation cannot be selected twice. Nevertheless, this approach provides an estimation with good enough precision for our use case while keeping the computation time very low.
Estimation of minimum chunk size for other metrics, such as ROC AUC, precision, recall etc. is performed in a similar manner.
Minimum Chunk Size for Multivariate Drift
To ensure that there is no significant noise present in multivariate drift results NannyML suggests a minimum chunk size based on the number of features used to perform data reconstruction according to the function below.
This result is based on internal testing. It is merely a suggestion because multidimensional data can have difficult to foresee instabilities.
Minimum Chunk for Univariate Drift
To ensure that there is no significant noise present in Univariate Drift Detection the recommended minimum chunk size is 500. It is a rule of thumb that should cover most common cases.
References