Unseen Values Detection

Just The Code

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference, analysis, analysis_targets = nml.load_titanic_dataset()
>>> display(reference.head())

>>> selected_columns = [
...     'Sex', 'Ticket', 'Cabin', 'Embarked',
>>> ]
>>> calc = nml.UnseenValuesCalculator(
...     column_names=selected_columns,
>>> )

>>> calc.fit(reference)
>>> results = calc.calculate(analysis)
>>> display(results.filter(period='all').to_df())

>>> for column_name in results.column_names:
...     results.filter(column_names=column_name).plot().show()

Walkthrough

NannyML defines unseen values as categorical feature values that are not present in the reference period. NannyML’s approach to unseen values detection is simple. The reference period is used to create a set of expected values for each categorical feature. For each chunk in the analysis period NannyML calculates the number of unseen values. There is an option, called normalize, to convert the count of values to a relative ratio if needed. If unseen values are detected in a chunk, an alert is raised for the relevant feature.

We begin by loading the titanic dataset provided by the NannyML package.

>>> import nannyml as nml
>>> from IPython.display import display

>>> reference, analysis, analysis_targets = nml.load_titanic_dataset()
>>> display(reference.head())

	PassengerId	Pclass	Name	Sex	Age	SibSp	Ticket	Fare	Cabin	Embarked	boat	body	home.dest	Survived
0	1	3	Braund, Mr. Owen Harris	male	22	1	A/5 21171	7.25	nan	S	nan	nan	Bridgerule, Devon	0
1	2	1	Cumings, Mrs. John Bradley (Florence Briggs Thayer)	female	38	1	PC 17599	71.2833	C85	C	4	nan	New York, NY	1
2	3	3	Heikkinen, Miss. Laina	female	26	0	STON/O2. 3101282	7.925	nan	S	nan	nan	nan	1
3	4	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35	1	113803	53.1	C123	S	D	nan	Scituate, MA	1
4	5	3	Allen, Mr. William Henry	male	35	0	373450	8.05	nan	S	nan	nan	Lower Clapton, Middlesex or Erdington, Birmingham	0

The UnseenValuesCalculator class implements the functionality needed for unseen values calculations. We need to instantiate it with appropriate parameters:

The names of the columns to be evaluated. They need to be categorical columns.
Optionally, a boolean option indicating whether we want the absolute count of the unseen value instances or their relative ratio. By default it is set to true.
Optionally, the name of the column containing the observation timestamps.
Optionally, a chunking approach or a predefined chunker. If neither is provided, the default chunker creating 10 chunks will be used.
Optionally, a threshold strategy to modify the default one. See available threshold options here.

Warning

Note that because of how unseen values are defined they will be 0 by definition for the reference period. Hence the StandardDeviationThreshold threshold option is not really applicable for this calculator.

>>> selected_columns = [
...     'Sex', 'Ticket', 'Cabin', 'Embarked',
>>> ]
>>> calc = nml.UnseenValuesCalculator(
...     column_names=selected_columns,
>>> )

Next, the fit() method needs to be called on the reference data, which provides the baseline that the analysis data will be compared with for alert generation. Then the calculate() method will calculate the data quality results on the data provided to it.

The results can be filtered to only include a certain data period, method or column by using the filter method. You can evaluate the result data by converting the results into a DataFrame, by calling the to_df() method. By default this will return a DataFrame with a multi-level index. The first level represents the column, the second level represents resulting information such as the data quality metric values and the alert thresholds.

>>> calc.fit(reference)
>>> results = calc.calculate(analysis)
>>> display(results.filter(period='all').to_df())

	chunk key	chunk_index	start_index	end_index	period	alert	Ticket value	alert	Cabin value	alert	alert
0	[0:88]	0	0	88	reference	False	0	False	0	False	False
1	[89:177]	1	89	177	reference	False	0	False	0	False	False
2	[178:266]	2	178	266	reference	False	0	False	0	False	False
3	[267:355]	3	267	355	reference	False	0	False	0	False	False
4	[356:444]	4	356	444	reference	False	0	False	0	False	False
5	[445:533]	5	445	533	reference	False	0	False	0	False	False
6	[534:622]	6	534	622	reference	False	0	False	0	False	False
7	[623:711]	7	623	711	reference	False	0	False	0	False	False
8	[712:800]	8	712	800	reference	False	0	False	0	False	False
9	[801:890]	9	801	890	reference	False	0	False	0	False	False
10	[0:40]	0	0	40	analysis	False	0.609756	True	0.0731707	True	False
11	[41:81]	1	41	81	analysis	False	0.634146	True	0.219512	True	False
12	[82:122]	2	82	122	analysis	False	0.731707	True	0.146341	True	False
13	[123:163]	3	123	163	analysis	False	0.634146	True	0.0731707	True	False
14	[164:204]	4	164	204	analysis	False	0.536585	True	0.097561	True	False
15	[205:245]	5	205	245	analysis	False	0.658537	True	0.0731707	True	False
16	[246:286]	6	246	286	analysis	False	0.756098	True	0.0731707	True	False
17	[287:327]	7	287	327	analysis	False	0.707317	True	0.097561	True	False
18	[328:368]	8	328	368	analysis	False	0.536585	True	0.0487805	True	False
19	[369:417]	9	369	417	analysis	False	0.571429	True	0.183673	True	False

More information on accessing the information contained in the Result can be found on the Working with results page.

The next step is visualizing the results, which is done using the plot() method. It is recommended to filter results for each column and plot separately.

>>> for column_name in results.column_names:
...     results.filter(column_names=column_name).plot().show()

Insights

We see that most of the dataset columns don’t have unseen values. The Ticket and Cabin columns are the most interesting with regards to unseen values.

What Next

We can also inspect the dataset for missing values in the Missing Values Tutorial. Then we can look for any Data Drift present in the dataset using Detecting Data Drift functionality of NannyML.